Commentroversy, plus owls are the trendiest fowl.

Commentroversy is a five-year-old word according to Urban Dictionary, but it was new to me, when I saw it in the comment section of Yarn Harlot's latest blog post.  And the storm in a teacup that has so many commentators making puns about stitches and needles? The USOC asked Ravelry not to use the term "Ravelympics" to describe the event in which knitters set ambitious knitting goals during the real Olympics. Read the Yarn Harlot or Gawker, I think the shitstorm is funnier than the issue itself and don't have much else to say.

In unrelated news, I bought this t-shirt because it made me laugh.  But then I had to try to explain it.  Does it have something to do with YOLO being an acronym for "You Only Live Once"? I do not know.  But I did have a suspicion that owls are an arty crafty Thing.  And I confirmed they are a Thing, without even counting macrame owls.  You thought that ducks and chickens were overdone on home decorations?   Owls are the trendiest fowl by far.  In my totally scientific study based on Etsy search totals, here is the order of popularity of animals in kitsch:

Bird: 294,000
Butterfly: 144,000
Cat*: 118,000
Owl: 112,000
Dog*: 110,000
Fish: 67,000
Bear: 63,000
Horse: 56,000
Rabbit: 49,000
Mouse**: 47,000
Monkey: 43,000
Elephant: 42,000
Bee: 32,000
Deer: 25,000
Frog: 25,000
Ladybug: 24,000
Turtle: 23,000
Duck: 22,000
Lion: 20,000
Turkey: 19,000
Pig: 18,000
Fox: 17,000
Chicken: 17,000
Cow: 17,000
Sheep: 16,000
Sparrow: 15,000
Wolf: 13,000
Eagle: 10,000
Crab: 8,800
Panda***: 8,200
Parrot: 6,900
Beetle: 5,800
Lizard: 5,700

Rat: 4,400
Gorilla****: 1,200

* I subtracted the hits for pet accessories, because there's handmade dog and cat collars, beds, and toys, which don't count as Cute Handmade [Animal] Themed Tchotchkes.  If you just count housewares, for example, owls beat cats.


** Turns out, half the mouses on Etsy are Minnie.


*** You think you're cute, panda?  You're small-time, niche, pissant cute.  You can't even reach the tail of a stuffed owl.  I mean, really, you got beat out by crabs.  Go home and eat shoots and leaves.


**** Disturbing.


***** Tiger and Zebra were omitted due to the hard-to-control-for number of results with tiger eye gems, tiger stripe and zebra stripe fabrics. ***** <-- self-referential footnote

Better software development estimates

Is it possible to estimate software tasks better?  In certain circumstances, my experience tells me yes!  It's not just a wish (/me waves to commenter on last post).

First, to establish what kind of estimates I'm talking about.  It's not possible to estimate anything past two months, in my experience.  When I was working at Microsoft and heard lots of estimates from other teams, I realized that "This software product will take us two years" really meant "We have no idea and we will never finish a product that recognizably matches our plan".  A one year estimate turn into a somewhat recognizable outcome in two or three years.  Even a six month estimate, while it might turn into a year-long project fairly reliably, ends up finishing quite different tasks than the planned tasks which originally led to the six month estimate.

Other things that make a difference in estimation accuracy:
 * What language is being used?  Snags can be a bigger time dilator with a compiled or lower-level language like C, whereas hitting a snag in Python might not throw out the estimate that much.
 * Is something new being integrated? Any time a task involves compiling and linking a new library, or adding a new Ruby gem, I know the estimate is weaker.
 * How routine is it? Adding a new page to a Ruby on Rails project can be pretty damn predictable.
 * How many people are involved?  Estimating a one-person task is way more accurate than estimating a team project.  Even a one-person task that has an issue that requires another person's answer is less accurate than the task that can be done by one person independently.
 * How many other estimated tasks are dependencies?  If you have a chain of tasks to estimate, a change in one task can throw out the others.

And then there's bias.  Have you ever noticed that some developers always estimate high and some always estimate low?  It's intriguing, because even though they're consistently wrong, they are consistently wrong in the same direction.  That means they are actually giving management good information, if management knows their biases.  I once managed one optimist and one pessimist in the same team for over two years.  The optimist always estimated about 1/3 of his actual time required. The pessimist always estimated about three times the actual time required.  I would assign what looked like three months of work to the pessimist, and what looked like a week and a half of work to the optimist, and they would finish around the same time.

One of the things I really do love about Agile (and Pivotal Tracker reifies this) is how it understands the above points.
 * Task length? Agile encourages developers to break work down into smaller tasks (no three-week tasks in any agile process I've ever seen)
 * How many people are involved?  Agile encourages issues to be resolved before the estimate is even made, and it is designed for single-person tasks.
 *  How many unfinished dependencies are there?  Agile encourages planning with a shorter horizon, so the chain of dependencies is usually reduced.
 * Consistent bias?  Agile tracks velocity, not accuracy, so a consistent bias is simply a constant part of a consistent velocity.

With all that, one of the main things that intrigues me, and this is what I'm unpacking from my previous post, is whether better feedback would help developer estimates get even better than Agile already makes them.  Agile does not measure time spent so it doesn't give developers feedback that would allow them to either fix a consistent overall bias, or to start to recognize tasks that need to be estimated a little higher.

Scrum thinks people are stupid

One thing I both love and hate about the hardcore agile process, particularly the Scrum process, is that it tends to work around participants' biases, weaknesses, and lack of trust, rather than try to fix them. Scrum may not even leave room for improvements in trust or reliability. Some examples:

Developers' estimation biases are worked around, not fixed
Because developers have biases in estimating tasks (biases that are consistent over time and on average), scrum "velocity" measures estimates against results.  It doesn't attempt to fix estimates, e.g. by showing a developer the difference between their estimates and time spent.  In fact, velocity doesn't measure time spent at all, and lumps all developers in a team, over-estimators and under-estimators together, in one velocity measurement.

This is probably for the best -- it's one simple measure that's remarkably consistent.  Still, I wonder if it wouldn't be more useful in the long run to learn to estimate better.  I've never seen a really good estimation feedback loop in the software development context, but wouldn't it be neat to try?

Product owners' changes are either completely allowed or disallowed
Traditionally engineering teams have to train product owners not to change the product plans all the time.  This involves frequent team arguments about product plan changes.  Instead, scrum tries to carve out one small space where product changes are forbidden, and allow all other changes without argument.  In the current interval, the product owner cannot make any changes or the current plans are all tossed out and the estimation process is restarted, a consequence severe enough to effectively forbid any small changes.

This rule does keep product owners off the developers' backs.  If the product owner is thinking of a change to this week's plans, the consequences of an "abnormal sprint termination" probably stop them.  If the product owner is thinking of a change to next week's plans, the upcoming sprint planning meeting is where they'll discuss it.  Either way, the product owner does not walk up to a developer and say "Hey!  I've got a great idea!"

If keeping the product owner off the developer's back sounds like a really good idea to you, well you probably haven't worked with trusted, experienced product owners.  And if the team has processes that reify the distrust, then there's less chance to build trust.

Demos help people who can't analyse abstract plans
Weekly demos are a course correction mechanism. In order for anything to be marked done, it must be demoed.  And when a feature is demoed, the product owner can see the practical consequences better than they could when the feature was designed.  Now the product owner is able to immediately add things to the backlog, which might get done in the next iteration, and this is good.  Iterative design for the win.

The practical consequence of this appears to be less specification and planning work, which is good (avoid overplanning), but taken to the point where the product owners do not feel any particular pressure to understand and analyze the design.  Instead of sitting in front of the wireframes and thinking it through -- "What happens if I press this if it were implemented?  If this were a real system and the item had been deleted, what would the user need to see? " It's a difficult what-if skill and takes practice.   It doesn't make design any less iterative!  Instead, it moves the iterations into the design phase rather than the costly implementation phase.

Frequent demos, and the scrum rules which allow for any changes in the backlog, seem to remove some of the need to develop abstract design skills.  That makes me a little sad.  Still, frequent demos and iterations are a tool I'd use in any software development process.

Optimizing considered harmful
In hardcore scrum, developers are practically forbidden from doing any changes now that would make future work easier.  It's discouraged, and the way the system tracks tasks makes it unrewarding to do.

Let's say that today my task is to create a feature for users to delete items. There's also a story in the backlog or icebox for undeleting items (e.g. finding them in a trash folder and returning them).  Well, the way success is structured in scrum, I estimate the time to delete items at the beginning of the period where I do that work, and it makes everybody happier if I figure out how to delete items without much work, and we make more features fit in this period.  It doesn't help me now to estimate high in order to prepare for the 'undelete' feature.  It doesn't help me in the future either -- when we get to the scrum meeting where we estimate the  'undelete' feature, it might not even be me doing the feature.  (In theory, developers are supposed to be interchangeable in agile/scrum).  Even if it is me,  it's no big deal to have to do the undelete by rewriting the way delete worked -- I just build that into my estimates for this period.

There's no overall project schedule view that would have showed the value for doing these two features together and doing delete right the first time.  There are other ways where optimizing is actively discouraged:

• Literally, optimizing speed or resource use is discouraged.  Functional stories only say that the feature has to work.  Other stories in the future might or might not say that the feature has to work in 2 seconds.  Security might also be an afterthought!
• Optimizing by expertise is discouraged.  Everybody in the team is supposed to be interchangeable.  If all of this week's features are GUI features, then everybody does GUI work. If all of the next week's features are security features, then everybody does security work. 
• Optimizing the sequence of work by engineering constraints is discouraged.  If it would be faster to do feature A after the more general feature B is completed, too bad.  If there's a pileup of work with dependencies, and it's slipping to the end of the backlog where the dependencies will start to slow down each task, too bad.  Only the priority sorting of the product owner is valid.

I've seen passionate arguments for "You Aren't Gonna Need It" (YAGNI), and they're right.  Engineers often optimize prematurely.  Engineers often predict what they think the user will need and can be wrong.   But none of the passionate arguments for YAGNI think that it can be used without a sense of balance, right?  So when scrum methodology encourages developers to always put off optional work, scrum puts its thumb on the balance.  It short-circuits the discussions of complex tradeoffs and simply says do it later. 

Summary

What's common among all these traits is an attitude in scrum not only that people are fallible, but they're routinely wrong, and that they can't be trusted to work together on complex issues for the greater good. That means that it works best in environments where that attitude is more true -- e.g. a contract development team, comprised of decent engineers but not particularly specialized, who don't know much about the end use and maybe don't care too much, working with product owners who aren't experienced in software design, are way too busy to write specs and discuss complex tradeoffs, and are always tempted to change things they shouldn't.  

Agile should be done differently when the engineers are specialized, care about what they're doing and what it will be used for, work closely with the product owners in the same business unit, and the product owners are smart and can be trusted to learn good software development practices.  Just how it should be done differently is an open question to me. There are agile process variants that take different attitudes (e.g. those that encourage planning a whole release) and for the kind of team I am currently working in and enjoy working in, I'm interested in those variants.

One more thing about using Sorcery in a Rails project

One more thing about using Sorcery in a Rails project. I did mention that to make unit tests work, I had to stub out "current_user" in every single controller:

  Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  end

Even in Cucumber scenarios that do not use one of these steps that stub out current_user, I found that I had to explicitly unstub the method. I did not investigate why this is the case; I'm guessing that Cucumber would have to reload the whole environment between running each scenario, which would take too long. Perhaps there's a better way for Sorcery to provide "current user" functionality without patching each controller, or perhaps there's a better way for me to test mocking logged in users. In any case, this works, explicitly adding a step to make sure that the mocked-up logged-in user is no longer stubbed in:

  Given /^I am not logged in$/ do
    begin
      VisitsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    begin
      DoctorsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    // etc...
  end

Pennants Afghan pattern

I finished a major knitting project and wrote up the pattern for it as a PDF, licensed under Creative Commons.
We've agreed to share the blanket at home, where it will live on the couch:

I'll be posting the pattern to Ravelry too, several commenters there have asked for it based on seeing my project photos.

Given I'm a wizard

I'm working on a Rails Web site (ShareTheVisit) and using a gem called Sorcery. I like the idea behind Sorcery, as a library that offloads the common features around picking passwords, activating accounts and so on. The Railscast on Sorcery really helped too. However, I've run into a few snags.

The major snag is the automation that Sorcery does on the object you designate for the user object. For example, if you tell Sorcery that a "User" instance is what it logs in, then the User account needs a password according to Sorcery. If you also tell Sorcery that you wish to use the email activation feature, then it does its magic every time a User object is created.

What's the problem with this? Our use case required me to create some information around a user who hadn't joined the site yet. E.g. I'm inviting my doctor to the site, and when she joins, I need her account associated with mine. For better or for worse, I decided to do this by creating a model for the invited doctor, but it couldn't be a User instance because that needs a password which we're not going to go choosing for the doctor, and even if we let the doctor choose the password later, creating the User account too early makes the email activation happy too early. So instead of creating a User instance when the doctor is invited, I created an Account model for the doctor. When the doctor comes to the site and chooses a password, then I create the User instance, associated it with the Account, and the activation email is sent by Sorcery.

So far so good, but the email activation is still a problem around invitations. If somebody comes to the site and registers ad-hoc, we want Sorcery to send an email before activating the account, so that we are certain that email will work for resetting passwords and such. But if somebody clicks on a link in an invitation they received in email, doesn't that validate the email address even more smoothly? So I'd like to skip the activation-required email depending on whether the registration form came with a token indicating the User was invited via a working email address. No luck doing this directly, Sorcery does its magic whether I'd like it to or not, when the User account is created. I hacked around it like this, setting the email initially to something bogus that will fail to be delivered, before resetting the email after the save event.

  def register(account, params)
    user = User.new(params[:user])
    user.account = account
    if ValidEmailToken.exists?(:email => account.email, :token => params[:token])
      user.email = "#{params[:token]}+temporary@example.com"  #Temporary -- make the "activation required" email fail to be sent
      if user.save
        user.activate!
        auto_login(user)
        user.email = account.email
        user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    else
      user.email = account.email
      if user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    end
    return user
  end

Ugh. Even uglier, I had to put this in ApplicationHelper rather than in either the User or Account model as I would have liked. Sorcery only makes auto_login available to controllers (and ApplicationHelper is a mixin to controllers). I'm going to have to totally refactor this when I figure out how. I'd like to reunite the User and Account objects or at least reduce the redundancy -- because of the path by which I got this working, an Account has an email address when a new person is invited by an existing user, and a User has an email field too because that's what Sorcery needs.

Another snag came in automated functional testing. I'm using Cucumber and Capybara. I write tests like this:

 Scenario: Creating a visit with a new patient but no contact info
    Given I'm a doctor
    When I am attempting to create a new visit
    And I can add a patient
    And I click Create Visit
    Then I should see an error with "Need contact information"

To make this work I need to write the code behind each step, including "Given I'm a doctor".

 Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  
    @doctor = Doctor.create!(:first_name => "Gregory", :last_name => "House", :degree =>"MD", :user => @user)
  end

This shows how I had to stub out the magic Sorcery method "current_user" in each controller, because Sorcery does something like (I'm guessing, and I'm too lazy to read the Sorcery code to confirm) add methods to each actual controller. I definitely tried stubbing out "current_user" in the ApplicationController, which each other controller extends, but that just did not work.

I hope these tips help somebody else make Sorcery work for them, or to choose something else; or that somebody can let me know if I should be doing something different. I'm coding by myself these days so when I ask myself, "Self, is this a good idea?" I don't usually get a very good answer. It's quite probable I'm overlooking some decently obvious improvements to my situation.

To fully unpack the title of this blog post, "Given I'm a wizard"? The keyword "Given" is how Cucumber, in its domain-specific language, expresses pre-requisites, as in the example above. Obviously I'm a wizard if I'm using Sorcery, plus "In the basement rolling dice... I'm a wizard!" And yes, in the D&D campaign I'm currently playing in, I'm a sorcerer.