Bleak House

Bleak House by Charles Dickens
My rating: 5 of 5 stars

I love Bleak House.  Dickens uses incredibly lush and complex metaphorical descriptions for London, estate houses, and especially, the Chancery Court.  What makes this novel one of my favourite Dickens books is the sweet and good unreliable narrator, Esther Summerson.  She is not perfect (which is even better than perfect).  Her imperfections lie in her lack of self-confidence and too-trusting nature.  Even as she tells in the nicest and most trusting tone about some other character's actions, the reader can tell that other character's truer nature.  And eventually Esther comes around to the reader's opinion, saddened if the other character turns out to be less than her ideal.


View all my reviews

Two meetings was I invited to, 
And sorry I could not both attend, 
at two o'clock, what could I do,
the time was firm for each, I knew,
so I composed regrets to send.

I chose the meeting Paul had called 
because my thoughts were needed there,
and progress on V2 was stalled 
because the techs were so enthralled
with a framework with no market share.

And while Paul's engineers discussed
Their plans for building on Hadoop, 
And writing all their code in Rust, 
And if the crowdsource trend is bust;
I thought about that other group.

I'll have to read the notes they took,
No matter how obtuse or dense,
and at their data models look,
and find a time I can rebook,
If I am to make a difference.

.... apologies to Robert Frost.

Modern Web architecture, cookie-cutter, view from the trenches

Web architecture has come together quite nicely of late.  The components work together better than ever before, and it's easier to put together a great Web site or service -- putting more emphasis on the design and user experience, of course, which remain important and hard.

This list of steps is an overview of the cookie cutter approach.  It works for most new Web sites and services.  Many people I've worked with believe they have something different, a special wonderful unique thing that is not right for the cookie cutter approach, and to they extent that they're right, their differences should be minimized (focus on making your value differentiated, not your architecture).  So, if you're getting started on a new site or even a new service within a service, try to put off being special and different as long and as far as you can.  To minimize early development costs while remaining flexible and scalable (new hires as well as increased usage), try to make the default choice for every step below until it's proven you need to be different.

I came up with this list by using these components myself in situations both early and at scale, by talking with lots of other developers about what actually works and is easy, and by keeping abreast of developments by following a few blogs and twitterers I admire.

1. Pick a modern Web framework

The popular favourites are Python and Django, and Ruby on Rails.  If you are or have a technical person already, personal preference is a fine way to make this choice.  These both have data abstractions, views, templates, lots of inherent flexibility, and a rich set of free extensions (e.g. Ruby 'gems').  If you're thinking about node.js, see "Qualms" below.


2. Pick a data store

Hosted MySql or PostgreSQL both work great for a long time and a lot of scaling.  Use hosted databases for online test, staging and production and either the same database or SQLite for development and local testing.  It would be fine to just go with whatever is cheapest and easiest, which might mean making this decision together with "pick a hosting service" (E.g. Heroku uses PostgreSQL by default).  NoSQL solutions go under "Qualms" below.


3. Pick a hosting service

Look.  Just don't run servers yourself, OK?  Not until you're the size where your hosting costs annually are approaching the annual cost (roughly twice salary) of a dev/ops employee.  Don't use EC2 either, that's still a lot of manual work.  Startup efforts don't have the manpower or processes to handle an early new customer and a vulnerability disclosure somewhere in the Web stack, in the same week.  So leave the operation and upkeep of the Web stack up to the hosting service. I like Heroku.


4. Design and implement landing page

This is where you'll need to pick a CSS layout library like Skeleton.  Use this library's columns and rows to lay out your pages.


5. Plug in a user login module if you need login

Either pick one that supports passwords, email verification, forgetting passwords etc, or pick one that supports using Facebook, Twitter, Google, etc identities.  Plugging this in should be very quick if you're still on the cookie-cutter path.  Bootstrappers can configure their service to send those validation and forgotten-password emails via a free GMail account initially.  Django comes with authentication, whereas for Rails I've used both AuthLogic and Sorcery.


6. Add JQuery to your Web site if you didn't already do it for the login feature

A combination of other Javascript libraries could be used to replace JQuery, but JQuery does both common widgets and functions (like handsome buttons) and AJAX -- the AJAX part is necessary for accessing data from your service in the next two steps.


7. Start with some kind of site template 


You probably need Web page headers, footers, and common colours and fonts.  A combination of Web framework page templates, including partial pages (e.g. the Web page footer is commonly a partial Web page that is included on every full page in your site) and CSS templates solve this.


8. Work out your core data model, using your Web framework and migrations

Your core data model is that special wonderful thing you're building.  It could be real estate listings, restaurant reviews, task lists or BBQ recipes.  Frame it as objects and collections.


9. Expose your data as a Web API with JSON, and show in the Web page with AJAX. 

Read up on how your chosen Web framework supports REST and apply it to your data model.  Read up on how your chosen Web framework supports AJAX.  Often the way to expose the data model is a direct extension of your data model. However, sometimes it's driven more by the view.  If the UI always shows collections of comments, you don't need a way to get just a single comment; you jump straight to "GET all the comments for this item."  Use JQuery or another Javascript library to make those calls and load data dynamically into pages.

This step and the previous step will happen over and over again as you add functionality.

Additional steps

• While in development, use free email; soon move to a hosted email provider.

To begin with use your Web framework's default email manager (ActionMailer in Rails), and just use any IMAP account to send mail, such as a free GMail account.  This is one of the things you'll need to upgrade sooner rather than later, by moving to a paid, hosted email provider.  Sendgrid is working OK for me and took less than half a day to sign up, set up and switch all test/demo/production servers to.  Sending SMS is similar -- in addition to choosing a SMS service you'll need to plug in a module.


• When or if you need native mobile or 3rd party app support, firm up your Web API

It's pretty cheap to get started with mobile functionality via HTTP/HTML/AJAX, at least for prototype, demo or even MVP.  There are probably cases where a native app is needed right away, but that's not too common.   The mobile client can use the same Web API that the javascript "client" uses, only it now becomes more important for that Web API to be stable, because a native mobile app or 3rd party app won't be updated at the same time as the Web site and API are updated.


• When or if you need to do any authentication with other sites or services, use OAuth.

OAuth comes into play for two situations: to authenticate users to another site such as Twitter, but also to authenticate apps or other sites that get special permissions on your service.  Rather than cook up your own special key logic, just use OAuth, plugging in one of the libraries that already exist.


• While you move along, let your Web framework help you as much as possible.

Django and Rails both have a ton of useful stuff.  The biggest one in my view is their database migrations assistance.  Bundler, in Rails, is key to looking after all the gems you'll have after the above steps. Try to also be familiar with the scripts and scaffolds that allow one to build new functionality fast in the completely cookie-cutter format. 

• While you move along, start to add automated tests

I add these right at the beginning.  Selenium, and capybara let me automatically load Web pages from my site, fill in forms, press buttons, and do simple tests on the results. RSpec and its mocking features are critical both for unit tests and for mocking out dependencies.  I run these myself and also, as soon as there are more than 1 developer, on a continuous integration server.  I'm currently trying out Semaphore as my first hosted integration server but would still go to the trouble of running Jenkins on EC2 if Semaphore doesn't work out.

Qualms

Node.js has great promise to unify Web programming under one language, bridging the divide between Web frontend javascript programmers and Web backend programmers using Ruby, Python or other.  A lot of people I respect  for being simultaneously practical, realistic and visionary, do like Node.js.  My qualm is that there is no standard framework to use with node.js -- the level of services provided by Node.js is more primitive than that of Django or Rails, and thus additional components are needed.

NoSQL is very powerful and often appropriate in large-scale Web sites and services.  I've used it before.  However, I have qualms suggesting a NoSQL solution at the outset, because the tools to hook frameworks and NoSQL together, as well as the tools to manage NoSQL systems, are not the default tools most familiar to Web developers, and not the most tested and evolved.  That said, a startup with experience using NoSQL might, without my labeling them crazy, use hosted Mongo, Redis or CouchDB, provided they still abstract away the store from the Web framework's data model.  In the long run this is something a hugely successful system probably needs to scale.   A combination of SQL and NoSQL is especially powerful for large or complex services but if you're at the "pick a data store" step you're not there yet.

Summary

The subject matter or domain used to matter a lot when a startup began to build a Web site or service.  These days, it doesn't seem to matter as much -- most startups I've seen can benefit from following most of these steps.  It's a real benefit to keep the startup's special thing limited and cordoned off (think of it as making your specialness more concentrated).  It's easier to hire and onboard people, as well as to benefit from community advances, when the overall Web architecture is cookie cutter.

Girls in games

The cultural gestalt these days seems to include more talk about women and video-games than ever before.  Suddenly, women are 47% of video-gamers and game designers (and game advertisers) are still figuring out how to deal with that.  If you haven't seen any of Anita Sarkeesian's videos before, they're really good.  Her concepts help me understand a bunch of common patterns and gave me names to more easily identify them (the "smurfette effect" for example).

Anita's concepts have not ruined my enjoyment of video games, but I may be getting more selective.  We have a PS2 and have been playing some old games.  One of them is Okami which is an outstanding game in many ways: playable, artistic, incredibly rich and creative.  In the game, the player's character is a wolf who begins a quest to regain divine powers the character once had, and rid the world of demons and darkness so it can bloom and the sun can shine again. When the world blooms it's simply gorgeous; all the art is painterly and beautiful.

Although the wolf fights, and is the best fighter in the world, its gender is ambiguous. My son thought the wolf was a 'he', missing the repeated interactions when the wolf meets other minor gods and they always call the wolf "Amaterasu, mother of us all".  Throughout the world there are reasonable ratios of male and female characters, adult and child and often the female is more powerful than the corresponding male (in the main city, the empress is clearly more important than the emperor).  Both male and female characters are killed off, not just the females.  Females need rescuing more than males but not exclusively.  So there's more balance than normal.

A few things grate only a tiny bit; the main non-player character who fights with a sword is alcoholic, lazy, scared and male, and goes off to save his brave, supportive lady love.  The character who seems good but is secretly taken over by a demon is called "busty babe" over and over by the pixie (and she is pictured as having unreal boobs), and there are scenes about this 2-inch tall male pixie trying to literally get into her shirt.  All in all, did not make me love the game any less -- the overall balance is so pleasing that an annoyingly-stereotyped interaction could be enjoyed as mild low humour.

And then there's this kind of thing.  *sigh*  I thought the Internets were supposed to know all about me by now and target ads right at me.  Clearly the Internets know I'm a gamer, but can't they tell I'm female?

Commentroversy, plus owls are the trendiest fowl.

Commentroversy is a five-year-old word according to Urban Dictionary, but it was new to me, when I saw it in the comment section of Yarn Harlot's latest blog post.  And the storm in a teacup that has so many commentators making puns about stitches and needles? The USOC asked Ravelry not to use the term "Ravelympics" to describe the event in which knitters set ambitious knitting goals during the real Olympics. Read the Yarn Harlot or Gawker, I think the shitstorm is funnier than the issue itself and don't have much else to say.

In unrelated news, I bought this t-shirt because it made me laugh.  But then I had to try to explain it.  Does it have something to do with YOLO being an acronym for "You Only Live Once"? I do not know.  But I did have a suspicion that owls are an arty crafty Thing.  And I confirmed they are a Thing, without even counting macrame owls.  You thought that ducks and chickens were overdone on home decorations?   Owls are the trendiest fowl by far.  In my totally scientific study based on Etsy search totals, here is the order of popularity of animals in kitsch:

Bird: 294,000
Butterfly: 144,000
Cat*: 118,000
Owl: 112,000
Dog*: 110,000
Fish: 67,000
Bear: 63,000
Horse: 56,000
Rabbit: 49,000
Mouse**: 47,000
Monkey: 43,000
Elephant: 42,000
Bee: 32,000
Deer: 25,000
Frog: 25,000
Ladybug: 24,000
Turtle: 23,000
Duck: 22,000
Lion: 20,000
Turkey: 19,000
Pig: 18,000
Fox: 17,000
Chicken: 17,000
Cow: 17,000
Sheep: 16,000
Sparrow: 15,000
Wolf: 13,000
Eagle: 10,000
Crab: 8,800
Panda***: 8,200
Parrot: 6,900
Beetle: 5,800
Lizard: 5,700

Rat: 4,400
Gorilla****: 1,200

* I subtracted the hits for pet accessories, because there's handmade dog and cat collars, beds, and toys, which don't count as Cute Handmade [Animal] Themed Tchotchkes.  If you just count housewares, for example, owls beat cats.


** Turns out, half the mouses on Etsy are Minnie.


*** You think you're cute, panda?  You're small-time, niche, pissant cute.  You can't even reach the tail of a stuffed owl.  I mean, really, you got beat out by crabs.  Go home and eat shoots and leaves.


**** Disturbing.


***** Tiger and Zebra were omitted due to the hard-to-control-for number of results with tiger eye gems, tiger stripe and zebra stripe fabrics. ***** <-- self-referential footnote

Better software development estimates

Is it possible to estimate software tasks better?  In certain circumstances, my experience tells me yes!  It's not just a wish (/me waves to commenter on last post).

First, to establish what kind of estimates I'm talking about.  It's not possible to estimate anything past two months, in my experience.  When I was working at Microsoft and heard lots of estimates from other teams, I realized that "This software product will take us two years" really meant "We have no idea and we will never finish a product that recognizably matches our plan".  A one year estimate turn into a somewhat recognizable outcome in two or three years.  Even a six month estimate, while it might turn into a year-long project fairly reliably, ends up finishing quite different tasks than the planned tasks which originally led to the six month estimate.

Other things that make a difference in estimation accuracy:
 * What language is being used?  Snags can be a bigger time dilator with a compiled or lower-level language like C, whereas hitting a snag in Python might not throw out the estimate that much.
 * Is something new being integrated? Any time a task involves compiling and linking a new library, or adding a new Ruby gem, I know the estimate is weaker.
 * How routine is it? Adding a new page to a Ruby on Rails project can be pretty damn predictable.
 * How many people are involved?  Estimating a one-person task is way more accurate than estimating a team project.  Even a one-person task that has an issue that requires another person's answer is less accurate than the task that can be done by one person independently.
 * How many other estimated tasks are dependencies?  If you have a chain of tasks to estimate, a change in one task can throw out the others.

And then there's bias.  Have you ever noticed that some developers always estimate high and some always estimate low?  It's intriguing, because even though they're consistently wrong, they are consistently wrong in the same direction.  That means they are actually giving management good information, if management knows their biases.  I once managed one optimist and one pessimist in the same team for over two years.  The optimist always estimated about 1/3 of his actual time required. The pessimist always estimated about three times the actual time required.  I would assign what looked like three months of work to the pessimist, and what looked like a week and a half of work to the optimist, and they would finish around the same time.

One of the things I really do love about Agile (and Pivotal Tracker reifies this) is how it understands the above points.
 * Task length? Agile encourages developers to break work down into smaller tasks (no three-week tasks in any agile process I've ever seen)
 * How many people are involved?  Agile encourages issues to be resolved before the estimate is even made, and it is designed for single-person tasks.
 *  How many unfinished dependencies are there?  Agile encourages planning with a shorter horizon, so the chain of dependencies is usually reduced.
 * Consistent bias?  Agile tracks velocity, not accuracy, so a consistent bias is simply a constant part of a consistent velocity.

With all that, one of the main things that intrigues me, and this is what I'm unpacking from my previous post, is whether better feedback would help developer estimates get even better than Agile already makes them.  Agile does not measure time spent so it doesn't give developers feedback that would allow them to either fix a consistent overall bias, or to start to recognize tasks that need to be estimated a little higher.

Scrum thinks people are stupid

One thing I both love and hate about the hardcore agile process, particularly the Scrum process, is that it tends to work around participants' biases, weaknesses, and lack of trust, rather than try to fix them. Scrum may not even leave room for improvements in trust or reliability. Some examples:

Developers' estimation biases are worked around, not fixed
Because developers have biases in estimating tasks (biases that are consistent over time and on average), scrum "velocity" measures estimates against results.  It doesn't attempt to fix estimates, e.g. by showing a developer the difference between their estimates and time spent.  In fact, velocity doesn't measure time spent at all, and lumps all developers in a team, over-estimators and under-estimators together, in one velocity measurement.

This is probably for the best -- it's one simple measure that's remarkably consistent.  Still, I wonder if it wouldn't be more useful in the long run to learn to estimate better.  I've never seen a really good estimation feedback loop in the software development context, but wouldn't it be neat to try?

Product owners' changes are either completely allowed or disallowed
Traditionally engineering teams have to train product owners not to change the product plans all the time.  This involves frequent team arguments about product plan changes.  Instead, scrum tries to carve out one small space where product changes are forbidden, and allow all other changes without argument.  In the current interval, the product owner cannot make any changes or the current plans are all tossed out and the estimation process is restarted, a consequence severe enough to effectively forbid any small changes.

This rule does keep product owners off the developers' backs.  If the product owner is thinking of a change to this week's plans, the consequences of an "abnormal sprint termination" probably stop them.  If the product owner is thinking of a change to next week's plans, the upcoming sprint planning meeting is where they'll discuss it.  Either way, the product owner does not walk up to a developer and say "Hey!  I've got a great idea!"

If keeping the product owner off the developer's back sounds like a really good idea to you, well you probably haven't worked with trusted, experienced product owners.  And if the team has processes that reify the distrust, then there's less chance to build trust.

Demos help people who can't analyse abstract plans
Weekly demos are a course correction mechanism. In order for anything to be marked done, it must be demoed.  And when a feature is demoed, the product owner can see the practical consequences better than they could when the feature was designed.  Now the product owner is able to immediately add things to the backlog, which might get done in the next iteration, and this is good.  Iterative design for the win.

The practical consequence of this appears to be less specification and planning work, which is good (avoid overplanning), but taken to the point where the product owners do not feel any particular pressure to understand and analyze the design.  Instead of sitting in front of the wireframes and thinking it through -- "What happens if I press this if it were implemented?  If this were a real system and the item had been deleted, what would the user need to see? " It's a difficult what-if skill and takes practice.   It doesn't make design any less iterative!  Instead, it moves the iterations into the design phase rather than the costly implementation phase.

Frequent demos, and the scrum rules which allow for any changes in the backlog, seem to remove some of the need to develop abstract design skills.  That makes me a little sad.  Still, frequent demos and iterations are a tool I'd use in any software development process.

Optimizing considered harmful
In hardcore scrum, developers are practically forbidden from doing any changes now that would make future work easier.  It's discouraged, and the way the system tracks tasks makes it unrewarding to do.

Let's say that today my task is to create a feature for users to delete items. There's also a story in the backlog or icebox for undeleting items (e.g. finding them in a trash folder and returning them).  Well, the way success is structured in scrum, I estimate the time to delete items at the beginning of the period where I do that work, and it makes everybody happier if I figure out how to delete items without much work, and we make more features fit in this period.  It doesn't help me now to estimate high in order to prepare for the 'undelete' feature.  It doesn't help me in the future either -- when we get to the scrum meeting where we estimate the  'undelete' feature, it might not even be me doing the feature.  (In theory, developers are supposed to be interchangeable in agile/scrum).  Even if it is me,  it's no big deal to have to do the undelete by rewriting the way delete worked -- I just build that into my estimates for this period.

There's no overall project schedule view that would have showed the value for doing these two features together and doing delete right the first time.  There are other ways where optimizing is actively discouraged:

• Literally, optimizing speed or resource use is discouraged.  Functional stories only say that the feature has to work.  Other stories in the future might or might not say that the feature has to work in 2 seconds.  Security might also be an afterthought!
• Optimizing by expertise is discouraged.  Everybody in the team is supposed to be interchangeable.  If all of this week's features are GUI features, then everybody does GUI work. If all of the next week's features are security features, then everybody does security work. 
• Optimizing the sequence of work by engineering constraints is discouraged.  If it would be faster to do feature A after the more general feature B is completed, too bad.  If there's a pileup of work with dependencies, and it's slipping to the end of the backlog where the dependencies will start to slow down each task, too bad.  Only the priority sorting of the product owner is valid.

I've seen passionate arguments for "You Aren't Gonna Need It" (YAGNI), and they're right.  Engineers often optimize prematurely.  Engineers often predict what they think the user will need and can be wrong.   But none of the passionate arguments for YAGNI think that it can be used without a sense of balance, right?  So when scrum methodology encourages developers to always put off optional work, scrum puts its thumb on the balance.  It short-circuits the discussions of complex tradeoffs and simply says do it later. 

Summary

What's common among all these traits is an attitude in scrum not only that people are fallible, but they're routinely wrong, and that they can't be trusted to work together on complex issues for the greater good. That means that it works best in environments where that attitude is more true -- e.g. a contract development team, comprised of decent engineers but not particularly specialized, who don't know much about the end use and maybe don't care too much, working with product owners who aren't experienced in software design, are way too busy to write specs and discuss complex tradeoffs, and are always tempted to change things they shouldn't.  

Agile should be done differently when the engineers are specialized, care about what they're doing and what it will be used for, work closely with the product owners in the same business unit, and the product owners are smart and can be trusted to learn good software development practices.  Just how it should be done differently is an open question to me. There are agile process variants that take different attitudes (e.g. those that encourage planning a whole release) and for the kind of team I am currently working in and enjoy working in, I'm interested in those variants.

One more thing about using Sorcery in a Rails project

One more thing about using Sorcery in a Rails project. I did mention that to make unit tests work, I had to stub out "current_user" in every single controller:

  Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  end

Even in Cucumber scenarios that do not use one of these steps that stub out current_user, I found that I had to explicitly unstub the method. I did not investigate why this is the case; I'm guessing that Cucumber would have to reload the whole environment between running each scenario, which would take too long. Perhaps there's a better way for Sorcery to provide "current user" functionality without patching each controller, or perhaps there's a better way for me to test mocking logged in users. In any case, this works, explicitly adding a step to make sure that the mocked-up logged-in user is no longer stubbed in:

  Given /^I am not logged in$/ do
    begin
      VisitsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    begin
      DoctorsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    // etc...
  end

Pennants Afghan pattern

I finished a major knitting project and wrote up the pattern for it as a PDF, licensed under Creative Commons.
We've agreed to share the blanket at home, where it will live on the couch:

I'll be posting the pattern to Ravelry too, several commenters there have asked for it based on seeing my project photos.

Given I'm a wizard

I'm working on a Rails Web site (ShareTheVisit) and using a gem called Sorcery. I like the idea behind Sorcery, as a library that offloads the common features around picking passwords, activating accounts and so on. The Railscast on Sorcery really helped too. However, I've run into a few snags.

The major snag is the automation that Sorcery does on the object you designate for the user object. For example, if you tell Sorcery that a "User" instance is what it logs in, then the User account needs a password according to Sorcery. If you also tell Sorcery that you wish to use the email activation feature, then it does its magic every time a User object is created.

What's the problem with this? Our use case required me to create some information around a user who hadn't joined the site yet. E.g. I'm inviting my doctor to the site, and when she joins, I need her account associated with mine. For better or for worse, I decided to do this by creating a model for the invited doctor, but it couldn't be a User instance because that needs a password which we're not going to go choosing for the doctor, and even if we let the doctor choose the password later, creating the User account too early makes the email activation happy too early. So instead of creating a User instance when the doctor is invited, I created an Account model for the doctor. When the doctor comes to the site and chooses a password, then I create the User instance, associated it with the Account, and the activation email is sent by Sorcery.

So far so good, but the email activation is still a problem around invitations. If somebody comes to the site and registers ad-hoc, we want Sorcery to send an email before activating the account, so that we are certain that email will work for resetting passwords and such. But if somebody clicks on a link in an invitation they received in email, doesn't that validate the email address even more smoothly? So I'd like to skip the activation-required email depending on whether the registration form came with a token indicating the User was invited via a working email address. No luck doing this directly, Sorcery does its magic whether I'd like it to or not, when the User account is created. I hacked around it like this, setting the email initially to something bogus that will fail to be delivered, before resetting the email after the save event.

  def register(account, params)
    user = User.new(params[:user])
    user.account = account
    if ValidEmailToken.exists?(:email => account.email, :token => params[:token])
      user.email = "#{params[:token]}+temporary@example.com"  #Temporary -- make the "activation required" email fail to be sent
      if user.save
        user.activate!
        auto_login(user)
        user.email = account.email
        user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    else
      user.email = account.email
      if user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    end
    return user
  end

Ugh. Even uglier, I had to put this in ApplicationHelper rather than in either the User or Account model as I would have liked. Sorcery only makes auto_login available to controllers (and ApplicationHelper is a mixin to controllers). I'm going to have to totally refactor this when I figure out how. I'd like to reunite the User and Account objects or at least reduce the redundancy -- because of the path by which I got this working, an Account has an email address when a new person is invited by an existing user, and a User has an email field too because that's what Sorcery needs.

Another snag came in automated functional testing. I'm using Cucumber and Capybara. I write tests like this:

 Scenario: Creating a visit with a new patient but no contact info
    Given I'm a doctor
    When I am attempting to create a new visit
    And I can add a patient
    And I click Create Visit
    Then I should see an error with "Need contact information"

To make this work I need to write the code behind each step, including "Given I'm a doctor".

 Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  
    @doctor = Doctor.create!(:first_name => "Gregory", :last_name => "House", :degree =>"MD", :user => @user)
  end

This shows how I had to stub out the magic Sorcery method "current_user" in each controller, because Sorcery does something like (I'm guessing, and I'm too lazy to read the Sorcery code to confirm) add methods to each actual controller. I definitely tried stubbing out "current_user" in the ApplicationController, which each other controller extends, but that just did not work.

I hope these tips help somebody else make Sorcery work for them, or to choose something else; or that somebody can let me know if I should be doing something different. I'm coding by myself these days so when I ask myself, "Self, is this a good idea?" I don't usually get a very good answer. It's quite probable I'm overlooking some decently obvious improvements to my situation.

To fully unpack the title of this blog post, "Given I'm a wizard"? The keyword "Given" is how Cucumber, in its domain-specific language, expresses pre-requisites, as in the example above. Obviously I'm a wizard if I'm using Sorcery, plus "In the basement rolling dice... I'm a wizard!" And yes, in the D&D campaign I'm currently playing in, I'm a sorcerer.

Sarasa hat pattern

I put a bunch of work into a chart to knit myself a hat, so I decided to follow through and make it a pattern.

I was inspired by this fabric in a book of mine:

I knit it up and blocked it:

Added tassels:

And modeled it myself, too. All over Thanksgiving weekend.

Here's the pattern.

The Story of Snow White and the Seven Dwarves and R2-D2
As requested by a four-and-a-half-year-old: "Yeah, tell me that one.  Snow White.  And put R2-D2 in it. And R2-D2 has to use his cannon."


Once upon a time, in a land far, far away, lived a young princess named Snow White.  She was called Snow White because of her pale white skin.  Her mother was dead, and her father was the king.  One day her father remarried, and the new queen did not like Snow White.  After a while it was clear that Snow White was just in her way.  Snow White's best friend and trusty companion was R2-D2 the droid.

Then when Snow White was about sixteen, her father, the king, died suddenly.  He left his kingdom to his daughter, but the step-mother intervened and ran things.  She made evil choices that made the people of the kingdom work harder, and give more of their money to the queen, and go hungry.  Snow White did not like this.  Nor did R2-D2.

One day, R2-D2 was rolling along a castle corridor, when, with his fine droid sensors, he heard a conversation between the Evil Queen and a woodcutter.  The Evil Queen was saying:  "I can't take it any more.  Snow White is criticizing my treatment of the rabble, and some of my ministers are agreeing with her.  You must kill her when she's out in the forest today.  Say that wolves got her."

R2-D2 went immediately to Snow White, of course, and warned her.  He recommended that Snow White should escape the castle, to avoid being killed.  Meanwhile R2-D2 would call in his friend the Prince from another country, who would come and help them deal with the Evil Queen.  R2-D2 explained to Snow White how she could find the Seven Dwarves, who lived together and would hide her until it was safe to return to the castle.

Snow White went out into the forest earlier than usual and evaded the woodcutter. She found the Seven Dwarves' cottage, and asked for their help.  Of course they agreed and said she could stay.  The Seven Dwarves were Grumpy, Sleepy, Sneezy, Doc, Nosey, Geeky, and Skip.  Each day they went away to work in the mines, and came back in the evening.  Snow White started to help out by cleaning up after breakfast, and preparing a nice supper for when they came back, so the Seven Dwarves grew to love her indeed.

The Evil Queen knew that Snow White was now hiding, so she came up with a different plan.  She used magic to disguise herself as a poor old woman selling apples.  She put one special apple in her basket, just for Snow White; it had poison. She went around the area of the castle, asking if anybody had seen a pretty young girl, new in the area.  Eventually she found a village where some people had noticed a pretty young girl who had recently started living with the Seven Dwarves, and so she went to investigate.  She found Snow White, all alone in the middle of the day.

"Hello, my deary", she said.  "Would you like to buy some apples?"

"Oh, those look delicious," Snow White answered.  "But I have no money, I'm sorry."

"Well, you're such a pretty dear, " the Evil Witch disguised as a poor old woman said, "I'll give you this apple if you give me the pleasure of eating it.  An old woman like me doesn't enjoy life as much as you young folks, so I'll just take a break and watch you eat this apple."  And she handed Snow White the apple prepared with poison.  


Snow White took a bite of the apple and before she even finished chewing it, she swooned.  That means, she fell over, right onto the ground, as if asleep -- or dead.  The Evil Queen laughed and went back to the castle, convinced that Snow White was dead.

Meanwhile, don't forget, R2-D2 had sent a message to his friend, the Prince from another country.  It took a long time in those days for a message to get to the other country, and for the Prince to ride to Snow White's country.  The Prince was just arriving when R2-D2 heard the news the Evil Queen was spreading -- that Snow White was not just missing, but found dead, and that the Evil Queen was going to rule the country.

R2-D2 rolled out to meet the Prince.  When the two friends found each other on the road, R2-D2 led the Prince to the Seven Dwarves cottage.  The Seven Dwarves had just arrived, and were moaning sadly about poor Snow White.

With his sensors, R2-D2 could tell that Snow White was just breathing very slightly, only barely alive.  In droid speak (doot dweet peep brrrrr peep peep doot) he instructed the Prince to lift Snow White onto the table so that R2-D2 could use his gripper arm to open Snow White's mouth.  R2-D2 found pieces of apple in her mouth, because she hadn't even finished swallowing the apple, so he took them out.  Then he instructed the Prince in how to sit Snow White up to whack her back, to dislodge more pieces of apple.  Together, they got enough of the poison apple out of Snow White so that she could recover.

The next day, Snow White, the Prince, and R2-D2 headed to the castle.  Snow White took a bow and arrows just in case, and the Prince had his sword.  When they arrived at the castle, the gates were locked and the Evil Queen was instructing the guards not to let them in.

"She's not the real Snow White! She's an impostor! Keep her out of the castle!" she yelled to the guards.

"I am the real Snow White!" Snow White shouted back.  "You guards, you know me.  I grew up in this castle!"  And the did know her, so some of them put down their swords, and their bows and arrows.  The Evil Queen grabbed one of them and tried to threaten his life with her dagger, so he would obey her, so Snow White shot a warning arrow.  The Evil Queen had to hide from the arrows so she let the guards go.

The gate was still locked, so R2-D2 used his cannon to blast it open.  Boom!  The gate crashed open, and the threesome walked into the castle through the gate.  But the Evil Queen was waiting for them, and she tried to jump on Snow White and kill her with a dagger.  Luckily, the Prince was ready with his sword, and he swung it to protect Snow White, and cut off the Evil Queen's head.

Snow White thanked the Prince for saving her, although she said "You shouldn't have killed her.  We should have brought the Evil Queen to justice and put her in prison.".

"I know," said the Prince, "But it was an accident.  It was in defense."

"That's true," said Snow White.  "Say, you seem like a nice guy.  How about you stay here with me?  I'm going to rule this kingdom, and do a much better job than the Evil Queen, too.  If you give me good advice, and help me rule this kingdom, maybe we could get married. "

"That sounds like a good idea," said the Prince.

R2-D2 said "Peeeeep doot dweet-deet doot brrrrr peep!"

And they all lived happily ever after.

My third post on doing RESTish JSON APIs in Rails is about testing the API. I'm assuming the network API should remain stable, not have fields added or removed without consideration (and ideally, documentation), especially if there are independently-written clients that use the server's API. So normal Rails "functional" testing is not enough here.

"Functional" testing is in scare quotes because it's got a specific meaning to Rails, not a generally-accepted meaning to programmers. It is how Rails defaults encourage programmers to test the actions of their controller classes, which is to say the results of their Web request handling (e.g. in the Rails guide, especially the section on testing views).

It gets pretty tedious to assert all the things are in a JSON response that should be:

assert_response :success
  data = JSON.parse(@response.body)["data"]
  assert_not_nil @data["comments"]
  assert_equal 1, @data["comments"].length
  assert_equal "The User", @data["comments"][0]["display_name"]
  assert_equal "the comment text", @data["comments"][0]["comment_text"]

Worse, this only tests that things that are tested for are there, not that new things don't creep in. It's pretty easy to modify code in a way that passes what looked like reasonable functional tests, but breaks the client.

While working on improving our testing for API responses, I've found a few things are important to me.

• The tests should show what an entire response ought to look like (aids readability)
• The tests should flag things that aren't supposed to be there as bugs
• The test responses should be templates, so that variables can vary
• The output of a test should show the difference, so I don't have to wade through pages of desired output and pages of actual output

It took us a while (thanks to Tom, Steve and others at work) to put all the pieces together, and there are probably more improvements to be made, but I'm happy enough with what we came up with to blog about it. We use

• Cucumber to describe the request, the response status and the response body
• Custom cucumber step definitions to let the step define the entire response body as a Ruby Hash
• A custom template "Wildcard" object to let variables vary within the response body
• Rails feature to compare hashes and find diff, to produce readable output

So now here's what one of our API response tests looks like!

Feature: Commenting
  Background: 
    Given the following post exists:
      | id | 1234 |
    And the following comment exists:
      | id | 882281 |
      | post_id | 1234 |
      | comment_text | "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience." |
      | user_id | 101 |
  
  Scenario: Retrieve a comment 
    When I GET "/posts/1234/comments/882281" with a user session
    Then the response should have the following comment:
    """
      {
        "comment_text"=> "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience.",
        "created_at"=> Wildcard.new(String),
        "updated_at"=> Wildcard.new(String),
        "creator" => {
          user_id => 101,
          handle => "gertrude",
          display_name => "Queen Gertrude",
          number_comments => Wildcard.new(Fixnum)
        },
        "post_url" => Wildcard.new(String)
      }
    """

The Wildcard class looks like this:

class Wildcard
  def initialize(must_be_class=Object)
    @must_be_class = must_be_class
  end
  def ==(other)
    other.class == @must_be_class
  end
end

And the Cucumber step that implements "the response should have the following comment" looks like this:

Then /^the response should have the following ([^"]*):$/ do |element, string|
  the_hash = eval(string)
  response_data = JSON.parse(@response.body)["data"][element]
  assert_equal({}, response_data.diff(the_hash))
end

Note, what this means is that our API responses have "data" as the top element, with a "comment" element inside -- that part is asserted to be there, so that the contents of the comment can be compared with the Rails hash diff method. If the comment element in the response had an extra child, this would produce an error with the extra child in the diff. If the comment was missing a child, this would produce an error with the extra child in the diff. And if the comment has a child with a different value than what's in the template, this would produce an error with the returned value in the diff.

I realize this is not Behavior Driven Development (BDD). One can use Cucumber to do BDD, but the two are not inextricably linked. And our intent here is not to do BDD but to maintain a stable API. I think this works.

This is another post outlining "How I spent several days frustratingly getting something to work" so that "you don't have to". I think the theory behind these kinds of posts is the "you don't have to" part but the reality is "OMG I have to rant and boy it would help put this behind me if I could pretend there was some USEFUL lesson."

I already posted that I've been using Jenkins and RVM. The theory is great: Jenkins lets you set up lots of recurring jobs and have one dashboard to see that all of your software is basically working, while RVM allows you to choose a different Ruby version and gemset for each project that might need that.

The first Jenkins job I set up seemed to use the RVM Ruby-1.8.7 just fine, but when I tried to set up a second job that required Ruby-1.9.2, I never could get it working. First it tried to use the system ruby, which I blew away. Then I had hours tracking down references to the system ruby. The command "rvm use 1.9.2" appeared to work and chose the right rvm ruby directory. But even when it worked, when the very next command run by jenkins was "ruby -v", the machine either couldn't find the ruby install at all, or found the system one, but never the one just successfully activated by RVM. Even more frustrating, this worked at the command line, but not when running a job from Jenkins. (If anybody can explain how Jenkins runs jobs differently and what effect that has on context, I'd love to know -- having a model of what's going on would help.)

Finally I discovered "rvm-shell", which can force execution of a command within the context of what RVM set up. So now every line of the jenkins job script that uses ruby, bundle or rake, is wrapped in that:

rvm use 1.9.2@profiles
rvm-shell -c "ruby -v"
rvm-shell -c "bundle install"

rvm-shell -c "rake db:migrate"

# Finally, run your tests
rvm-shell -c "rake"

It worked. 56 is the magic number -- that is, build #56 was the first one to work on this project. Phew.

Many modern Web services or Web APIs use REST style with JSON request and response bodies. The first problem doing this with Rails is that Rails isn't entirely conducive to doing REST, which I already discussed, but the second problem is that Rails isn't very conducive to doing JSON in a way that keeps the API stable. As is normal with Rails, it gets the basics running really fast, but its default assumptions and "magic" utilities need to be bypassed for some purposes.

This seems to be the default approach -- just call Rails "as_json" or "to_json" methods on the model instance, which gets all of the model's database columns but none of the model's dynamic information. This example has an array of Message instances, and without any monkey-patching or overriding as_json, the effect of this will be to export all of the ActiveRecord fields of Message (but not other content such as attributes) as JSON elements. For example, if the Message model uses a "messages" table with a column called "title", this code would expose the value in that column as "title" in the JSON as well.


render :json => message_list.as_json


But if some of the database columns on the model are for internal use and not for the API, the API implementor needs to override the default behavior to include only the columns that are intended for the API:


# ... model for Message
def as_json(opts = {})
opts = {:only => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
super(opts)
end

#... controller for Message
render :json => message_list.as_json


(One could also use "except" instead of "only" as an option for as_json, but then if a new column gets added to the Message model, it would automatically be added to the API whether you want it or not).

It gets even more complicated if some of the model's information needs to be converted, combined, or otherwise manipulated:


# ... model for Message
def as_json(opts = {})
# change id to be called message_id because existing clients use message_id
out = { 'message_id' => id.to_s(&:as_json) }

opts = {:only => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
out.merge!(super(opts))
end

#... controller for Message
render :json => message_list.as_json


It starts to get ugly, and a lot of the ugliness lives in the Model, not the Controller... wait. Oops. In MVC, isn't this what the View is for?

Enter RJSON. This is a template type for Rails, by Tom Lieber. The Ruby hash (like a dictionary) is already very familiar to Ruby developers, so why not have the Ruby developer construct a hash in a template file. A RJSON template file named "messages.rjson" might look like this:


{
:messages => @messages.map do |msg|
{
:message_id => msg.id.to_s,
:title => msg.title,
:body => msg.body,
:thumbnail_url => msg.thumbnail_url,
:background_url => msg.background_url,
:action => msg.action,
:display_on_launch => msg.display_on_launch,
:ok_button_string => msg.ok_button_string,
:cancel_button_string => msg.cancel_button_string
}
}


And its controller looks like this, where @messages is the array placed in context for the view to use:


@messages = Message.find_all_by_user_id(user_id)
render :file => "messages"


This is simple, but even much more complicated examples remain readable. Partials can also be used; if some data structure gets re-used exactly the same way in more than one part of the API, that data structure can be a partial view. It's easy to see what the API should look like from the view, which is part of the point of views.

I recommend it, which is to say: by all means get your Web API prototyped quickly using as_json, but when you start to get serious about the API end-points and field names, documenting the API and keeping it stable, start using RJSON.

ETA: Tom pointed me to a recent gem published by somebody else with the same basic plan. It's also at github (same name, Jan de Poorter)

I've been setting up Jenkins to run Rails project unit and functional tests as builds.  This has been difficult, and 90% of the problems have been in getting different versions of various gems available.  RVM is supposed to help, as is Bundler.   I should have started out with a very methodical plan for giving access to the jenkins user, then having the jenkins user install rvm and create gemsets, then have the jenkins user install gems and run bundle install. Instead I got things working quickly with super-user permissions and then builds fail because it's hard to 'sudo' commands during an automated build.

Here's one problem running "bundle install" as the jenkins user:

Using rails (3.0.4)
Using right_http_connection (1.3.0) from git://github.com/rightscale/right_http_connection.git (at master) /usr/lib/ruby/1.8/open-uri.rb:32:in `initialize': Permission denied - right_http_connection-1.3.0.gem (Errno::EACCES)
from /usr/lib/ruby/1.8/open-uri.rb:32:in `open_uri_original_open'
from /usr/lib/ruby/1.8/open-uri.rb:32:in `open'
from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:73:in `write_package'
from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:38:in `build'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `chdir'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:559:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:58:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/rubygems_integration.rb:93:in `with_build_args'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:57:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:49:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:8:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/cli.rb:220:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `send'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor.rb:263:in `dispatch'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/base.rb:386:in `start'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/bin/bundle:13
from /usr/bin/bundle:19:in `load'
from /usr/bin/bundle:19



After reading up on similar but different errors via Google, I concluded this was probably a local filesystem permission error.  I tried granting the jenkins user various permissions in /usr/lib/ruby and /usr/local/rvm/gems -- one misleading clue was that the "right_http_connection-1.3.0.gem" file in /usr/lib/ruby/1.8/gems/cache was the only gem in that directory with a different user/group than the other gems.  Fixing that didn't fix the problem, nor did deleting that gem file.  I then realized that bundler puts gems in a different place: .bundle in the project directory.  Oh, that's owned by root -- great, fix that.  Oops, didn't work.  Finally I deleted .bundle and ran bundle install again -- as the jenkins user -- and it worked.  

(This is probably boring to most readers but it's a small payment for all the other bloggers' posts I've read that gave me clues in various debugging dead-ends. )

This week I was asked to review draft-ietf-hybi-thewebsocketprotocol-10, The Websocket Protocol, as it gets closer to being an IETF RFC. After I sent my review, I was prompted to look back and reminded myself that I wrote the first message to apps-review back when it had a separate mailing list, asking for the first review from the newly-formed team. This explains why I'm still reviewing IETF documents despite having no current IETF activity -- I feel I owe a few people a few reviews in return. 

Websockets itself also brought back the memories.  I approved the BOF, the "Birds of a Feather" meeting that organized the HyBi Working Group (WG), and approved the formation of the WG.  I felt it was very important that the Websocket protocol be developed in a venue where we had server developers and transport and security experts as well as browser developers. We also had to ensure that the protocol was considered quite separate from the Websocket API which is part of HTML5.  Much painful experience shows that while protocol libraries are hard enough to upgrade, the protocols stay around for long and are even harder to fix or replace, so backward compatibility, versioning and extensibility are crucial.

While IETF progress for Websocket protocol was, as almost everything at the IETF is, agonizingly slow, I was glad to see that the near-end result has pretty solid HTTP compliance, solid framing and transport text, and deals with security concerns that weren't even on the radar before the IETF effort. Personal kudos to Joe Hildebrand, Alexey Melnikov, Ian Fette, Ian Hickson, Adam Barth and Salvatore Loreto, and kudos to everybody else who contributed.

Resourceful != RESTful

Resourceful routing in Rails is certainly useful.  As is customary in Rails, you get a lot of stuff achieved with a very small amount of declarations.  And at first glance it looks like REST!  Yay!  Except... it will lead one slightly astray.  I've come across a few ways in which "resourceful" routing in Rails doesn't really follow REST principles, in ways that aren't merely theoretic but can affect intermediary behavior. 

Resourceful routing defines paths or routes for objects that the server developer would like to have manipulated with CRUD operations.  The developer can declare an 'invitation' to be a resource, and the routes to index all invitations, create a new invitation, download an invitation, update or delete the invitation, are automatically created.  But, problem the first:

• Resourceful routing uses POST to create a new resource.  PUT is defined as creating a new resource in HTTP, and intermediaries can use that information-- but only if PUT is used.  If POST is used, an intermediary can't tell that a new resources was created. 

All routes share common error handling.  Route not found? Return 404!  But Rails puts the method in as part of the route definition.  Thus, problem the second:

• Rails returns 404 Not Found if the client uses an unsupported method on a resource that exists.  So for example if I apply resourceful routing such that a resource can be downloaded with GET and updated with PUT, but don't define a POST variant route for that URL, then when the client tries to POST to that URL the server returns 404 because the route (with correct method) was not found.  In theory an intermediary could mark the resource as missing and delete its cached representation.  Instead, there's a perfectly good error to use in HTTP when a method is not supported on a URL, and that is 405 Method Not Allowed.

Rails has some magic to collect all the parameters on an incoming request and collect them in a params hash.  Thus, if my Web page takes a query parameter named "user_id", I just pull params["user_id"] and I've got it.  If my Web form sends the parameter inside the body instead, it's still there in params["user_id"]!  This lets server developers easily change their mind between having GET or POST used as the form submission method.  However, it creates two problems with intermediaries, linked to the way resources are named. 

• URL parameters are mixed with body parameters. This may not cause problems for a POST, where the response typically isn't cachable anyway.  But it's a bad choice in for GET requests, where the URL containing parameters affects caching, while other parameters are unseen by intermediaries.  

• Query parameters are treated the same as path elements.  Routes are defined as paths that can have parameters in them.  So if the routes file defines a route for "/v1/store/buy/:product/with_coins", the :product path element could be any string, and Rails will pass that string into the application just as if it were a URL query parameter.  However, caches are supposed to work differently if a URL has query parameters than if it does not, so treating them as the same is misleading the developer.

These kinds of things can be subtle to track down.  For example, it would be pretty hard to track down a bug where an intermediary was returning a cached response to a GET request when the Web server programmer had the response varying by parameters, and some client was sending those parameters in the body where the intermediary couldn't see them.  I guess the saving grace, if you call it that, is that real intermediary caching isn't as common as I had thought.  Even when intermediaries exist and could cache, Rails applications don't commonly seem to take advantage of that.

I was wondering if the term "guys" is becoming more gender-neutral over time. Have you ever referred to a group including some women as "guys"? I've used the term to refer to groups made up entirely of women, but only in the second-person plural sense in casual or sporty situations. E.g. I'd say "Are you guys ready" to a group of women I was about to go running with. I wouldn't say "I'm going running with the guys from work" if it was women from work because that would imply men. And I'd never ever use the "guy" for a woman or girl, in first, second or third person singular context. I might in a stretch say "I'm just one of the guys", but that's plural first person. Very strange but wiktionary seems to document this as common.

Even stranger, however, is the derivation of the word "guy". From a proper name to a derogatory term to a generic term. I found this on Online Etymology, but I'm going to reword the explanation in chronological order with some of the inferences filled in, because I had to read their explanation three times for it to make sense.

1. Guy Fawkes got his name from Old German word for "wood" or "warrior", or possibly Welsh for "lively" or French for "guide" -- but it was a standard boy's name at the time at any rate.
2. Guy Fawkes planned the failed Gunpowder Plot to blow up the House of Lords.
3. Thereafter, Guy Fawkes Day was celebrated with fireworks and bonfires, and an effigy of Guy Fawkes paraded through the streets being set before fire. The effigy would have been a straw man wearing cast-off clothing.
   
4. The name Guy would have been so associated with this effigy, that calling somebody a "guy" must have brought to mind a badly-dressed scarecrow figure at that time.
5. The term became more generic, from meaning "badly-dressed fellow" to meaning "ordinary man", over the course of just a generation.
   

Note that "guy" is definitely an American term, even though the Gunpowder Plot was in Britain. Guy Fawkes day was widely celebrated in US before the revolution, so the Guy effigy would have been a familiar figure in the US back when it came to mean "grotesquely or badly dressed".

I wonder if there's a term for when a derogatory word becomes unobjectionable, and whether this usually happens by being appropriated (the people the term refers to use the word proudly) or just by being watered down.

I learned about Medium Maximization behavior the other day and thought about how Scrum works. It's an interesting lens.

When people engage in medium maximization, they maximize a proxy for a real goal, rather than maximizing progress towards the real goal. The proxy or medium could be points on an exam as a proxy for understanding a topic, or air miles as a proxy for plane tickets. People do odd things for air miles that they wouldn't do for an equivalent direct reward. More details at http://faculty.chicagobooth.edu/christopher.hsee/vita/Papers/MediumMaximization.pdf - there are quite a few interesting side effects involved.

Applying to scrum

Scrum assigns story points to small accomplishable goals intended to eventually combine to reach a finished project. Since story points are not the same as the goal, story points are a medium, and we should expect participants to put more weight on accomplishing story points than on finishing the project, changing their behavior compared to when story points are not used.

I fully believe this behavior works as much for the managers and stakeholders in Scrum as for the engineers. In fact it might be more so: the only thing managers can measure is story points, not actual useful code, so managers are probably more satisfied by story point burndown than engineers are. Engineers have more visibility into what's being achieved towards a completed project, so that can have more influence on their choices, but I'm quite sure engineers are influenced too.

Predictions

A hypothesis like this can be tested by making predictions that can be tested and measured. I don't propose to actually study this myself, but making predictions and finding things that can be measured is fun.

In the studies done by Christopher Hsee and co-authors, participants had to accomplish tasks to get rewards. In the control set, the participants contemplated doing more or less work for different rewards. When a medium was introduced, and more points were disproportionally offered for more work, participants did much more work on average, even though the rewards that the points could be exchanged for were the same. Applying this to scrum, we should find that scrum participants do more work to burn down more story points, especially if the scale is non-linear. Imagine that management could somehow set the story points such that doing 40 hours of work accomplished 40 story points, but 50 hours of work accomplished 75 story points. Participants would likely work more, and this can probably be measured.

Another prediction comes from the studies that show that the behavior of maximizing the proxy points continues even if the primary reinforcer is removed. In software development, a team might continue working on story points even if the product plan had been cancelled, unless they were told to stop. This sounds unlikely and wrong, but imagine instead a project is rumoured to be canceled soon due to a possible merger. A manager might prefer their team make solid progress on story points even with that rumour flying around. If development teams show less loss of morale and effort when such rumours are flying around when using scrum, than not, this would be good evidence that the teams were maximizing story point burndown rather than project release progress.

The last prediction is based on the results of applying a linear medium -- for each chunk of work you get the same number of points -- to a non-linear reward. In one study, participants had a reward with decreasing rates of return offered. When a linear medium was offered, but the exchange rate for points to reward had the same decreasing rate of return, participants did much more work. In software engineering it's commonly observed that the last 20% of the work takes the last 80% of the time, so software projects are arguably non-linear in the same way the rewards in this study were. If story points offer a linear medium with respect to effort -- which in classic scrum they do -- we should see engineers happy to do more work even as the project gets into its depressing cleanup and bugfix phase. (See note below)

Joking Suggestions

(I'm only playing with this lens, not suggesting real changes to scrum processes. Much of the focus on Medium Maximization is on figuring out how to get the most effort out of participants. If the goal of scrum is to get more effort out of people then Medium Maximization is very relevant, but if the goal is agility, job satisfaction, schedule predictability, or product quality, it might be less relevant. )

If the goal of scrum is to maximize engineering effort, then Medium Maximization studies do suggest that scrum teams should *earn* story points rather than burning them down. Scrum tracking tools should award points to the whole team (working for the team is definitely stronger than working individual) and display the current point total prominently. The tools might display both the overall point accumulation, and the current scrum/sprint or rolling total of last X days.

Calibrating the way story points scale is very important. The allocation of story points should definitely not show a decreasing margin of return. Very large tasks should never have insufficient story points, else work stops earlier. If we assume that story points are still assigned by engineering estimates, we can still use processes to influence this:

1. Since we know people underestimate large chunks of work worse than they estimate small chunks of work, we should avoid having large chunks of work. Scrum often does this already by having a ceiling on story points. Some scrum practitioners assign no more than 5 story points: anything greater requires them to break down the story. That might not always have the intended effect -- in a meeting that's already taking too much time from too many people, the team might estimate the story at the maximum value rather than immediately have to reconsider the whole definition and breakdown of the story.
2. Some types of engineering effort are often overlooked -- merging code branches, testing, deploying -- which is bad, because fewer story points are estimated for the last bit of effort than that effort truly warranted. In order to avoid that trap, we could always make those chores explicit and have estimates based on past merges rather than let engineers re-estimate too optimistically. (Yes, engineers estimate the same type of task over-optimistically, over and over and over again). Alternatively, a team might add a minimum of X story points to any story that required merging, testing or deployment, and again that X value would be based on real history.
3. We might have the engineers give all their estimates for a story and always pick the largest estimate.
   

Caveats

Rather than write the above sections with lots of caveats, I saved my own criticisms for this section.

• I stated that using story points which estimate effort detaches work from the real goal of adding functionality to a software project. But I neglected to mention that part of scrum is tying story points to demos of releasable functionality. That presumably helps keep story points better moored to real progress.
  
• Scrum and assigning story points helps break down big projects. Breaking down big projects into small tasks and tracking those tasks is nearly ubiquitously practiced in software engineering, probably because it's a good idea. Enforcing a good idea through well-rehearsed process is probably a good idea too.
• Medium maximization is easier to measure if your goals and rewards are clear. In software engineering, the goals are often unclear: product designers don't know exactly how everything ought to work until they can start to see it work, and then make changes. Features get cut due to lack of time. So detaching work from shipping a "complete product" might be a *good* thing -- it stops engineers from doing work on features that might get cancelled next week, and stops them from putting effort into scalability or robustness or abstractions that might not be needed. And if you don't know your goals at all, then scrum at least keeps people from yelling at each other over an apparent lack of progress vs lack of specifications.

• On the OTHER other hand, I imagine medium maximization is stronger in the absence of clear goals and clear relationships between the medium and the goal. If you don't know if you plan to go to Vancouver or to London next year, and if the airline can always change how many points gets you a ticket to each destination, then you can't plan exactly how many air miles you're going to need and how to get those air miles at least cost. Unclear medium/goal relationships will make people try to accumulate more of the medium than they estimate they might need, in order not to fall short. When engineers really don't know where the project is going, scrum will at least keep them resolving story points, and thus preserve the appearance of forward motion.