Sarasa hat pattern

I put a bunch of work into a chart to knit myself a hat, so I decided to follow through and make it a pattern.

I was inspired by this fabric in a book of mine:

I knit it up and blocked it:

Added tassels:

And modeled it myself, too. All over Thanksgiving weekend.

Here's the pattern.

The Story of Snow White and the Seven Dwarves and R2-D2
As requested by a four-and-a-half-year-old: "Yeah, tell me that one.  Snow White.  And put R2-D2 in it. And R2-D2 has to use his cannon."


Once upon a time, in a land far, far away, lived a young princess named Snow White.  She was called Snow White because of her pale white skin.  Her mother was dead, and her father was the king.  One day her father remarried, and the new queen did not like Snow White.  After a while it was clear that Snow White was just in her way.  Snow White's best friend and trusty companion was R2-D2 the droid.

Then when Snow White was about sixteen, her father, the king, died suddenly.  He left his kingdom to his daughter, but the step-mother intervened and ran things.  She made evil choices that made the people of the kingdom work harder, and give more of their money to the queen, and go hungry.  Snow White did not like this.  Nor did R2-D2.

One day, R2-D2 was rolling along a castle corridor, when, with his fine droid sensors, he heard a conversation between the Evil Queen and a woodcutter.  The Evil Queen was saying:  "I can't take it any more.  Snow White is criticizing my treatment of the rabble, and some of my ministers are agreeing with her.  You must kill her when she's out in the forest today.  Say that wolves got her."

R2-D2 went immediately to Snow White, of course, and warned her.  He recommended that Snow White should escape the castle, to avoid being killed.  Meanwhile R2-D2 would call in his friend the Prince from another country, who would come and help them deal with the Evil Queen.  R2-D2 explained to Snow White how she could find the Seven Dwarves, who lived together and would hide her until it was safe to return to the castle.

Snow White went out into the forest earlier than usual and evaded the woodcutter. She found the Seven Dwarves' cottage, and asked for their help.  Of course they agreed and said she could stay.  The Seven Dwarves were Grumpy, Sleepy, Sneezy, Doc, Nosey, Geeky, and Skip.  Each day they went away to work in the mines, and came back in the evening.  Snow White started to help out by cleaning up after breakfast, and preparing a nice supper for when they came back, so the Seven Dwarves grew to love her indeed.

The Evil Queen knew that Snow White was now hiding, so she came up with a different plan.  She used magic to disguise herself as a poor old woman selling apples.  She put one special apple in her basket, just for Snow White; it had poison. She went around the area of the castle, asking if anybody had seen a pretty young girl, new in the area.  Eventually she found a village where some people had noticed a pretty young girl who had recently started living with the Seven Dwarves, and so she went to investigate.  She found Snow White, all alone in the middle of the day.

"Hello, my deary", she said.  "Would you like to buy some apples?"

"Oh, those look delicious," Snow White answered.  "But I have no money, I'm sorry."

"Well, you're such a pretty dear, " the Evil Witch disguised as a poor old woman said, "I'll give you this apple if you give me the pleasure of eating it.  An old woman like me doesn't enjoy life as much as you young folks, so I'll just take a break and watch you eat this apple."  And she handed Snow White the apple prepared with poison.  


Snow White took a bite of the apple and before she even finished chewing it, she swooned.  That means, she fell over, right onto the ground, as if asleep -- or dead.  The Evil Queen laughed and went back to the castle, convinced that Snow White was dead.

Meanwhile, don't forget, R2-D2 had sent a message to his friend, the Prince from another country.  It took a long time in those days for a message to get to the other country, and for the Prince to ride to Snow White's country.  The Prince was just arriving when R2-D2 heard the news the Evil Queen was spreading -- that Snow White was not just missing, but found dead, and that the Evil Queen was going to rule the country.

R2-D2 rolled out to meet the Prince.  When the two friends found each other on the road, R2-D2 led the Prince to the Seven Dwarves cottage.  The Seven Dwarves had just arrived, and were moaning sadly about poor Snow White.

With his sensors, R2-D2 could tell that Snow White was just breathing very slightly, only barely alive.  In droid speak (doot dweet peep brrrrr peep peep doot) he instructed the Prince to lift Snow White onto the table so that R2-D2 could use his gripper arm to open Snow White's mouth.  R2-D2 found pieces of apple in her mouth, because she hadn't even finished swallowing the apple, so he took them out.  Then he instructed the Prince in how to sit Snow White up to whack her back, to dislodge more pieces of apple.  Together, they got enough of the poison apple out of Snow White so that she could recover.

The next day, Snow White, the Prince, and R2-D2 headed to the castle.  Snow White took a bow and arrows just in case, and the Prince had his sword.  When they arrived at the castle, the gates were locked and the Evil Queen was instructing the guards not to let them in.

"She's not the real Snow White! She's an impostor! Keep her out of the castle!" she yelled to the guards.

"I am the real Snow White!" Snow White shouted back.  "You guards, you know me.  I grew up in this castle!"  And the did know her, so some of them put down their swords, and their bows and arrows.  The Evil Queen grabbed one of them and tried to threaten his life with her dagger, so he would obey her, so Snow White shot a warning arrow.  The Evil Queen had to hide from the arrows so she let the guards go.

The gate was still locked, so R2-D2 used his cannon to blast it open.  Boom!  The gate crashed open, and the threesome walked into the castle through the gate.  But the Evil Queen was waiting for them, and she tried to jump on Snow White and kill her with a dagger.  Luckily, the Prince was ready with his sword, and he swung it to protect Snow White, and cut off the Evil Queen's head.

Snow White thanked the Prince for saving her, although she said "You shouldn't have killed her.  We should have brought the Evil Queen to justice and put her in prison.".

"I know," said the Prince, "But it was an accident.  It was in defense."

"That's true," said Snow White.  "Say, you seem like a nice guy.  How about you stay here with me?  I'm going to rule this kingdom, and do a much better job than the Evil Queen, too.  If you give me good advice, and help me rule this kingdom, maybe we could get married. "

"That sounds like a good idea," said the Prince.

R2-D2 said "Peeeeep doot dweet-deet doot brrrrr peep!"

And they all lived happily ever after.

My third post on doing RESTish JSON APIs in Rails is about testing the API. I'm assuming the network API should remain stable, not have fields added or removed without consideration (and ideally, documentation), especially if there are independently-written clients that use the server's API. So normal Rails "functional" testing is not enough here.

"Functional" testing is in scare quotes because it's got a specific meaning to Rails, not a generally-accepted meaning to programmers. It is how Rails defaults encourage programmers to test the actions of their controller classes, which is to say the results of their Web request handling (e.g. in the Rails guide, especially the section on testing views).

It gets pretty tedious to assert all the things are in a JSON response that should be:

assert_response :success
  data = JSON.parse(@response.body)["data"]
  assert_not_nil @data["comments"]
  assert_equal 1, @data["comments"].length
  assert_equal "The User", @data["comments"][0]["display_name"]
  assert_equal "the comment text", @data["comments"][0]["comment_text"]

Worse, this only tests that things that are tested for are there, not that new things don't creep in. It's pretty easy to modify code in a way that passes what looked like reasonable functional tests, but breaks the client.

While working on improving our testing for API responses, I've found a few things are important to me.

• The tests should show what an entire response ought to look like (aids readability)
• The tests should flag things that aren't supposed to be there as bugs
• The test responses should be templates, so that variables can vary
• The output of a test should show the difference, so I don't have to wade through pages of desired output and pages of actual output

It took us a while (thanks to Tom, Steve and others at work) to put all the pieces together, and there are probably more improvements to be made, but I'm happy enough with what we came up with to blog about it. We use

• Cucumber to describe the request, the response status and the response body
• Custom cucumber step definitions to let the step define the entire response body as a Ruby Hash
• A custom template "Wildcard" object to let variables vary within the response body
• Rails feature to compare hashes and find diff, to produce readable output

So now here's what one of our API response tests looks like!

Feature: Commenting
  Background: 
    Given the following post exists:
      | id | 1234 |
    And the following comment exists:
      | id | 882281 |
      | post_id | 1234 |
      | comment_text | "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience." |
      | user_id | 101 |
  
  Scenario: Retrieve a comment 
    When I GET "/posts/1234/comments/882281" with a user session
    Then the response should have the following comment:
    """
      {
        "comment_text"=> "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience.",
        "created_at"=> Wildcard.new(String),
        "updated_at"=> Wildcard.new(String),
        "creator" => {
          user_id => 101,
          handle => "gertrude",
          display_name => "Queen Gertrude",
          number_comments => Wildcard.new(Fixnum)
        },
        "post_url" => Wildcard.new(String)
      }
    """

The Wildcard class looks like this:

class Wildcard
  def initialize(must_be_class=Object)
    @must_be_class = must_be_class
  end
  def ==(other)
    other.class == @must_be_class
  end
end

And the Cucumber step that implements "the response should have the following comment" looks like this:

Then /^the response should have the following ([^"]*):$/ do |element, string|
  the_hash = eval(string)
  response_data = JSON.parse(@response.body)["data"][element]
  assert_equal({}, response_data.diff(the_hash))
end

Note, what this means is that our API responses have "data" as the top element, with a "comment" element inside -- that part is asserted to be there, so that the contents of the comment can be compared with the Rails hash diff method. If the comment element in the response had an extra child, this would produce an error with the extra child in the diff. If the comment was missing a child, this would produce an error with the extra child in the diff. And if the comment has a child with a different value than what's in the template, this would produce an error with the returned value in the diff.

I realize this is not Behavior Driven Development (BDD). One can use Cucumber to do BDD, but the two are not inextricably linked. And our intent here is not to do BDD but to maintain a stable API. I think this works.

This is another post outlining "How I spent several days frustratingly getting something to work" so that "you don't have to". I think the theory behind these kinds of posts is the "you don't have to" part but the reality is "OMG I have to rant and boy it would help put this behind me if I could pretend there was some USEFUL lesson."

I already posted that I've been using Jenkins and RVM. The theory is great: Jenkins lets you set up lots of recurring jobs and have one dashboard to see that all of your software is basically working, while RVM allows you to choose a different Ruby version and gemset for each project that might need that.

The first Jenkins job I set up seemed to use the RVM Ruby-1.8.7 just fine, but when I tried to set up a second job that required Ruby-1.9.2, I never could get it working. First it tried to use the system ruby, which I blew away. Then I had hours tracking down references to the system ruby. The command "rvm use 1.9.2" appeared to work and chose the right rvm ruby directory. But even when it worked, when the very next command run by jenkins was "ruby -v", the machine either couldn't find the ruby install at all, or found the system one, but never the one just successfully activated by RVM. Even more frustrating, this worked at the command line, but not when running a job from Jenkins. (If anybody can explain how Jenkins runs jobs differently and what effect that has on context, I'd love to know -- having a model of what's going on would help.)

Finally I discovered "rvm-shell", which can force execution of a command within the context of what RVM set up. So now every line of the jenkins job script that uses ruby, bundle or rake, is wrapped in that:

rvm use 1.9.2@profiles
rvm-shell -c "ruby -v"
rvm-shell -c "bundle install"

rvm-shell -c "rake db:migrate"

# Finally, run your tests
rvm-shell -c "rake"

It worked. 56 is the magic number -- that is, build #56 was the first one to work on this project. Phew.

Many modern Web services or Web APIs use REST style with JSON request and response bodies. The first problem doing this with Rails is that Rails isn't entirely conducive to doing REST, which I already discussed, but the second problem is that Rails isn't very conducive to doing JSON in a way that keeps the API stable. As is normal with Rails, it gets the basics running really fast, but its default assumptions and "magic" utilities need to be bypassed for some purposes.

This seems to be the default approach -- just call Rails "as_json" or "to_json" methods on the model instance, which gets all of the model's database columns but none of the model's dynamic information. This example has an array of Message instances, and without any monkey-patching or overriding as_json, the effect of this will be to export all of the ActiveRecord fields of Message (but not other content such as attributes) as JSON elements. For example, if the Message model uses a "messages" table with a column called "title", this code would expose the value in that column as "title" in the JSON as well.


render :json => message_list.as_json


But if some of the database columns on the model are for internal use and not for the API, the API implementor needs to override the default behavior to include only the columns that are intended for the API:


# ... model for Message
def as_json(opts = {})
opts = {:only => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
super(opts)
end

#... controller for Message
render :json => message_list.as_json


(One could also use "except" instead of "only" as an option for as_json, but then if a new column gets added to the Message model, it would automatically be added to the API whether you want it or not).

It gets even more complicated if some of the model's information needs to be converted, combined, or otherwise manipulated:


# ... model for Message
def as_json(opts = {})
# change id to be called message_id because existing clients use message_id
out = { 'message_id' => id.to_s(&:as_json) }

opts = {:only => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
out.merge!(super(opts))
end

#... controller for Message
render :json => message_list.as_json


It starts to get ugly, and a lot of the ugliness lives in the Model, not the Controller... wait. Oops. In MVC, isn't this what the View is for?

Enter RJSON. This is a template type for Rails, by Tom Lieber. The Ruby hash (like a dictionary) is already very familiar to Ruby developers, so why not have the Ruby developer construct a hash in a template file. A RJSON template file named "messages.rjson" might look like this:


{
:messages => @messages.map do |msg|
{
:message_id => msg.id.to_s,
:title => msg.title,
:body => msg.body,
:thumbnail_url => msg.thumbnail_url,
:background_url => msg.background_url,
:action => msg.action,
:display_on_launch => msg.display_on_launch,
:ok_button_string => msg.ok_button_string,
:cancel_button_string => msg.cancel_button_string
}
}


And its controller looks like this, where @messages is the array placed in context for the view to use:


@messages = Message.find_all_by_user_id(user_id)
render :file => "messages"


This is simple, but even much more complicated examples remain readable. Partials can also be used; if some data structure gets re-used exactly the same way in more than one part of the API, that data structure can be a partial view. It's easy to see what the API should look like from the view, which is part of the point of views.

I recommend it, which is to say: by all means get your Web API prototyped quickly using as_json, but when you start to get serious about the API end-points and field names, documenting the API and keeping it stable, start using RJSON.

ETA: Tom pointed me to a recent gem published by somebody else with the same basic plan. It's also at github (same name, Jan de Poorter)

I've been setting up Jenkins to run Rails project unit and functional tests as builds.  This has been difficult, and 90% of the problems have been in getting different versions of various gems available.  RVM is supposed to help, as is Bundler.   I should have started out with a very methodical plan for giving access to the jenkins user, then having the jenkins user install rvm and create gemsets, then have the jenkins user install gems and run bundle install. Instead I got things working quickly with super-user permissions and then builds fail because it's hard to 'sudo' commands during an automated build.

Here's one problem running "bundle install" as the jenkins user:

Using rails (3.0.4)
Using right_http_connection (1.3.0) from git://github.com/rightscale/right_http_connection.git (at master) /usr/lib/ruby/1.8/open-uri.rb:32:in `initialize': Permission denied - right_http_connection-1.3.0.gem (Errno::EACCES)
from /usr/lib/ruby/1.8/open-uri.rb:32:in `open_uri_original_open'
from /usr/lib/ruby/1.8/open-uri.rb:32:in `open'
from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:73:in `write_package'
from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:38:in `build'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `chdir'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:559:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:58:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/rubygems_integration.rb:93:in `with_build_args'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:57:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:49:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:8:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/cli.rb:220:in `install'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `send'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `run'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor.rb:263:in `dispatch'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/base.rb:386:in `start'
from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/bin/bundle:13
from /usr/bin/bundle:19:in `load'
from /usr/bin/bundle:19



After reading up on similar but different errors via Google, I concluded this was probably a local filesystem permission error.  I tried granting the jenkins user various permissions in /usr/lib/ruby and /usr/local/rvm/gems -- one misleading clue was that the "right_http_connection-1.3.0.gem" file in /usr/lib/ruby/1.8/gems/cache was the only gem in that directory with a different user/group than the other gems.  Fixing that didn't fix the problem, nor did deleting that gem file.  I then realized that bundler puts gems in a different place: .bundle in the project directory.  Oh, that's owned by root -- great, fix that.  Oops, didn't work.  Finally I deleted .bundle and ran bundle install again -- as the jenkins user -- and it worked.  

(This is probably boring to most readers but it's a small payment for all the other bloggers' posts I've read that gave me clues in various debugging dead-ends. )

This week I was asked to review draft-ietf-hybi-thewebsocketprotocol-10, The Websocket Protocol, as it gets closer to being an IETF RFC. After I sent my review, I was prompted to look back and reminded myself that I wrote the first message to apps-review back when it had a separate mailing list, asking for the first review from the newly-formed team. This explains why I'm still reviewing IETF documents despite having no current IETF activity -- I feel I owe a few people a few reviews in return. 

Websockets itself also brought back the memories.  I approved the BOF, the "Birds of a Feather" meeting that organized the HyBi Working Group (WG), and approved the formation of the WG.  I felt it was very important that the Websocket protocol be developed in a venue where we had server developers and transport and security experts as well as browser developers. We also had to ensure that the protocol was considered quite separate from the Websocket API which is part of HTML5.  Much painful experience shows that while protocol libraries are hard enough to upgrade, the protocols stay around for long and are even harder to fix or replace, so backward compatibility, versioning and extensibility are crucial.

While IETF progress for Websocket protocol was, as almost everything at the IETF is, agonizingly slow, I was glad to see that the near-end result has pretty solid HTTP compliance, solid framing and transport text, and deals with security concerns that weren't even on the radar before the IETF effort. Personal kudos to Joe Hildebrand, Alexey Melnikov, Ian Fette, Ian Hickson, Adam Barth and Salvatore Loreto, and kudos to everybody else who contributed.

Resourceful != RESTful

Resourceful routing in Rails is certainly useful.  As is customary in Rails, you get a lot of stuff achieved with a very small amount of declarations.  And at first glance it looks like REST!  Yay!  Except... it will lead one slightly astray.  I've come across a few ways in which "resourceful" routing in Rails doesn't really follow REST principles, in ways that aren't merely theoretic but can affect intermediary behavior. 

Resourceful routing defines paths or routes for objects that the server developer would like to have manipulated with CRUD operations.  The developer can declare an 'invitation' to be a resource, and the routes to index all invitations, create a new invitation, download an invitation, update or delete the invitation, are automatically created.  But, problem the first:

• Resourceful routing uses POST to create a new resource.  PUT is defined as creating a new resource in HTTP, and intermediaries can use that information-- but only if PUT is used.  If POST is used, an intermediary can't tell that a new resources was created. 

All routes share common error handling.  Route not found? Return 404!  But Rails puts the method in as part of the route definition.  Thus, problem the second:

• Rails returns 404 Not Found if the client uses an unsupported method on a resource that exists.  So for example if I apply resourceful routing such that a resource can be downloaded with GET and updated with PUT, but don't define a POST variant route for that URL, then when the client tries to POST to that URL the server returns 404 because the route (with correct method) was not found.  In theory an intermediary could mark the resource as missing and delete its cached representation.  Instead, there's a perfectly good error to use in HTTP when a method is not supported on a URL, and that is 405 Method Not Allowed.

Rails has some magic to collect all the parameters on an incoming request and collect them in a params hash.  Thus, if my Web page takes a query parameter named "user_id", I just pull params["user_id"] and I've got it.  If my Web form sends the parameter inside the body instead, it's still there in params["user_id"]!  This lets server developers easily change their mind between having GET or POST used as the form submission method.  However, it creates two problems with intermediaries, linked to the way resources are named. 

• URL parameters are mixed with body parameters. This may not cause problems for a POST, where the response typically isn't cachable anyway.  But it's a bad choice in for GET requests, where the URL containing parameters affects caching, while other parameters are unseen by intermediaries.  

• Query parameters are treated the same as path elements.  Routes are defined as paths that can have parameters in them.  So if the routes file defines a route for "/v1/store/buy/:product/with_coins", the :product path element could be any string, and Rails will pass that string into the application just as if it were a URL query parameter.  However, caches are supposed to work differently if a URL has query parameters than if it does not, so treating them as the same is misleading the developer.

These kinds of things can be subtle to track down.  For example, it would be pretty hard to track down a bug where an intermediary was returning a cached response to a GET request when the Web server programmer had the response varying by parameters, and some client was sending those parameters in the body where the intermediary couldn't see them.  I guess the saving grace, if you call it that, is that real intermediary caching isn't as common as I had thought.  Even when intermediaries exist and could cache, Rails applications don't commonly seem to take advantage of that.

I was wondering if the term "guys" is becoming more gender-neutral over time. Have you ever referred to a group including some women as "guys"? I've used the term to refer to groups made up entirely of women, but only in the second-person plural sense in casual or sporty situations. E.g. I'd say "Are you guys ready" to a group of women I was about to go running with. I wouldn't say "I'm going running with the guys from work" if it was women from work because that would imply men. And I'd never ever use the "guy" for a woman or girl, in first, second or third person singular context. I might in a stretch say "I'm just one of the guys", but that's plural first person. Very strange but wiktionary seems to document this as common.

Even stranger, however, is the derivation of the word "guy". From a proper name to a derogatory term to a generic term. I found this on Online Etymology, but I'm going to reword the explanation in chronological order with some of the inferences filled in, because I had to read their explanation three times for it to make sense.

1. Guy Fawkes got his name from Old German word for "wood" or "warrior", or possibly Welsh for "lively" or French for "guide" -- but it was a standard boy's name at the time at any rate.
2. Guy Fawkes planned the failed Gunpowder Plot to blow up the House of Lords.
3. Thereafter, Guy Fawkes Day was celebrated with fireworks and bonfires, and an effigy of Guy Fawkes paraded through the streets being set before fire. The effigy would have been a straw man wearing cast-off clothing.
   
4. The name Guy would have been so associated with this effigy, that calling somebody a "guy" must have brought to mind a badly-dressed scarecrow figure at that time.
5. The term became more generic, from meaning "badly-dressed fellow" to meaning "ordinary man", over the course of just a generation.
   

Note that "guy" is definitely an American term, even though the Gunpowder Plot was in Britain. Guy Fawkes day was widely celebrated in US before the revolution, so the Guy effigy would have been a familiar figure in the US back when it came to mean "grotesquely or badly dressed".

I wonder if there's a term for when a derogatory word becomes unobjectionable, and whether this usually happens by being appropriated (the people the term refers to use the word proudly) or just by being watered down.

I learned about Medium Maximization behavior the other day and thought about how Scrum works. It's an interesting lens.

When people engage in medium maximization, they maximize a proxy for a real goal, rather than maximizing progress towards the real goal. The proxy or medium could be points on an exam as a proxy for understanding a topic, or air miles as a proxy for plane tickets. People do odd things for air miles that they wouldn't do for an equivalent direct reward. More details at http://faculty.chicagobooth.edu/christopher.hsee/vita/Papers/MediumMaximization.pdf - there are quite a few interesting side effects involved.

Applying to scrum

Scrum assigns story points to small accomplishable goals intended to eventually combine to reach a finished project. Since story points are not the same as the goal, story points are a medium, and we should expect participants to put more weight on accomplishing story points than on finishing the project, changing their behavior compared to when story points are not used.

I fully believe this behavior works as much for the managers and stakeholders in Scrum as for the engineers. In fact it might be more so: the only thing managers can measure is story points, not actual useful code, so managers are probably more satisfied by story point burndown than engineers are. Engineers have more visibility into what's being achieved towards a completed project, so that can have more influence on their choices, but I'm quite sure engineers are influenced too.

Predictions

A hypothesis like this can be tested by making predictions that can be tested and measured. I don't propose to actually study this myself, but making predictions and finding things that can be measured is fun.

In the studies done by Christopher Hsee and co-authors, participants had to accomplish tasks to get rewards. In the control set, the participants contemplated doing more or less work for different rewards. When a medium was introduced, and more points were disproportionally offered for more work, participants did much more work on average, even though the rewards that the points could be exchanged for were the same. Applying this to scrum, we should find that scrum participants do more work to burn down more story points, especially if the scale is non-linear. Imagine that management could somehow set the story points such that doing 40 hours of work accomplished 40 story points, but 50 hours of work accomplished 75 story points. Participants would likely work more, and this can probably be measured.

Another prediction comes from the studies that show that the behavior of maximizing the proxy points continues even if the primary reinforcer is removed. In software development, a team might continue working on story points even if the product plan had been cancelled, unless they were told to stop. This sounds unlikely and wrong, but imagine instead a project is rumoured to be canceled soon due to a possible merger. A manager might prefer their team make solid progress on story points even with that rumour flying around. If development teams show less loss of morale and effort when such rumours are flying around when using scrum, than not, this would be good evidence that the teams were maximizing story point burndown rather than project release progress.

The last prediction is based on the results of applying a linear medium -- for each chunk of work you get the same number of points -- to a non-linear reward. In one study, participants had a reward with decreasing rates of return offered. When a linear medium was offered, but the exchange rate for points to reward had the same decreasing rate of return, participants did much more work. In software engineering it's commonly observed that the last 20% of the work takes the last 80% of the time, so software projects are arguably non-linear in the same way the rewards in this study were. If story points offer a linear medium with respect to effort -- which in classic scrum they do -- we should see engineers happy to do more work even as the project gets into its depressing cleanup and bugfix phase. (See note below)

Joking Suggestions

(I'm only playing with this lens, not suggesting real changes to scrum processes. Much of the focus on Medium Maximization is on figuring out how to get the most effort out of participants. If the goal of scrum is to get more effort out of people then Medium Maximization is very relevant, but if the goal is agility, job satisfaction, schedule predictability, or product quality, it might be less relevant. )

If the goal of scrum is to maximize engineering effort, then Medium Maximization studies do suggest that scrum teams should *earn* story points rather than burning them down. Scrum tracking tools should award points to the whole team (working for the team is definitely stronger than working individual) and display the current point total prominently. The tools might display both the overall point accumulation, and the current scrum/sprint or rolling total of last X days.

Calibrating the way story points scale is very important. The allocation of story points should definitely not show a decreasing margin of return. Very large tasks should never have insufficient story points, else work stops earlier. If we assume that story points are still assigned by engineering estimates, we can still use processes to influence this:

1. Since we know people underestimate large chunks of work worse than they estimate small chunks of work, we should avoid having large chunks of work. Scrum often does this already by having a ceiling on story points. Some scrum practitioners assign no more than 5 story points: anything greater requires them to break down the story. That might not always have the intended effect -- in a meeting that's already taking too much time from too many people, the team might estimate the story at the maximum value rather than immediately have to reconsider the whole definition and breakdown of the story.
2. Some types of engineering effort are often overlooked -- merging code branches, testing, deploying -- which is bad, because fewer story points are estimated for the last bit of effort than that effort truly warranted. In order to avoid that trap, we could always make those chores explicit and have estimates based on past merges rather than let engineers re-estimate too optimistically. (Yes, engineers estimate the same type of task over-optimistically, over and over and over again). Alternatively, a team might add a minimum of X story points to any story that required merging, testing or deployment, and again that X value would be based on real history.
3. We might have the engineers give all their estimates for a story and always pick the largest estimate.
   

Caveats

Rather than write the above sections with lots of caveats, I saved my own criticisms for this section.

• I stated that using story points which estimate effort detaches work from the real goal of adding functionality to a software project. But I neglected to mention that part of scrum is tying story points to demos of releasable functionality. That presumably helps keep story points better moored to real progress.
  
• Scrum and assigning story points helps break down big projects. Breaking down big projects into small tasks and tracking those tasks is nearly ubiquitously practiced in software engineering, probably because it's a good idea. Enforcing a good idea through well-rehearsed process is probably a good idea too.
• Medium maximization is easier to measure if your goals and rewards are clear. In software engineering, the goals are often unclear: product designers don't know exactly how everything ought to work until they can start to see it work, and then make changes. Features get cut due to lack of time. So detaching work from shipping a "complete product" might be a *good* thing -- it stops engineers from doing work on features that might get cancelled next week, and stops them from putting effort into scalability or robustness or abstractions that might not be needed. And if you don't know your goals at all, then scrum at least keeps people from yelling at each other over an apparent lack of progress vs lack of specifications.

• On the OTHER other hand, I imagine medium maximization is stronger in the absence of clear goals and clear relationships between the medium and the goal. If you don't know if you plan to go to Vancouver or to London next year, and if the airline can always change how many points gets you a ticket to each destination, then you can't plan exactly how many air miles you're going to need and how to get those air miles at least cost. Unclear medium/goal relationships will make people try to accumulate more of the medium than they estimate they might need, in order not to fall short. When engineers really don't know where the project is going, scrum will at least keep them resolving story points, and thus preserve the appearance of forward motion.

A few weeks ago, I took a class with Kathy Zimmerman at Stitches West. Kathy designs beautiful sweaters, and cabling is definitely her signature design element. She's also the author (star?) of an instructional DVD on cabling, and owns a yarn store in Ligonier PA.

Kathy showed off some of her sweaters, which was a class unto itself on proper finishing techniques. She introduced cabling charting, which I already love, and got us started customizing charts. I was inspired by a couple of her cables and immediately started sketching freehand, then translated into this:

Then we spent the rest of the class swatching our designs. I knit the bottom chart of of handspun three-ply from Frank the sheep

and the top one out of white worsted Plymouth Yarn Encore, which shows off cables nicely.

We also sketched out a plan for an entire sweater based on the brown sample! I don't know yet if that's what I'll make from that yarn when I'm done spinning it, but possibly. I'll need to check how much yardage I have when I'm done.

Jodi Green posted a link to TVO Archives. That is, the archives of public television in Ontario -- a personal wayback machine. I found "The Polka-dot Door", one episode, but more than enough to remind me of watching that show when I was six. Yikes. Today that show seems surreal and creepy.

Some of the interviews and forum shows are interesting to me now. There is an interview with Margaret Atwood at age 36 (she says "I'm old..."), where she comments a lot on how she is perceived and received by the public.

• "If they're afraid of successful women, you're viewed as a witch. If they view you as a mother figure, they want you to solve their problems."
• "I was talking about economics and politics, and we all know that girls aren't supposed to think about those things, much less talk about them."

Twenty years later interview on the publication of Oryx and Crake, she talks more about literature and politics than on perceptions of herself:

• "You do get these ideas for books that come more or less complete, you just work them out."
• "You can't tell authors what they need to do." on whether authors should be told to discuss politics
• "America is our big neighbour. And if they go down, we go down. So of course we worry. If a person blows you up on some street or other, he isn't going to see your teeny-weeny maple leaf."
• "I got tired of people saying, why do you always write about women." (on Jimmy, main character of Oryx and Crake)

Also an interview with Mordechai Richler, 1994, about his book "This Year on Jerusalem", where he talks about his changing opinions on Zionism from 1943 through to the date of the interview.

• "I'm saying the Israeli's invoke the dead for mean political purposes."
• "I don't think most Israelis want to be on the West Bank as an army of occupation fighting Arab kids. It's very embarrassing."

And more cool stuff, like a discussion of the letters between Pierre Trudeau and Marshal McLuhan, comparing them to a Philosopher King and Court Philosopher, or a collation of opinions on selling movies, with Sydney Pollack, Robert Altman, John Milius and Roger Corman.

Apple, I am incredibly frustrated.

I used to have an iPhoto plugin that published photos to Flickr. When I last updated iPhoto, I was initially happy to see that iPhoto seemed to have the same functionality built-in. I created an album that shared photos to Flickr and used it for a few months, dragging new photos in when I wanted them uploaded. Then I needed to move stuff to a new laptop so I went and archived years-old photos and deleted iPhoto albums I didn't need.

Two weeks later, I find missing photos in my Flickr photo stream, and broken links on Ravelry. Slowly I piece it together: when I cleaned out iPhoto on the old laptop, it silently deleted Flickr photos. Gone are the uploaded versions, titles, tags, descriptions I wrote, comments other people left, and counts of views. I can recover the photos themselves but I can't recover metadata. Most difficult to fix are broken links on other sites.

How could the designers have possibly thought it would be a good feature to silently delete published photos that have metadata of their own? Do the words "Share" and "Publish" not imply putting the photos out there? I tried this again and the words "Synch" and "delete" are not in the iPhoto UI -- the fact that I granted iPhoto permission to delete photos does not imply that deleting an album will delete them online. After all, deleting an album in iPhoto does not delete the photo in iPhoto, so the UI has trained me to think of albums as selections of photos, not containers of photos.

Apple, you had the chance to fix this. When I go looking for support for this, I see posts from obviously distressed and surprised users about thousands of photos deleted, and workarounds posted in response. Surely this allowed you to fix the problem within the last year? You could have issued an update to Flickr with the feature "Saves you from unknowingly destroying personal information online", but you did not. I rather expect this kind of behavior from you Apple, your pride in infallible UI design is well-known, but that does not stop me from being disappointed.

Flickr, I'm disappointed in you too. I know this is not your fault, but many postings in your forums have asked you to disable iPhoto's permission to delete. You could have done that or made mass-deletes "soft-deletes" or asked the flickr account holder to confirm the loss of metadata. You, too, had the chance to save users from losing image metadata.

I've noticed an interesting link between art and software, specifically a parallel to the Scrum process and the existence of Certified Scrum Masters: there are ZenTangles with a specific process for creating art or doodles, and Certified ZenTangle Teachers.

Besides the terminology, both ideas are about regulating and confining what are normally considered open-ended, unbounded problem areas. In ZenTangles, there are arbitrary rules: how many strings to start your ZenTangle with, precisely what fine-tipped pen to use, and the most emphasized restriction, the size of paper is 3.5" by 3.5". Scrum often fetishizes details like planning poker, estimate roshambo, and post-it notes (full disclosure: I've participated deeply in the fetishization of stickie planning).

Then, both approaches have the practitioner routinely break down work into small manageable units and focus on completing units. Breaking down even further, Scrum has recipes for small parts of scrum (like this recipe for a retrospective) and ZenTangles have recipes for line and space doodles. Finally, both approaches celebrate the fact that the "finished" output (a sprint, interval or doodle) is small and done quickly.

Are Zentangles art? Does Scrum produce good design? When do you follow the limitations and when do you depart from them?

Updated: here's my first zentangle, doodled while I was writing this post! Not on 3.5 square paper, nor with the right pen, nor did I follow the "string" process first -- I was always bad at staying within creative bounds.

I downloaded and played Civ V last night. I watched through the high-quality animated movie intro, and noticed how carefully the old man, the hut, and the young man were culturally mixed. Their skin was brown, but could have been from suntans and wear/age. The old man's hair was grey, the young man's hair was hidden. The clothing (including a turban-derived headgear on the young man) was carefully invented and mixed inspirations from different cultures, as far as I could tell. I am convinced this introduction was designed to be culturally neutral, as if the old man and his son could have been the early leaders of any of the cultures used in Civilization.

So it finally struck me: it's all male. They couldn't make it gender-neutral.

The original Civilization had 13 male leaders and one female (Elisabeth I was the default leader of the English, though you could type in "Henry VIII" or "Tony Blair" or "Rowan Atkinson" if you felt like it.) I didn't really notice at the time. Civ II had a female and a male leader for each civilization, which was in-your-face obvious gender equality. It meant that some famous women got exposure (like Catherine for the Russians) but others were pretty unknown (I'm thinking Nazca for the Aztecs, and Ishtar for the Babylonians -- and can you even guess which Civ had "Bortei" as its female leader?)

This kind of public, forced gender equality was what finally made me notice once in a while what role females were expected to play in video games. Like being rescued in Super Mario, rather than being the rescuer. Back when I was just a teenage gamer, I didn't ever notice. I was oblivious. It still can take me a while before I notice, like I spent all that time noticing the careful cultural neutrality in Civ V before it finally hit me there was no attempt at gender neutrality in that introduction.

My point? Given how blunt and ham-handed the stereotyped roles for females in video games are, and how long it took me to start noticing, even though I am a woman, is an indication of how poor I am (and I know most people are) at even noticing gender bias.

I swatched for the Prince of Wales fair isle sweater recreation project (last post) before I even received the yarn I ordered. I used two skeins of Elemental Affects. I also decided to try mixing handspun with commercial fair isle yarn:



The smaller swatch is the one that includes handspun, pale blue, dark blue and dark green, along with the natural light and dark yarns used in the gauge swatch. I learned that it looks nice even when the yarn is not perfect. I also improved the tension on the floats.

Then I received the Jamison and Smith yarn from England:



First I noticed that the natural colour I'd hoped to use for the background was grey, and not at all fawn or beige. That's too bad -- the fawn colour was what I was aiming for. I didn't order enough of the fawny and beige colours to use much of any of them, although I did steal some "sh. FC45" from a Jamison and Smith kit I purchased in the same order. Since I don't want to order just a couple more balls, I'm going to alternate just one or two rows of the background colours at a time.

Since I wanted to use some handspun anyway, I blended an interesting roving with white to get another pale yarn option:



This looked really light when it was in a rolag, but on the bobbin and in the finished yarn, it got darker.



It's clearly the darkest of the light yarns:



It's also very close to the same darkness as the orange I need to use as a "dark" colour. The Prince of Wales painting has bright orange cartouche shapes on a fawn background as one of the peerie patterns, so the orange has to be considered a dark or pattern colour.

I wasn't getting far with using markers to choose colours, so I went to embroidery.



This worked really great. I was able to improvise a little as I went, removing two rows from the charted pattern to make it slightly smaller and make the big X's more visible. Click through to the Flickr image to see my notes on how I modified the embroidered sample as I went along. I also viewed the sample in black and white - I think the contrast is OK overall even though, or perhaps because, the contrast is not even. The orange peerie will draw attention because it's bright orange, the zigzag peerie will draw attention because it's got the darkest yarns, and the big motif will draw attention because it's biggest.

Next step, I believe, is casting on. Perhaps I should do another sample using the real yarns and testing the border ribbing, but I think it will be OK if I don't.

Hang on folks, I'm taking this blog on a sharp shift of topic.

I decided a long time ago to knit a traditional fair isle sweater for a friend who's a sharp, if dated, dresser. I'm finally getting around to it: I purchased Alice Starmore's book of Fair Isle Knitting along with another couple of books, and found the next piece of inspiration I needed:



This is HRH the Prince of Wales in about 1921, painted by John St. Helier Lander. The prince is wearing a fair isle sweater. It's been recreated before, but I'm still enjoying using it as inspiration.

My friend lives in San Diego, so I need to make it a vest. My next step was to take his measurements:



And try to design the actual stitch pattern.



I wasn't very satisfied with the usefulness of using markers. It just feels so far from the way the yarn looks. At least I learned not to use very much light blue in the background, if any, because it makes the background much less fawn-coloured than it ought to be. I also found out how many rows the pattern would be, and at the row gauge I expected, calculated that I could get 5 vertical repeats of the pattern. That's not quite as much as I'd like so I may delete a couple rows (the top and bottom row of the large pattern.

I also ordered a bunch of Jamison and Smith yarn for the project. I ordered the Winterberry Ladies Slipover kit, as well as a bunch of extra pale colours. Oh and I ordered another kit to make for myself later.

As I waited for the Jamison and Smith to arrive, I purchased a couple natural-coloured balls of Elemental Affects yarn. There's not as many colours and certainly not as many heathered yarns, but if I need some I should be able to mix both brands. I used the Elemental Affects to make preliminary swatches, which I'll take pictures of for the next post.

Should a constrained device be a RESTful server or a client? I had been assuming server, and I think I can justify this, although I'm not saying that choosing to put the low-power device in the client role is wrong, because it may depend on constraints and use cases. I do think it's a bad idea to require both client and server roles in the more constrained devices so I'm treating this as an 'either' choice, not and/or. Here is my rough analysis, as part 3 in a series on designing a REST framework for constrained devices (parts 1 and 2).

Roles: Reactive, storing, resource owner

To begin with let's separate the three roles commonly included in the definition of a "server". The first role: "a server is a reactive process" [Andrews, 1991] while a client is a triggering process. The second role: the server stores authoritative versions of resources. Most client/server remote file systems have both these roles in the server: the server reacts to requests and enacts storage operations, while the client is the triggering process making requests, but also controls the namespace has constraints on resource state. These client/server file systems scale moderately well but are inflexible and hard to extend compared to HTTP. In REST, the server takes on a third role: the server manages the namespace and resource state.

It's not clear how much mix-and-matching can be done with these roles in practice. Would it work reasonably well to keep the resource and namespace ownership together with the storage, but to make that agent also be the triggering process instead of the reactive process? I don't know examples of that kind of system in practice, nor do I know how to analyze a theoretical system against fuzzy goals like "flexible" and "scalable". But in the discussion of which roles to assign to the low-power device, I try to keep the three roles separate in case that helps shine light on questions like "should the low-power device proactively send requests such as notifications".

Benefits to consider

1a. Continuity: A server in charge of its own storage, namespace and resource model can be installed and host the same resources, responding to requests at any time, for years without changes. The conditions of its use can even change within limits. A Web site that, when launched, handles a few browser requests a day for certain resources, can later have some "mashup" service querying those same resources at automated intervals and extracting the data. Automated clients don't have this ability to be used in different ways without changing their configuration or code. Applying this to our low-power example: a sensor can handle requests from COAP gateways during normal functioning and by laptop-based clients during configuration, testing or development of new applications, without needing to know why it is handling any request (modulo authorization [1]).

1b. Flexibility: The flip-side of continuity is that features can be added. A device can be upgraded without disrupting the operation of the other low-power devices around it, because an upgraded device can host the same resources as the original device, plus new resources.

2. Scaling: In previous posts, I talked about how scaling large is related to scaling small, due to the relationship between power and load. The flip-side of scaling up to a large load handled with a fixed amount of memory and processing power (the normal problem for HTTP servers), is scaling down the memory and processing power for a fixed load (the scaling problem for low-power devices). In the Web today, what scales better, HTTP servers or HTTP clients? It's well-known that HTTP servers scale well, but there's little concern for clients scaling. It is hard to write a program that load-tests HTTP servers by implementing many clients over many connections -- all too often, the load-testing client machine(s) run out of resources before the servers do.

In HTTP, the server is stateless (see where Roy's dissertation describes "Client Stateless Server" and "Layered Client Cache Stateless Server" for the fuller picture), but the client may not be. The client needs to figure out which servers to talk to, what resources to request from the server, how to interpret those resources and what to do next. In Web browsers, it's the human user who makes many of those choices and may have mental state. An automated client might well include a state machine as the client attempts to achieve a goal involving multiple resources and perhaps even multiple servers. At a minimum, the client knows "What page I'm on now" and "what context did I make a resource request in" as part of its state. In contrast to the client, the server can be stateless, reactive, and doesn't need to know who it's going to communicate with as long as they're authorized [1]. With good application design, a server ought to be implementable with very little use of dynamic memory since it is stateless.

3. Naturalness of resource model: the most natural thing seems to be to model sensor readings and the capabilities of the most constrained devices as resources. Further, if the sensor is used for many years without being upgraded, those resources can have extremely stable URIs. The most natural agent to own these resources is the sensor. This related to the flexibility benefit because of the naturalness of extending the resource model. A sensor v1 can have it resource with a set of readings, and a new device with additional readings can simply have additional resources. A multi-purpose sensor can have all the resources that each single-purpose sensor would have.

4. Naturalness of user model: User interfaces are on the side of the triggering process or client as the user initiates requests for information or changes of state, and their client sends that out. It will be very rare for a human to push a button or otherwise interact directly with a sensor and cause it to act as a client. Much more common will be human requests on a laptop or control panel, which can act as a client and send a request over the local network for sensor information.

---

To conclude, I think these roles and benefits fit together sensibly for a particular kind of constrained device. I am probably focusing too much on a particular idea of a constrained device and its use cases, so I'm happy to admit that this is not the answer for all devices and use cases and would be interested in hearing analysis about other cases. I'm also getting some good comments to the past two posts, so at some point I foresee a post in this series just to discuss the comments.

---

[1] Authorization does not necessarily break any of the assumptions of statelessness. For example, authorization decisions could be pushed down to a lower layer in the simplest cases: the authorization to communicate with a sensor is the same authorization needed to request its sensor data. In more complex cases, the authorization decisions can be explicitly made above the COAP layer but this is more code and storage on the low-power device. In any case, this investigation is starting to make me think that authorization should either be a network function or an application function, not a COAP function or at the resource transfer layer.

REST, with proper use of hypermedia, can be very appropriate for constrained devices. In my last post, I talked about how HTTP has a lot of cruft that could be removed if one were to design a HTTP-lite for constrained devices. The REST architectural style works for constrained devices a lot better than the HTTP syntax and HTTP feature list do. Naturally I'm working from Fielding's thesis, but some different points and emphases are warranted for the context of constrained devices interacting with automated agents, as contrasted to the user deciding which HTML link to click on to interact with a Web server. In order to dive into this, we'll need to really understand how REST requires using server-controlled hypermedia to advertise and navigate the server's resources.

REST involves

• Stateless design. That means no state machine thinking! The client can't assume server state. [1]
  
• A uniform interface on resources, similar to CRUD
  
• Caching support for resource representations
• Navigation to server-named resources via server-controlled hypermedia

The last point is often misunderstood. Despite the fact that we call HTTP client messages requests, it's far more common for programmers and engineers to treat them as commands or instructions (SOAP and RPC illustrate this very well). Many APIs designed to be used over the Web assume that the client decides what should be done and tells the server to do it. This isn't RESTful, and it's not just a quibble: putting the client in control of the interactions puts the client in control of the server's performance and ability to scale.

I didn't understand this when I first read Roy's thesis or when I first worked on WebDAV. I needed examples to make sense of the principle and its utility, particularly when the server does not use HTML and the client is not driven by an active user clicking on links. Thus, this post has examples of non-RESTful navigation and state discovery, as well as RESTful examples:

• The Twitter API: not RESTful in resource naming or discovery
• WebDAV: somewhat RESTful but made a few mistakes in extending HTTP
• Atom: quite RESTful application that sits very lightly and comfortably on HTTP

The Twitter API has fixed resources. Those resources have URIs state which are known to the client in advance. For example, the account profile update resource is named "http://api.twitter.com/1/account/update_profile.format". This limits the Twitter API's ability to rejigger its namespace to add features or to scale by separating resources along a different axis. Next, the client must know a set of permanently-named parameters to be used in a POST request to the URI, to update the profile on the account. The programming model is clearly that twitter client controls the Twitter server by sending commands over HTTP POST. The idea of CRUD is vaguely there (many of the API resources can be retrieved in full with GET, others updated with POST) but diluted by the invention of special update URLs. Caching is possible, at least.

WebDAV is an IETF standard that follows the idea of a limited set of consistent methods well, but doesn't do resource discovery by server-controlled hypermedia. WebDAV set out to provide more authoring functionality such as functions to organize resources and interact with a file system model, so that Webmasters and other Web content creators could have something more suited to Web development than FTP. The WebDAV designers created the PROPFIND request which works a lot like most remote file systems file queries work. In other words, it puts control over querying server resources in the hands of the client. The client determines the scope of the request and the list of properties to be returned, and in the first implementations of WebDAV, the server had little choice but to comply or fail entirely. WebDAV servers had lots of code to handle all the different possible variations on the PROPFIND request, parse its body, and scale differentially based on the different characteristics of the properties that might be requested. All the information to reply to any PROPFIND request that the client might construct, had to be available to any server that might have that request made. Of course, this approach seriously affected performance and scaling.

Atom Feeds was the first standard I saw that did user-free resource discovery via server-controlled hypermedia. An Atom feed is a document offered by the server in XML format, containing links in semantic markup. So, instead of querying the server for blog entries that match a certain pattern, the client simply asks for a feed document via GET. With Atom, the server can precompute or do lazy computation of feed documents, can cache them, can break them down when a feed gets large. This design leaves the scope and detail level of collection membership documents entirely in the servers hands.


More examples can be found at a site that classifies APIs along a continuum of RESTfulness. My quibble with that site is that not all APIs fall along that linear spectrum. WebDAV would have a green box for RESTful identification of resources and a green box for self-descriptive messages, but it would have mixed results for manipulating resources through representations, because collections are not manipulated through representations. It would also have mixed results for using hypermedia "as the engine of application state" because the client makes assumptions about giving resources URIs based on what collection they were put in.

Clearly I need to work on a third post in this series and perhaps a fourth, because I would still like to talk about why constrained devices should operate in the role of the server and provide documents similar to Atom feeds about their state and data. I also have thoughts about a hypothetical framework and how it could be applied in a specific type of constrained device in a way that may not need any dynamic application memory. Hope you can stand to wait.


[1]I think part of what took me so long to understand this was the amount of meaning packed into the word "state". Not only does this mean "what state is the server in, among the states in its state machine", but it also means "what are things named" and "what things exist". So unpacking the injunction that clients shouldn't presuppose state, also means that clients shouldn't presuppose names or existence of resources.

It's not entirely intuitive, but the same architectural style that allows Web servers to handle large scale usage and iterate so successfully, may also help constrained devices (like sub-10$-material-cost sensors) to handle small scale loads and power usage and avoid being upgraded for many years.

A lot of discussions of using REST in constrained devices have glossed over whether it's really interoperable HTTP that's being used (e.g. compressed HTTP) and whether the low-power device would be in the role of the client or the server. My opinion at this point is that for truly constrained devices, compressing HTTP itself would be the wrong choice, and that the device must be in the role of the server. The rest of this series explores why I believe that and should lay open the assumptions that might allow somebody to correct my information and revise my opinion.

The first part of the series attempts to explain is why HTTP is the wrong transfer protocol choice for some constrained devices. In theory it's a massively extensible protocol, with ways to extend the operation set, upgrade versions, add status responses, add headers, add resource content types and so on. Again in theory, almost all headers can be combined with each other and with different methods unless forbidden. In practice, the deployed base makes some HTTP extensions, and particularly the combinatorial expansion of extensions and optional features working together, quite difficult, and much special-case code must be written to handle the cases where widely-deployed software fails to implement some detail correctly. Here's a few examples of problems we've had over the years.

1. In developing WebDAV, we tried to use the standard "OPTIONS *" request (the special-case URI '*' means "give me the options or capabilities of the server"). However, we found that this is poorly implemented. Java servlets, for example, did not support the '*' URI. Other HTTP frameworks supported the '*' URI but did not allow the response to OPTIONS * to be extended by software extending the HTTP framework. Ugh.

2. HTTP has several different ways to indicate the content-length (historical reasons as well as for different use cases). One is the MIME type "multipart/byteranges". Another is chunked transfer encoding, where the server (and thus the client) does not need to know the total length until the last chunk is transferred. A third is "Content-Length". A fourth is for the server to terminate the connection, although this feature opens up the possibility of truncation attacks. Some of these methods work poorly with TCP connection continuation. Both clients and servers have to support almost all of these.

3. Both absolute and relative URIs are allowed in several locations, even though they're not a good idea in all locations. The resolution of a relative URI can be tricky and complicate client software. Implementations of this make it worse; e.g. the "Location" header was defined to allow only absolute URIs, but many implementations have been found which use their generic URI parsing or generating code to allow relative URIs in that field as well.

4. Parsing headers with quoting, separator (comma and semi-colon), whitespace and continuation rules is difficult. See http://greenbytes.de/tech/httpbis/issue-14.xhtml, http://greenbytes.de/tech/httpbis/issue-30.xhtml, http://greenbytes.de/tech/httpbis/issue-62.xhtml, http://greenbytes.de/tech/httpbis/issue-77.xhtml etc. Some headers defined in other specifications besides RFC2616 even used the syntax rules incorrectly and have to be special-cased.

5. The support for the Expect: header and 100 Continue response has never been good. In theory it's required, so there's no way for a client to ask if a server supports it, thus the client could end up waiting quite a while before giving up on the server initial response. Instead quite a few clients ignore the specification text and start sending their request bodies right after the headers including the "Expect" header, without waiting for the server 100 Continue response. This kind of feature also makes it harder to integrate authentication (what happens if the client uses this header when an authentication handshake needs to be initiated? ) and connection/transport features, as well as to implement intermediaries.

There are many more examples in a long issues list (http://greenbytes.de/tech/httpbis/index.xhtml), and some of the discussions of these issues get quite lengthy in considering how features are actually implemented and how they work combined with other features.

Developing a very simple HTTP client is simple, particularly if the client is talking to a single known HTTP server or is making only a small number of kinds of requests. Developing a limited-use HTTP server can be fairly simple too, although it gets to be significantly more complicated if the developer actually tries to handle all RFC2616 features and all unknown headers and methods properly. What turns out to be very hard is building an HTTP client library, server library or extensible server, because these are used in unexpected ways. It's easy for a developer using one of these libraries to break things, e.g. by manually adding a header that doesn't work with the headers added by the library. The library has to support using TLS and not; several kinds of authentication, several extensibility mechanisms and many failure modes.

The HTTP Components project at Apache talks about the many flaws and excessive simplicity of most HTTP client libraries, and states that "This is one reason why Jakarta, and other free and commercial vendors, have implemented independent HTTP clients". In other words, code re-use is seriously reduced by the way HTTP must be implemented. Some software companies are still selling HTTP client libraries to fill implementation gaps.

General-purpose HTTP servers -- ones that work for a large number of resources, a large number of requests, support TLS, content-negotiation, cache-control, redirect, authentication and other major features -- are even harder. When well implemented, HTTP server farms scale tremendously well. But much, much effort has gone into making those work.

When we look specifically at constrained devices, we see a much more limited set of use cases than the overall Web offers.

• Documents are not large! Thus, document range and response continuation features are not desired. Only one transfer-encoding should be necessary, and conditional request features won't be worthwhile.
• Documents from constrained devices are not intended for direct user consumption. There is no need for language and charset negotiation.
• Even negotiating content type may be rare. A constrained device will state what it supports and not negotiate.
• A constrained device will never act as a proxy, so need not support for Via headers and a bunch of status codes like 504. Further, if we define a non-HTTP, non-proxied transfer protocol that can be converted to HTTP at the first proxy step (or converted from HTTP at the last step), then the non-proxied transfer protocol doesn't need any proxy features at all.
• Redirects are not necessary or could at a minimum be drastically simplified, making most of the 300 status responses unnecessary.
• All of the 100 level informative status responses are unnecessary.
• The authorization model for constrained devices is quite different than the authentication model assumed in HTTP (and associated logic like the 401 status code behavior is therefore unnecessary). Access control will be simpler than in current Web servers.

Imagine instead of a messy HTTP client library or server stack, we had a protocol with about a third the features, less extreme extensibility and simplified parsing. If well-specified and interop-tested early and often, I imagine such a protocol could be implemented in framework form in a tenth the size of an HTTP server framework. In even more constrained cases, a very specific constrained transfer protocol implementation (e.g. one which supported a subset of CRUD operations and did no optional features) could be 1/100th the size of a simple HTTP Web server. That starts to be an appropriate size for burning to a chip and shipping in a $1.00 to $10.00 device.

I tried to get some sanity checking on my estimates, and it's not easy. A Web server can sometimes be a wrapper on a file system, in which case most of the logic involved comes "free" from the file system. For example, a simple implementation of one style of redirect might be a trivial number of lines of code if the underlying file system supports symlinks. Andrew McGregor pointed me to the Contiki operating system for embedded objects, which has a Web server already. So that's a proof that embedded and limited devices can sometimes do straight HTTP.

In sum, if interoperability, flexibility and executable size are all simultaneous concerns, HTTP as-is will pose problems. While it may not be necessary for all devices to use a more space-efficient REST transfer protocol than HTTP, it may be necessary for some. If it is necessary to do a new transfer protocol, it should be possible to design a protocol an order of magnitude smaller just by cutting features and unneeded options from HTTP; and possibly smaller yet by being very careful about parsing and limiting extensibility.

More later on REST itself scaling down.