Not Invented Here

Sarasa hat pattern

2011-11-29T16:22:00.001-08:00

I put a bunch of work into a chart to knit myself a hat, so I decided to follow through and make it a pattern.

I was inspired by this fabric in a book of mine:

I knit it up and blocked it:

Added tassels:

And modeled it myself, too. All over Thanksgiving weekend.

Here's the pattern.

2011-09-08T20:47:00.000-07:00

The Story of Snow White and the Seven Dwarves and R2-D2
As requested by a four-and-a-half-year-old: "Yeah, tell me that one. Snow White. And put R2-D2 in it. And R2-D2 has to use his cannon."

Once upon a time, in a land far, far away, lived a young princess named Snow White. She was called Snow White because of her pale white skin. Her mother was dead, and her father was the king. One day her father remarried, and the new queen did not like Snow White. After a while it was clear that Snow White was just in her way. Snow White's best friend and trusty companion was R2-D2 the droid.

Then when Snow White was about sixteen, her father, the king, died suddenly. He left his kingdom to his daughter, but the step-mother intervened and ran things. She made evil choices that made the people of the kingdom work harder, and give more of their money to the queen, and go hungry. Snow White did not like this. Nor did R2-D2.

One day, R2-D2 was rolling along a castle corridor, when, with his fine droid sensors, he heard a conversation between the Evil Queen and a woodcutter. The Evil Queen was saying: "I can't take it any more. Snow White is criticizing my treatment of the rabble, and some of my ministers are agreeing with her. You must kill her when she's out in the forest today. Say that wolves got her."

R2-D2 went immediately to Snow White, of course, and warned her. He recommended that Snow White should escape the castle, to avoid being killed. Meanwhile R2-D2 would call in his friend the Prince from another country, who would come and help them deal with the Evil Queen. R2-D2 explained to Snow White how she could find the Seven Dwarves, who lived together and would hide her until it was safe to return to the castle.

Snow White went out into the forest earlier than usual and evaded the woodcutter. She found the Seven Dwarves' cottage, and asked for their help. Of course they agreed and said she could stay. The Seven Dwarves were Grumpy, Sleepy, Sneezy, Doc, Nosey, Geeky, and Skip. Each day they went away to work in the mines, and came back in the evening. Snow White started to help out by cleaning up after breakfast, and preparing a nice supper for when they came back, so the Seven Dwarves grew to love her indeed.

The Evil Queen knew that Snow White was now hiding, so she came up with a different plan. She used magic to disguise herself as a poor old woman selling apples. She put one special apple in her basket, just for Snow White; it had poison. She went around the area of the castle, asking if anybody had seen a pretty young girl, new in the area. Eventually she found a village where some people had noticed a pretty young girl who had recently started living with the Seven Dwarves, and so she went to investigate. She found Snow White, all alone in the middle of the day.

"Hello, my deary", she said. "Would you like to buy some apples?"

"Oh, those look delicious," Snow White answered. "But I have no money, I'm sorry."

"Well, you're such a pretty dear, " the Evil Witch disguised as a poor old woman said, "I'll give you this apple if you give me the pleasure of eating it. An old woman like me doesn't enjoy life as much as you young folks, so I'll just take a break and watch you eat this apple." And she handed Snow White the apple prepared with poison.

Snow White took a bite of the apple and before she even finished chewing it, she swooned. That means, she fell over, right onto the ground, as if asleep -- or dead. The Evil Queen laughed and went back to the castle, convinced that Snow White was dead.

Meanwhile, don't forget, R2-D2 had sent a message to his friend, the Prince from another country. It took a long time in those days for a message to get to the other country, and for the Prince to ride to Snow White's country. The Prince was just arriving when R2-D2 heard the news the Evil Queen was spreading -- that Snow White was not just missing, but found dead, and that the Evil Queen was going to rule the country.

R2-D2 rolled out to meet the Prince. When the two friends found each other on the road, R2-D2 led the Prince to the Seven Dwarves cottage. The Seven Dwarves had just arrived, and were moaning sadly about poor Snow White.

With his sensors, R2-D2 could tell that Snow White was just breathing very slightly, only barely alive. In droid speak (doot dweet peep brrrrr peep peep doot) he instructed the Prince to lift Snow White onto the table so that R2-D2 could use his gripper arm to open Snow White's mouth. R2-D2 found pieces of apple in her mouth, because she hadn't even finished swallowing the apple, so he took them out. Then he instructed the Prince in how to sit Snow White up to whack her back, to dislodge more pieces of apple. Together, they got enough of the poison apple out of Snow White so that she could recover.

The next day, Snow White, the Prince, and R2-D2 headed to the castle. Snow White took a bow and arrows just in case, and the Prince had his sword. When they arrived at the castle, the gates were locked and the Evil Queen was instructing the guards not to let them in.

"She's not the real Snow White! She's an impostor! Keep her out of the castle!" she yelled to the guards.

"I am the real Snow White!" Snow White shouted back. "You guards, you know me. I grew up in this castle!" And the did know her, so some of them put down their swords, and their bows and arrows. The Evil Queen grabbed one of them and tried to threaten his life with her dagger, so he would obey her, so Snow White shot a warning arrow. The Evil Queen had to hide from the arrows so she let the guards go.

The gate was still locked, so R2-D2 used his cannon to blast it open. Boom! The gate crashed open, and the threesome walked into the castle through the gate. But the Evil Queen was waiting for them, and she tried to jump on Snow White and kill her with a dagger. Luckily, the Prince was ready with his sword, and he swung it to protect Snow White, and cut off the Evil Queen's head.

Snow White thanked the Prince for saving her, although she said "You shouldn't have killed her. We should have brought the Evil Queen to justice and put her in prison.".

"I know," said the Prince, "But it was an accident. It was in defense."

"That's true," said Snow White. "Say, you seem like a nice guy. How about you stay here with me? I'm going to rule this kingdom, and do a much better job than the Evil Queen, too. If you give me good advice, and help me rule this kingdom, maybe we could get married. "

"That sounds like a good idea," said the Prince.

R2-D2 said "Peeeeep doot dweet-deet doot brrrrr peep!"

And they all lived happily ever after.

2011-08-30T17:05:00.000-07:00

My third post on doing RESTish JSON APIs in Rails is about testing the API. I'm assuming the network API should remain stable, not have fields added or removed without consideration (and ideally, documentation), especially if there are independently-written clients that use the server's API. So normal Rails "functional" testing is not enough here.

"Functional" testing is in scare quotes because it's got a specific meaning to Rails, not a generally-accepted meaning to programmers. It is how Rails defaults encourage programmers to test the actions of their controller classes, which is to say the results of their Web request handling (e.g. in the Rails guide, especially the section on testing views).

It gets pretty tedious to assert all the things are in a JSON response that should be:

assert_response :success

  data = JSON.parse(@response.body)["data"]

  assert_not_nil @data["comments"]

  assert_equal 1, @data["comments"].length

  assert_equal "The User", @data["comments"][0]["display_name"]

  assert_equal "the comment text", @data["comments"][0]["comment_text"]

Worse, this only tests that things that are tested for are there, not that new things don't creep in. It's pretty easy to modify code in a way that passes what looked like reasonable functional tests, but breaks the client.

While working on improving our testing for API responses, I've found a few things are important to me.

The tests should show what an entire response ought to look like (aids readability)
The tests should flag things that aren't supposed to be there as bugs
The test responses should be templates, so that variables can vary
The output of a test should show the difference, so I don't have to wade through pages of desired output and pages of actual output

It took us a while (thanks to Tom, Steve and others at work) to put all the pieces together, and there are probably more improvements to be made, but I'm happy enough with what we came up with to blog about it. We use

Cucumber to describe the request, the response status and the response body
Custom cucumber step definitions to let the step define the entire response body as a Ruby Hash
A custom template "Wildcard" object to let variables vary within the response body
Rails feature to compare hashes and find diff, to produce readable output

So now here's what one of our API response tests looks like!

Feature: Commenting

  Background: 

    Given the following post exists:

      | id | 1234 |

    And the following comment exists:

      | id | 882281 |

      | post_id | 1234 |

      | comment_text | "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience." |

      | user_id | 101 |

  

  Scenario: Retrieve a comment 

    When I GET "/posts/1234/comments/882281" with a user session

    Then the response should have the following comment:

    """

      {

        "comment_text"=> "O gentle son, upon the heat and flame of thy distemper sprinkle cool patience.",

        "created_at"=> Wildcard.new(String),

        "updated_at"=> Wildcard.new(String),

        "creator" => {

          user_id => 101,

          handle => "gertrude",

          display_name => "Queen Gertrude",

          number_comments => Wildcard.new(Fixnum)

        },

        "post_url" => Wildcard.new(String)

      }

    """

The Wildcard class looks like this:

class Wildcard

  def initialize(must_be_class=Object)

    @must_be_class = must_be_class

  end

  def ==(other)

    other.class == @must_be_class

  end

end

And the Cucumber step that implements "the response should have the following comment" looks like this:

Then /^the response should have the following ([^"]*):$/ do |element, string|

  the_hash = eval(string)

  response_data = JSON.parse(@response.body)["data"][element]

  assert_equal({}, response_data.diff(the_hash))

end

Note, what this means is that our API responses have "data" as the top element, with a "comment" element inside -- that part is asserted to be there, so that the contents of the comment can be compared with the Rails hash diff method. If the comment element in the response had an extra child, this would produce an error with the extra child in the diff. If the comment was missing a child, this would produce an error with the extra child in the diff. And if the comment has a child with a different value than what's in the template, this would produce an error with the returned value in the diff.

I realize this is not Behavior Driven Development (BDD). One can use Cucumber to do BDD, but the two are not inextricably linked. And our intent here is not to do BDD but to maintain a stable API. I think this works.

2011-08-29T12:27:00.000-07:00

This is another post outlining "How I spent several days frustratingly getting something to work" so that "you don't have to". I think the theory behind these kinds of posts is the "you don't have to" part but the reality is "OMG I have to rant and boy it would help put this behind me if I could pretend there was some USEFUL lesson."

I already posted that I've been using Jenkins and RVM. The theory is great: Jenkins lets you set up lots of recurring jobs and have one dashboard to see that all of your software is basically working, while RVM allows you to choose a different Ruby version and gemset for each project that might need that.

The first Jenkins job I set up seemed to use the RVM Ruby-1.8.7 just fine, but when I tried to set up a second job that required Ruby-1.9.2, I never could get it working. First it tried to use the system ruby, which I blew away. Then I had hours tracking down references to the system ruby. The command "rvm use 1.9.2" appeared to work and chose the right rvm ruby directory. But even when it worked, when the very next command run by jenkins was "ruby -v", the machine either couldn't find the ruby install at all, or found the system one, but never the one just successfully activated by RVM. Even more frustrating, this worked at the command line, but not when running a job from Jenkins. (If anybody can explain how Jenkins runs jobs differently and what effect that has on context, I'd love to know -- having a model of what's going on would help.)

Finally I discovered "rvm-shell", which can force execution of a command within the context of what RVM set up. So now every line of the jenkins job script that uses ruby, bundle or rake, is wrapped in that:

rvm use 1.9.2@profiles

rvm-shell -c "ruby -v"

rvm-shell -c "bundle install"



rvm-shell -c "rake db:migrate"



# Finally, run your tests

rvm-shell -c "rake"

It worked. 56 is the magic number -- that is, build #56 was the first one to work on this project. Phew.

2011-08-23T13:51:00.000-07:00

Many modern Web services or Web APIs use REST style with JSON request and response bodies. The first problem doing this with Rails is that Rails isn't entirely conducive to doing REST, which I already discussed, but the second problem is that Rails isn't very conducive to doing JSON in a way that keeps the API stable. As is normal with Rails, it gets the basics running really fast, but its default assumptions and "magic" utilities need to be bypassed for some purposes.

This seems to be the default approach -- just call Rails "as_json" or "to_json" methods on the model instance, which gets all of the model's database columns but none of the model's dynamic information. This example has an array of Message instances, and without any monkey-patching or overriding as_json, the effect of this will be to export all of the ActiveRecord fields of Message (but not other content such as attributes) as JSON elements. For example, if the Message model uses a "messages" table with a column called "title", this code would expose the value in that column as "title" in the JSON as well.


render :json => message_list.as_json

But if some of the database columns on the model are for internal use and not for the API, the API implementor needs to override the default behavior to include only the columns that are intended for the API:


# ... model for Message
def as_json(opts = {})
opts = {:only  => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
super(opts)
end

#... controller for Message
render :json => message_list.as_json

(One could also use "except" instead of "only" as an option for as_json, but then if a new column gets added to the Message model, it would automatically be added to the API whether you want it or not).

It gets even more complicated if some of the model's information needs to be converted, combined, or otherwise manipulated:


# ... model for Message
def as_json(opts = {})
# change id to be called message_id because existing clients use message_id
out = { 'message_id' => id.to_s(&:as_json) }

opts = {:only  => [:title, :body, :thumbnail_url, :background_url, :action, :display_on_launch, :ok_button_string, :cancel_button_string]}
out.merge!(super(opts))
end

#... controller for Message
render :json => message_list.as_json

It starts to get ugly, and a lot of the ugliness lives in the Model, not the Controller... wait. Oops. In MVC, isn't this what the View is for?

Enter RJSON. This is a template type for Rails, by Tom Lieber. The Ruby hash (like a dictionary) is already very familiar to Ruby developers, so why not have the Ruby developer construct a hash in a template file. A RJSON template file named "messages.rjson" might look like this:


{
:messages => @messages.map do |msg|
{
:message_id => msg.id.to_s,
:title => msg.title,
:body => msg.body,
:thumbnail_url => msg.thumbnail_url,
:background_url => msg.background_url,
:action => msg.action,
:display_on_launch => msg.display_on_launch,
:ok_button_string => msg.ok_button_string,
:cancel_button_string => msg.cancel_button_string
}
}

And its controller looks like this, where @messages is the array placed in context for the view to use:


@messages = Message.find_all_by_user_id(user_id)
render :file => "messages"

This is simple, but even much more complicated examples remain readable. Partials can also be used; if some data structure gets re-used exactly the same way in more than one part of the API, that data structure can be a partial view. It's easy to see what the API should look like from the view, which is part of the point of views.

I recommend it, which is to say: by all means get your Web API prototyped quickly using as_json, but when you start to get serious about the API end-points and field names, documenting the API and keeping it stable, start using RJSON.

ETA: Tom pointed me to a recent gem published by somebody else with the same basic plan. It's also at github (same name, Jan de Poorter)

2011-08-10T16:54:00.000-07:00

I've been setting up Jenkins to run Rails project unit and functional tests as builds. This has been difficult, and 90% of the problems have been in getting different versions of various gems available. RVM is supposed to help, as is Bundler. I should have started out with a very methodical plan for giving access to the jenkins user, then having the jenkins user install rvm and create gemsets, then have the jenkins user install gems and run bundle install. Instead I got things working quickly with super-user permissions and then builds fail because it's hard to 'sudo' commands during an automated build.

Here's one problem running "bundle install" as the jenkins user:


Using rails (3.0.4) 
Using right_http_connection (1.3.0) from git://github.com/rightscale/right_http_connection.git (at master) /usr/lib/ruby/1.8/open-uri.rb:32:in `initialize': Permission denied - right_http_connection-1.3.0.gem (Errno::EACCES)
	from /usr/lib/ruby/1.8/open-uri.rb:32:in `open_uri_original_open'
	from /usr/lib/ruby/1.8/open-uri.rb:32:in `open'
	from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:73:in `write_package'
	from /usr/local/lib/site_ruby/1.8/rubygems/builder.rb:38:in `build'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `chdir'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:450:in `generate_bin'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/source.rb:559:in `install'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:58:in `run'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/rubygems_integration.rb:93:in `with_build_args'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:57:in `run'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:49:in `run'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/installer.rb:8:in `install'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/cli.rb:220:in `install'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `send'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/task.rb:22:in `run'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor.rb:263:in `dispatch'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/lib/bundler/vendor/thor/base.rb:386:in `start'
	from /usr/local/rvm/gems/ruby-1.8.7-p352@sonicnet/gems/bundler-1.0.17/bin/bundle:13
	from /usr/bin/bundle:19:in `load'
	from /usr/bin/bundle:19

After reading up on similar but different errors via Google, I concluded this was probably a local filesystem permission error. I tried granting the jenkins user various permissions in /usr/lib/ruby and /usr/local/rvm/gems -- one misleading clue was that the "right_http_connection-1.3.0.gem" file in /usr/lib/ruby/1.8/gems/cache was the only gem in that directory with a different user/group than the other gems. Fixing that didn't fix the problem, nor did deleting that gem file. I then realized that bundler puts gems in a different place: .bundle in the project directory. Oh, that's owned by root -- great, fix that. Oops, didn't work. Finally I deleted .bundle and ran bundle install again -- as the jenkins user -- and it worked.

(This is probably boring to most readers but it's a small payment for all the other bloggers' posts I've read that gave me clues in various debugging dead-ends. )

2011-07-22T13:37:00.000-07:00

This week I was asked to review draft-ietf-hybi-thewebsocketprotocol-10, The Websocket Protocol, as it gets closer to being an IETF RFC. After I sent my review, I was prompted to look back and reminded myself that I wrote the first message to apps-review back when it had a separate mailing list, asking for the first review from the newly-formed team. This explains why I'm still reviewing IETF documents despite having no current IETF activity -- I feel I owe a few people a few reviews in return.

Websockets itself also brought back the memories. I approved the BOF, the "Birds of a Feather" meeting that organized the HyBi Working Group (WG), and approved the formation of the WG. I felt it was very important that the Websocket protocol be developed in a venue where we had server developers and transport and security experts as well as browser developers. We also had to ensure that the protocol was considered quite separate from the Websocket API which is part of HTML5. Much painful experience shows that while protocol libraries are hard enough to upgrade, the protocols stay around for long and are even harder to fix or replace, so backward compatibility, versioning and extensibility are crucial.

While IETF progress for Websocket protocol was, as almost everything at the IETF is, agonizingly slow, I was glad to see that the near-end result has pretty solid HTTP compliance, solid framing and transport text, and deals with security concerns that weren't even on the radar before the IETF effort. Personal kudos to Joe Hildebrand, Alexey Melnikov, Ian Fette, Ian Hickson, Adam Barth and Salvatore Loreto, and kudos to everybody else who contributed.

2011-07-13T17:20:00.000-07:00

Resourceful != RESTful

Resourceful routing in Rails is certainly useful. As is customary in Rails, you get a lot of stuff achieved with a very small amount of declarations. And at first glance it looks like REST! Yay! Except... it will lead one slightly astray. I've come across a few ways in which "resourceful" routing in Rails doesn't really follow REST principles, in ways that aren't merely theoretic but can affect intermediary behavior.

Resourceful routing defines paths or routes for objects that the server developer would like to have manipulated with CRUD operations. The developer can declare an 'invitation' to be a resource, and the routes to index all invitations, create a new invitation, download an invitation, update or delete the invitation, are automatically created. But, problem the first:

Resourceful routing uses POST to create a new resource. PUT is defined as creating a new resource in HTTP, and intermediaries can use that information-- but only if PUT is used. If POST is used, an intermediary can't tell that a new resources was created.

All routes share common error handling. Route not found? Return 404! But Rails puts the method in as part of the route definition. Thus, problem the second:

Rails returns 404 Not Found if the client uses an unsupported method on a resource that exists. So for example if I apply resourceful routing such that a resource can be downloaded with GET and updated with PUT, but don't define a POST variant route for that URL, then when the client tries to POST to that URL the server returns 404 because the route (with correct method) was not found. In theory an intermediary could mark the resource as missing and delete its cached representation. Instead, there's a perfectly good error to use in HTTP when a method is not supported on a URL, and that is 405 Method Not Allowed.

Rails has some magic to collect all the parameters on an incoming request and collect them in a params hash. Thus, if my Web page takes a query parameter named "user_id", I just pull params["user_id"] and I've got it. If my Web form sends the parameter inside the body instead, it's still there in params["user_id"]! This lets server developers easily change their mind between having GET or POST used as the form submission method. However, it creates two problems with intermediaries, linked to the way resources are named.

URL parameters are mixed with body parameters. This may not cause problems for a POST, where the response typically isn't cachable anyway. But it's a bad choice in for GET requests, where the URL containing parameters affects caching, while other parameters are unseen by intermediaries.

Query parameters are treated the same as path elements. Routes are defined as paths that can have parameters in them. So if the routes file defines a route for "/v1/store/buy/:product/with_coins", the :product path element could be any string, and Rails will pass that string into the application just as if it were a URL query parameter. However, caches are supposed to work differently if a URL has query parameters than if it does not, so treating them as the same is misleading the developer.

These kinds of things can be subtle to track down. For example, it would be pretty hard to track down a bug where an intermediary was returning a cached response to a GET request when the Web server programmer had the response varying by parameters, and some client was sending those parameters in the body where the intermediary couldn't see them. I guess the saving grace, if you call it that, is that real intermediary caching isn't as common as I had thought. Even when intermediaries exist and could cache, Rails applications don't commonly seem to take advantage of that.

2011-07-11T11:33:00.000-07:00

I was wondering if the term "guys" is becoming more gender-neutral over time. Have you ever referred to a group including some women as "guys"? I've used the term to refer to groups made up entirely of women, but only in the second-person plural sense in casual or sporty situations. E.g. I'd say "Are you guys ready" to a group of women I was about to go running with. I wouldn't say "I'm going running with the guys from work" if it was women from work because that would imply men. And I'd never ever use the "guy" for a woman or girl, in first, second or third person singular context. I might in a stretch say "I'm just one of the guys", but that's plural first person. Very strange but wiktionary seems to document this as common.

Even stranger, however, is the derivation of the word "guy". From a proper name to a derogatory term to a generic term. I found this on Online Etymology, but I'm going to reword the explanation in chronological order with some of the inferences filled in, because I had to read their explanation three times for it to make sense.

Guy Fawkes got his name from Old German word for "wood" or "warrior", or possibly Welsh for "lively" or French for "guide" -- but it was a standard boy's name at the time at any rate.
Guy Fawkes planned the failed Gunpowder Plot to blow up the House of Lords.
Thereafter, Guy Fawkes Day was celebrated with fireworks and bonfires, and an effigy of Guy Fawkes paraded through the streets being set before fire. The effigy would have been a straw man wearing cast-off clothing.
The name Guy would have been so associated with this effigy, that calling somebody a "guy" must have brought to mind a badly-dressed scarecrow figure at that time.
The term became more generic, from meaning "badly-dressed fellow" to meaning "ordinary man", over the course of just a generation.

Note that "guy" is definitely an American term, even though the Gunpowder Plot was in Britain. Guy Fawkes day was widely celebrated in US before the revolution, so the Guy effigy would have been a familiar figure in the US back when it came to mean "grotesquely or badly dressed".

I wonder if there's a term for when a derogatory word becomes unobjectionable, and whether this usually happens by being appropriated (the people the term refers to use the word proudly) or just by being watered down.

2011-07-07T17:15:00.000-07:00

I learned about Medium Maximization behavior the other day and thought about how Scrum works. It's an interesting lens.

When people engage in medium maximization, they maximize a proxy for a real goal, rather than maximizing progress towards the real goal. The proxy or medium could be points on an exam as a proxy for understanding a topic, or air miles as a proxy for plane tickets. People do odd things for air miles that they wouldn't do for an equivalent direct reward. More details at http://faculty.chicagobooth.edu/christopher.hsee/vita/Papers/MediumMaximization.pdf - there are quite a few interesting side effects involved.

Applying to scrum

Scrum assigns story points to small accomplishable goals intended to eventually combine to reach a finished project. Since story points are not the same as the goal, story points are a medium, and we should expect participants to put more weight on accomplishing story points than on finishing the project, changing their behavior compared to when story points are not used.

I fully believe this behavior works as much for the managers and stakeholders in Scrum as for the engineers. In fact it might be more so: the only thing managers can measure is story points, not actual useful code, so managers are probably more satisfied by story point burndown than engineers are. Engineers have more visibility into what's being achieved towards a completed project, so that can have more influence on their choices, but I'm quite sure engineers are influenced too.

Predictions

A hypothesis like this can be tested by making predictions that can be tested and measured. I don't propose to actually study this myself, but making predictions and finding things that can be measured is fun.

In the studies done by Christopher Hsee and co-authors, participants had to accomplish tasks to get rewards. In the control set, the participants contemplated doing more or less work for different rewards. When a medium was introduced, and more points were disproportionally offered for more work, participants did much more work on average, even though the rewards that the points could be exchanged for were the same. Applying this to scrum, we should find that scrum participants do more work to burn down more story points, especially if the scale is non-linear. Imagine that management could somehow set the story points such that doing 40 hours of work accomplished 40 story points, but 50 hours of work accomplished 75 story points. Participants would likely work more, and this can probably be measured.

Another prediction comes from the studies that show that the behavior of maximizing the proxy points continues even if the primary reinforcer is removed. In software development, a team might continue working on story points even if the product plan had been cancelled, unless they were told to stop. This sounds unlikely and wrong, but imagine instead a project is rumoured to be canceled soon due to a possible merger. A manager might prefer their team make solid progress on story points even with that rumour flying around. If development teams show less loss of morale and effort when such rumours are flying around when using scrum, than not, this would be good evidence that the teams were maximizing story point burndown rather than project release progress.

The last prediction is based on the results of applying a linear medium -- for each chunk of work you get the same number of points -- to a non-linear reward. In one study, participants had a reward with decreasing rates of return offered. When a linear medium was offered, but the exchange rate for points to reward had the same decreasing rate of return, participants did much more work. In software engineering it's commonly observed that the last 20% of the work takes the last 80% of the time, so software projects are arguably non-linear in the same way the rewards in this study were. If story points offer a linear medium with respect to effort -- which in classic scrum they do -- we should see engineers happy to do more work even as the project gets into its depressing cleanup and bugfix phase. (See note below)

Joking Suggestions

(I'm only playing with this lens, not suggesting real changes to scrum processes. Much of the focus on Medium Maximization is on figuring out how to get the most effort out of participants. If the goal of scrum is to get more effort out of people then Medium Maximization is very relevant, but if the goal is agility, job satisfaction, schedule predictability, or product quality, it might be less relevant. )

If the goal of scrum is to maximize engineering effort, then Medium Maximization studies do suggest that scrum teams should *earn* story points rather than burning them down. Scrum tracking tools should award points to the whole team (working for the team is definitely stronger than working individual) and display the current point total prominently. The tools might display both the overall point accumulation, and the current scrum/sprint or rolling total of last X days.

Calibrating the way story points scale is very important. The allocation of story points should definitely not show a decreasing margin of return. Very large tasks should never have insufficient story points, else work stops earlier. If we assume that story points are still assigned by engineering estimates, we can still use processes to influence this:

Since we know people underestimate large chunks of work worse than they estimate small chunks of work, we should avoid having large chunks of work. Scrum often does this already by having a ceiling on story points. Some scrum practitioners assign no more than 5 story points: anything greater requires them to break down the story. That might not always have the intended effect -- in a meeting that's already taking too much time from too many people, the team might estimate the story at the maximum value rather than immediately have to reconsider the whole definition and breakdown of the story.
Some types of engineering effort are often overlooked -- merging code branches, testing, deploying -- which is bad, because fewer story points are estimated for the last bit of effort than that effort truly warranted. In order to avoid that trap, we could always make those chores explicit and have estimates based on past merges rather than let engineers re-estimate too optimistically. (Yes, engineers estimate the same type of task over-optimistically, over and over and over again). Alternatively, a team might add a minimum of X story points to any story that required merging, testing or deployment, and again that X value would be based on real history.
We might have the engineers give all their estimates for a story and always pick the largest estimate.

Caveats

Rather than write the above sections with lots of caveats, I saved my own criticisms for this section.

I stated that using story points which estimate effort detaches work from the real goal of adding functionality to a software project. But I neglected to mention that part of scrum is tying story points to demos of releasable functionality. That presumably helps keep story points better moored to real progress.
Scrum and assigning story points helps break down big projects. Breaking down big projects into small tasks and tracking those tasks is nearly ubiquitously practiced in software engineering, probably because it's a good idea. Enforcing a good idea through well-rehearsed process is probably a good idea too.
Medium maximization is easier to measure if your goals and rewards are clear. In software engineering, the goals are often unclear: product designers don't know exactly how everything ought to work until they can start to see it work, and then make changes. Features get cut due to lack of time. So detaching work from shipping a "complete product" might be a *good* thing -- it stops engineers from doing work on features that might get cancelled next week, and stops them from putting effort into scalability or robustness or abstractions that might not be needed. And if you don't know your goals at all, then scrum at least keeps people from yelling at each other over an apparent lack of progress vs lack of specifications.

On the OTHER other hand, I imagine medium maximization is stronger in the absence of clear goals and clear relationships between the medium and the goal. If you don't know if you plan to go to Vancouver or to London next year, and if the airline can always change how many points gets you a ticket to each destination, then you can't plan exactly how many air miles you're going to need and how to get those air miles at least cost. Unclear medium/goal relationships will make people try to accumulate more of the medium than they estimate they might need, in order not to fall short. When engineers really don't know where the project is going, scrum will at least keep them resolving story points, and thus preserve the appearance of forward motion.

2011-03-13T10:44:00.000-07:00

A few weeks ago, I took a class with Kathy Zimmerman at Stitches West. Kathy designs beautiful sweaters, and cabling is definitely her signature design element. She's also the author (star?) of an instructional DVD on cabling, and owns a yarn store in Ligonier PA.

Kathy showed off some of her sweaters, which was a class unto itself on proper finishing techniques. She introduced cabling charting, which I already love, and got us started customizing charts. I was inspired by a couple of her cables and immediately started sketching freehand, then translated into this:

Then we spent the rest of the class swatching our designs. I knit the bottom chart of of handspun three-ply from Frank the sheep

and the top one out of white worsted Plymouth Yarn Encore, which shows off cables nicely.

We also sketched out a plan for an entire sweater based on the brown sample! I don't know yet if that's what I'll make from that yarn when I'm done spinning it, but possibly. I'll need to check how much yardage I have when I'm done.

2011-03-03T13:09:00.001-08:00

Jodi Green posted a link to TVO Archives. That is, the archives of public television in Ontario -- a personal wayback machine. I found "The Polka-dot Door", one episode, but more than enough to remind me of watching that show when I was six. Yikes. Today that show seems surreal and creepy.

Some of the interviews and forum shows are interesting to me now. There is an interview with Margaret Atwood at age 36 (she says "I'm old..."), where she comments a lot on how she is perceived and received by the public.

"If they're afraid of successful women, you're viewed as a witch. If they view you as a mother figure, they want you to solve their problems."
"I was talking about economics and politics, and we all know that girls aren't supposed to think about those things, much less talk about them."

Twenty years later interview on the publication of Oryx and Crake, she talks more about literature and politics than on perceptions of herself:

"You do get these ideas for books that come more or less complete, you just work them out."
"You can't tell authors what they need to do." on whether authors should be told to discuss politics
"America is our big neighbour. And if they go down, we go down. So of course we worry. If a person blows you up on some street or other, he isn't going to see your teeny-weeny maple leaf."
"I got tired of people saying, why do you always write about women." (on Jimmy, main character of Oryx and Crake)

Also an interview with Mordechai Richler, 1994, about his book "This Year on Jerusalem", where he talks about his changing opinions on Zionism from 1943 through to the date of the interview.

"I'm saying the Israeli's invoke the dead for mean political purposes."
"I don't think most Israelis want to be on the West Bank as an army of occupation fighting Arab kids. It's very embarrassing."

And more cool stuff, like a discussion of the letters between Pierre Trudeau and Marshal McLuhan, comparing them to a Philosopher King and Court Philosopher, or a collation of opinions on selling movies, with Sydney Pollack, Robert Altman, John Milius and Roger Corman.

2011-02-26T13:55:00.000-08:00

Apple, I am incredibly frustrated.

I used to have an iPhoto plugin that published photos to Flickr. When I last updated iPhoto, I was initially happy to see that iPhoto seemed to have the same functionality built-in. I created an album that shared photos to Flickr and used it for a few months, dragging new photos in when I wanted them uploaded. Then I needed to move stuff to a new laptop so I went and archived years-old photos and deleted iPhoto albums I didn't need.

Two weeks later, I find missing photos in my Flickr photo stream, and broken links on Ravelry. Slowly I piece it together: when I cleaned out iPhoto on the old laptop, it silently deleted Flickr photos. Gone are the uploaded versions, titles, tags, descriptions I wrote, comments other people left, and counts of views. I can recover the photos themselves but I can't recover metadata. Most difficult to fix are broken links on other sites.

How could the designers have possibly thought it would be a good feature to silently delete published photos that have metadata of their own? Do the words "Share" and "Publish" not imply putting the photos out there? I tried this again and the words "Synch" and "delete" are not in the iPhoto UI -- the fact that I granted iPhoto permission to delete photos does not imply that deleting an album will delete them online. After all, deleting an album in iPhoto does not delete the photo in iPhoto, so the UI has trained me to think of albums as selections of photos, not containers of photos.

Apple, you had the chance to fix this. When I go looking for support for this, I see posts from obviously distressed and surprised users about thousands of photos deleted, and workarounds posted in response. Surely this allowed you to fix the problem within the last year? You could have issued an update to Flickr with the feature "Saves you from unknowingly destroying personal information online", but you did not. I rather expect this kind of behavior from you Apple, your pride in infallible UI design is well-known, but that does not stop me from being disappointed.

Flickr, I'm disappointed in you too. I know this is not your fault, but many postings in your forums have asked you to disable iPhoto's permission to delete. You could have done that or made mass-deletes "soft-deletes" or asked the flickr account holder to confirm the loss of metadata. You, too, had the chance to save users from losing image metadata.

2011-02-12T09:10:00.001-08:00

I've noticed an interesting link between art and software, specifically a parallel to the Scrum process and the existence of Certified Scrum Masters: there are ZenTangles with a specific process for creating art or doodles, and Certified ZenTangle Teachers.

Besides the terminology, both ideas are about regulating and confining what are normally considered open-ended, unbounded problem areas. In ZenTangles, there are arbitrary rules: how many strings to start your ZenTangle with, precisely what fine-tipped pen to use, and the most emphasized restriction, the size of paper is 3.5" by 3.5". Scrum often fetishizes details like planning poker, estimate roshambo, and post-it notes (full disclosure: I've participated deeply in the fetishization of stickie planning).

Then, both approaches have the practitioner routinely break down work into small manageable units and focus on completing units. Breaking down even further, Scrum has recipes for small parts of scrum (like this recipe for a retrospective) and ZenTangles have recipes for line and space doodles. Finally, both approaches celebrate the fact that the "finished" output (a sprint, interval or doodle) is small and done quickly.

Are Zentangles art? Does Scrum produce good design? When do you follow the limitations and when do you depart from them?

Updated: here's my first zentangle, doodled while I was writing this post! Not on 3.5 square paper, nor with the right pen, nor did I follow the "string" process first -- I was always bad at staying within creative bounds.

2011-01-13T14:13:00.000-08:00

I downloaded and played Civ V last night. I watched through the high-quality animated movie intro, and noticed how carefully the old man, the hut, and the young man were culturally mixed. Their skin was brown, but could have been from suntans and wear/age. The old man's hair was grey, the young man's hair was hidden. The clothing (including a turban-derived headgear on the young man) was carefully invented and mixed inspirations from different cultures, as far as I could tell. I am convinced this introduction was designed to be culturally neutral, as if the old man and his son could have been the early leaders of any of the cultures used in Civilization.

So it finally struck me: it's all male. They couldn't make it gender-neutral.

The original Civilization had 13 male leaders and one female (Elisabeth I was the default leader of the English, though you could type in "Henry VIII" or "Tony Blair" or "Rowan Atkinson" if you felt like it.) I didn't really notice at the time. Civ II had a female and a male leader for each civilization, which was in-your-face obvious gender equality. It meant that some famous women got exposure (like Catherine for the Russians) but others were pretty unknown (I'm thinking Nazca for the Aztecs, and Ishtar for the Babylonians -- and can you even guess which Civ had "Bortei" as its female leader?)

This kind of public, forced gender equality was what finally made me notice once in a while what role females were expected to play in video games. Like being rescued in Super Mario, rather than being the rescuer. Back when I was just a teenage gamer, I didn't ever notice. I was oblivious. It still can take me a while before I notice, like I spent all that time noticing the careful cultural neutrality in Civ V before it finally hit me there was no attempt at gender neutrality in that introduction.

My point? Given how blunt and ham-handed the stereotyped roles for females in video games are, and how long it took me to start noticing, even though I am a woman, is an indication of how poor I am (and I know most people are) at even noticing gender bias.

2010-08-01T10:43:00.000-07:00

I swatched for the Prince of Wales fair isle sweater recreation project (last post) before I even received the yarn I ordered. I used two skeins of Elemental Affects. I also decided to try mixing handspun with commercial fair isle yarn:

The smaller swatch is the one that includes handspun, pale blue, dark blue and dark green, along with the natural light and dark yarns used in the gauge swatch. I learned that it looks nice even when the yarn is not perfect. I also improved the tension on the floats.

Then I received the Jamison and Smith yarn from England:

First I noticed that the natural colour I'd hoped to use for the background was grey, and not at all fawn or beige. That's too bad -- the fawn colour was what I was aiming for. I didn't order enough of the fawny and beige colours to use much of any of them, although I did steal some "sh. FC45" from a Jamison and Smith kit I purchased in the same order. Since I don't want to order just a couple more balls, I'm going to alternate just one or two rows of the background colours at a time.

Since I wanted to use some handspun anyway, I blended an interesting roving with white to get another pale yarn option:

This looked really light when it was in a rolag, but on the bobbin and in the finished yarn, it got darker.

It's clearly the darkest of the light yarns:

It's also very close to the same darkness as the orange I need to use as a "dark" colour. The Prince of Wales painting has bright orange cartouche shapes on a fawn background as one of the peerie patterns, so the orange has to be considered a dark or pattern colour.

I wasn't getting far with using markers to choose colours, so I went to embroidery.

This worked really great. I was able to improvise a little as I went, removing two rows from the charted pattern to make it slightly smaller and make the big X's more visible. Click through to the Flickr image to see my notes on how I modified the embroidered sample as I went along. I also viewed the sample in black and white - I think the contrast is OK overall even though, or perhaps because, the contrast is not even. The orange peerie will draw attention because it's bright orange, the zigzag peerie will draw attention because it's got the darkest yarns, and the big motif will draw attention because it's biggest.

Next step, I believe, is casting on. Perhaps I should do another sample using the real yarns and testing the border ribbing, but I think it will be OK if I don't.

2010-07-22T20:53:00.000-07:00

Hang on folks, I'm taking this blog on a sharp shift of topic.

I decided a long time ago to knit a traditional fair isle sweater for a friend who's a sharp, if dated, dresser. I'm finally getting around to it: I purchased Alice Starmore's book of Fair Isle Knitting along with another couple of books, and found the next piece of inspiration I needed:

This is HRH the Prince of Wales in about 1921, painted by John St. Helier Lander. The prince is wearing a fair isle sweater. It's been recreated before, but I'm still enjoying using it as inspiration.

My friend lives in San Diego, so I need to make it a vest. My next step was to take his measurements:

And try to design the actual stitch pattern.

I wasn't very satisfied with the usefulness of using markers. It just feels so far from the way the yarn looks. At least I learned not to use very much light blue in the background, if any, because it makes the background much less fawn-coloured than it ought to be. I also found out how many rows the pattern would be, and at the row gauge I expected, calculated that I could get 5 vertical repeats of the pattern. That's not quite as much as I'd like so I may delete a couple rows (the top and bottom row of the large pattern.

I also ordered a bunch of Jamison and Smith yarn for the project. I ordered the Winterberry Ladies Slipover kit, as well as a bunch of extra pale colours. Oh and I ordered another kit to make for myself later.

As I waited for the Jamison and Smith to arrive, I purchased a couple natural-coloured balls of Elemental Affects yarn. There's not as many colours and certainly not as many heathered yarns, but if I need some I should be able to mix both brands. I used the Elemental Affects to make preliminary swatches, which I'll take pictures of for the next post.

2010-04-19T11:03:00.000-07:00

Should a constrained device be a RESTful server or a client? I had been assuming server, and I think I can justify this, although I'm not saying that choosing to put the low-power device in the client role is wrong, because it may depend on constraints and use cases. I do think it's a bad idea to require both client and server roles in the more constrained devices so I'm treating this as an 'either' choice, not and/or. Here is my rough analysis, as part 3 in a series on designing a REST framework for constrained devices (parts 1 and 2).

Roles: Reactive, storing, resource owner

To begin with let's separate the three roles commonly included in the definition of a "server". The first role: "a server is a reactive process" [Andrews, 1991] while a client is a triggering process. The second role: the server stores authoritative versions of resources. Most client/server remote file systems have both these roles in the server: the server reacts to requests and enacts storage operations, while the client is the triggering process making requests, but also controls the namespace has constraints on resource state. These client/server file systems scale moderately well but are inflexible and hard to extend compared to HTTP. In REST, the server takes on a third role: the server manages the namespace and resource state.

It's not clear how much mix-and-matching can be done with these roles in practice. Would it work reasonably well to keep the resource and namespace ownership together with the storage, but to make that agent also be the triggering process instead of the reactive process? I don't know examples of that kind of system in practice, nor do I know how to analyze a theoretical system against fuzzy goals like "flexible" and "scalable". But in the discussion of which roles to assign to the low-power device, I try to keep the three roles separate in case that helps shine light on questions like "should the low-power device proactively send requests such as notifications".

Benefits to consider

1a. Continuity: A server in charge of its own storage, namespace and resource model can be installed and host the same resources, responding to requests at any time, for years without changes. The conditions of its use can even change within limits. A Web site that, when launched, handles a few browser requests a day for certain resources, can later have some "mashup" service querying those same resources at automated intervals and extracting the data. Automated clients don't have this ability to be used in different ways without changing their configuration or code. Applying this to our low-power example: a sensor can handle requests from COAP gateways during normal functioning and by laptop-based clients during configuration, testing or development of new applications, without needing to know why it is handling any request (modulo authorization [1]).

1b. Flexibility: The flip-side of continuity is that features can be added. A device can be upgraded without disrupting the operation of the other low-power devices around it, because an upgraded device can host the same resources as the original device, plus new resources.

2. Scaling: In previous posts, I talked about how scaling large is related to scaling small, due to the relationship between power and load. The flip-side of scaling up to a large load handled with a fixed amount of memory and processing power (the normal problem for HTTP servers), is scaling down the memory and processing power for a fixed load (the scaling problem for low-power devices). In the Web today, what scales better, HTTP servers or HTTP clients? It's well-known that HTTP servers scale well, but there's little concern for clients scaling. It is hard to write a program that load-tests HTTP servers by implementing many clients over many connections -- all too often, the load-testing client machine(s) run out of resources before the servers do.

In HTTP, the server is stateless (see where Roy's dissertation describes "Client Stateless Server" and "Layered Client Cache Stateless Server" for the fuller picture), but the client may not be. The client needs to figure out which servers to talk to, what resources to request from the server, how to interpret those resources and what to do next. In Web browsers, it's the human user who makes many of those choices and may have mental state. An automated client might well include a state machine as the client attempts to achieve a goal involving multiple resources and perhaps even multiple servers. At a minimum, the client knows "What page I'm on now" and "what context did I make a resource request in" as part of its state. In contrast to the client, the server can be stateless, reactive, and doesn't need to know who it's going to communicate with as long as they're authorized [1]. With good application design, a server ought to be implementable with very little use of dynamic memory since it is stateless.

3. Naturalness of resource model: the most natural thing seems to be to model sensor readings and the capabilities of the most constrained devices as resources. Further, if the sensor is used for many years without being upgraded, those resources can have extremely stable URIs. The most natural agent to own these resources is the sensor. This related to the flexibility benefit because of the naturalness of extending the resource model. A sensor v1 can have it resource with a set of readings, and a new device with additional readings can simply have additional resources. A multi-purpose sensor can have all the resources that each single-purpose sensor would have.

4. Naturalness of user model: User interfaces are on the side of the triggering process or client as the user initiates requests for information or changes of state, and their client sends that out. It will be very rare for a human to push a button or otherwise interact directly with a sensor and cause it to act as a client. Much more common will be human requests on a laptop or control panel, which can act as a client and send a request over the local network for sensor information.

To conclude, I think these roles and benefits fit together sensibly for a particular kind of constrained device. I am probably focusing too much on a particular idea of a constrained device and its use cases, so I'm happy to admit that this is not the answer for all devices and use cases and would be interested in hearing analysis about other cases. I'm also getting some good comments to the past two posts, so at some point I foresee a post in this series just to discuss the comments.

[1] Authorization does not necessarily break any of the assumptions of statelessness. For example, authorization decisions could be pushed down to a lower layer in the simplest cases: the authorization to communicate with a sensor is the same authorization needed to request its sensor data. In more complex cases, the authorization decisions can be explicitly made above the COAP layer but this is more code and storage on the low-power device. In any case, this investigation is starting to make me think that authorization should either be a network function or an application function, not a COAP function or at the resource transfer layer.

2010-04-08T17:04:00.000-07:00

REST, with proper use of hypermedia, can be very appropriate for constrained devices. In my last post, I talked about how HTTP has a lot of cruft that could be removed if one were to design a HTTP-lite for constrained devices. The REST architectural style works for constrained devices a lot better than the HTTP syntax and HTTP feature list do. Naturally I'm working from Fielding's thesis, but some different points and emphases are warranted for the context of constrained devices interacting with automated agents, as contrasted to the user deciding which HTML link to click on to interact with a Web server. In order to dive into this, we'll need to really understand how REST requires using server-controlled hypermedia to advertise and navigate the server's resources.

REST involves

Stateless design. That means no state machine thinking! The client can't assume server state. [1]
A uniform interface on resources, similar to CRUD
Caching support for resource representations
Navigation to server-named resources via server-controlled hypermedia

The last point is often misunderstood. Despite the fact that we call HTTP client messages requests, it's far more common for programmers and engineers to treat them as commands or instructions (SOAP and RPC illustrate this very well). Many APIs designed to be used over the Web assume that the client decides what should be done and tells the server to do it. This isn't RESTful, and it's not just a quibble: putting the client in control of the interactions puts the client in control of the server's performance and ability to scale.

I didn't understand this when I first read Roy's thesis or when I first worked on WebDAV. I needed examples to make sense of the principle and its utility, particularly when the server does not use HTML and the client is not driven by an active user clicking on links. Thus, this post has examples of non-RESTful navigation and state discovery, as well as RESTful examples:

The Twitter API: not RESTful in resource naming or discovery
WebDAV: somewhat RESTful but made a few mistakes in extending HTTP
Atom: quite RESTful application that sits very lightly and comfortably on HTTP

The Twitter API has fixed resources. Those resources have URIs state which are known to the client in advance. For example, the account profile update resource is named "http://api.twitter.com/1/account/update_profile.format". This limits the Twitter API's ability to rejigger its namespace to add features or to scale by separating resources along a different axis. Next, the client must know a set of permanently-named parameters to be used in a POST request to the URI, to update the profile on the account. The programming model is clearly that twitter client controls the Twitter server by sending commands over HTTP POST. The idea of CRUD is vaguely there (many of the API resources can be retrieved in full with GET, others updated with POST) but diluted by the invention of special update URLs. Caching is possible, at least.

WebDAV is an IETF standard that follows the idea of a limited set of consistent methods well, but doesn't do resource discovery by server-controlled hypermedia. WebDAV set out to provide more authoring functionality such as functions to organize resources and interact with a file system model, so that Webmasters and other Web content creators could have something more suited to Web development than FTP. The WebDAV designers created the PROPFIND request which works a lot like most remote file systems file queries work. In other words, it puts control over querying server resources in the hands of the client. The client determines the scope of the request and the list of properties to be returned, and in the first implementations of WebDAV, the server had little choice but to comply or fail entirely. WebDAV servers had lots of code to handle all the different possible variations on the PROPFIND request, parse its body, and scale differentially based on the different characteristics of the properties that might be requested. All the information to reply to any PROPFIND request that the client might construct, had to be available to any server that might have that request made. Of course, this approach seriously affected performance and scaling.

Atom Feeds was the first standard I saw that did user-free resource discovery via server-controlled hypermedia. An Atom feed is a document offered by the server in XML format, containing links in semantic markup. So, instead of querying the server for blog entries that match a certain pattern, the client simply asks for a feed document via GET. With Atom, the server can precompute or do lazy computation of feed documents, can cache them, can break them down when a feed gets large. This design leaves the scope and detail level of collection membership documents entirely in the servers hands.

More examples can be found at a site that classifies APIs along a continuum of RESTfulness. My quibble with that site is that not all APIs fall along that linear spectrum. WebDAV would have a green box for RESTful identification of resources and a green box for self-descriptive messages, but it would have mixed results for manipulating resources through representations, because collections are not manipulated through representations. It would also have mixed results for using hypermedia "as the engine of application state" because the client makes assumptions about giving resources URIs based on what collection they were put in.

Clearly I need to work on a third post in this series and perhaps a fourth, because I would still like to talk about why constrained devices should operate in the role of the server and provide documents similar to Atom feeds about their state and data. I also have thoughts about a hypothetical framework and how it could be applied in a specific type of constrained device in a way that may not need any dynamic application memory. Hope you can stand to wait.

[1]I think part of what took me so long to understand this was the amount of meaning packed into the word "state". Not only does this mean "what state is the server in, among the states in its state machine", but it also means "what are things named" and "what things exist". So unpacking the injunction that clients shouldn't presuppose state, also means that clients shouldn't presuppose names or existence of resources.

2010-04-04T14:18:00.000-07:00

It's not entirely intuitive, but the same architectural style that allows Web servers to handle large scale usage and iterate so successfully, may also help constrained devices (like sub-10$-material-cost sensors) to handle small scale loads and power usage and avoid being upgraded for many years.

A lot of discussions of using REST in constrained devices have glossed over whether it's really interoperable HTTP that's being used (e.g. compressed HTTP) and whether the low-power device would be in the role of the client or the server. My opinion at this point is that for truly constrained devices, compressing HTTP itself would be the wrong choice, and that the device must be in the role of the server. The rest of this series explores why I believe that and should lay open the assumptions that might allow somebody to correct my information and revise my opinion.

The first part of the series attempts to explain is why HTTP is the wrong transfer protocol choice for some constrained devices. In theory it's a massively extensible protocol, with ways to extend the operation set, upgrade versions, add status responses, add headers, add resource content types and so on. Again in theory, almost all headers can be combined with each other and with different methods unless forbidden. In practice, the deployed base makes some HTTP extensions, and particularly the combinatorial expansion of extensions and optional features working together, quite difficult, and much special-case code must be written to handle the cases where widely-deployed software fails to implement some detail correctly. Here's a few examples of problems we've had over the years.

1. In developing WebDAV, we tried to use the standard "OPTIONS *" request (the special-case URI '*' means "give me the options or capabilities of the server"). However, we found that this is poorly implemented. Java servlets, for example, did not support the '*' URI. Other HTTP frameworks supported the '*' URI but did not allow the response to OPTIONS * to be extended by software extending the HTTP framework. Ugh.

2. HTTP has several different ways to indicate the content-length (historical reasons as well as for different use cases). One is the MIME type "multipart/byteranges". Another is chunked transfer encoding, where the server (and thus the client) does not need to know the total length until the last chunk is transferred. A third is "Content-Length". A fourth is for the server to terminate the connection, although this feature opens up the possibility of truncation attacks. Some of these methods work poorly with TCP connection continuation. Both clients and servers have to support almost all of these.

3. Both absolute and relative URIs are allowed in several locations, even though they're not a good idea in all locations. The resolution of a relative URI can be tricky and complicate client software. Implementations of this make it worse; e.g. the "Location" header was defined to allow only absolute URIs, but many implementations have been found which use their generic URI parsing or generating code to allow relative URIs in that field as well.

4. Parsing headers with quoting, separator (comma and semi-colon), whitespace and continuation rules is difficult. See http://greenbytes.de/tech/httpbis/issue-14.xhtml, http://greenbytes.de/tech/httpbis/issue-30.xhtml, http://greenbytes.de/tech/httpbis/issue-62.xhtml, http://greenbytes.de/tech/httpbis/issue-77.xhtml etc. Some headers defined in other specifications besides RFC2616 even used the syntax rules incorrectly and have to be special-cased.

5. The support for the Expect: header and 100 Continue response has never been good. In theory it's required, so there's no way for a client to ask if a server supports it, thus the client could end up waiting quite a while before giving up on the server initial response. Instead quite a few clients ignore the specification text and start sending their request bodies right after the headers including the "Expect" header, without waiting for the server 100 Continue response. This kind of feature also makes it harder to integrate authentication (what happens if the client uses this header when an authentication handshake needs to be initiated? ) and connection/transport features, as well as to implement intermediaries.

There are many more examples in a long issues list (http://greenbytes.de/tech/httpbis/index.xhtml), and some of the discussions of these issues get quite lengthy in considering how features are actually implemented and how they work combined with other features.

Developing a very simple HTTP client is simple, particularly if the client is talking to a single known HTTP server or is making only a small number of kinds of requests. Developing a limited-use HTTP server can be fairly simple too, although it gets to be significantly more complicated if the developer actually tries to handle all RFC2616 features and all unknown headers and methods properly. What turns out to be very hard is building an HTTP client library, server library or extensible server, because these are used in unexpected ways. It's easy for a developer using one of these libraries to break things, e.g. by manually adding a header that doesn't work with the headers added by the library. The library has to support using TLS and not; several kinds of authentication, several extensibility mechanisms and many failure modes.

The HTTP Components project at Apache talks about the many flaws and excessive simplicity of most HTTP client libraries, and states that "This is one reason why Jakarta, and other free and commercial vendors, have implemented independent HTTP clients". In other words, code re-use is seriously reduced by the way HTTP must be implemented. Some software companies are still selling HTTP client libraries to fill implementation gaps.

General-purpose HTTP servers -- ones that work for a large number of resources, a large number of requests, support TLS, content-negotiation, cache-control, redirect, authentication and other major features -- are even harder. When well implemented, HTTP server farms scale tremendously well. But much, much effort has gone into making those work.

When we look specifically at constrained devices, we see a much more limited set of use cases than the overall Web offers.

Documents are not large! Thus, document range and response continuation features are not desired. Only one transfer-encoding should be necessary, and conditional request features won't be worthwhile.
Documents from constrained devices are not intended for direct user consumption. There is no need for language and charset negotiation.
Even negotiating content type may be rare. A constrained device will state what it supports and not negotiate.
A constrained device will never act as a proxy, so need not support for Via headers and a bunch of status codes like 504. Further, if we define a non-HTTP, non-proxied transfer protocol that can be converted to HTTP at the first proxy step (or converted from HTTP at the last step), then the non-proxied transfer protocol doesn't need any proxy features at all.
Redirects are not necessary or could at a minimum be drastically simplified, making most of the 300 status responses unnecessary.
All of the 100 level informative status responses are unnecessary.
The authorization model for constrained devices is quite different than the authentication model assumed in HTTP (and associated logic like the 401 status code behavior is therefore unnecessary). Access control will be simpler than in current Web servers.

Imagine instead of a messy HTTP client library or server stack, we had a protocol with about a third the features, less extreme extensibility and simplified parsing. If well-specified and interop-tested early and often, I imagine such a protocol could be implemented in framework form in a tenth the size of an HTTP server framework. In even more constrained cases, a very specific constrained transfer protocol implementation (e.g. one which supported a subset of CRUD operations and did no optional features) could be 1/100th the size of a simple HTTP Web server. That starts to be an appropriate size for burning to a chip and shipping in a $1.00 to $10.00 device.

I tried to get some sanity checking on my estimates, and it's not easy. A Web server can sometimes be a wrapper on a file system, in which case most of the logic involved comes "free" from the file system. For example, a simple implementation of one style of redirect might be a trivial number of lines of code if the underlying file system supports symlinks. Andrew McGregor pointed me to the Contiki operating system for embedded objects, which has a Web server already. So that's a proof that embedded and limited devices can sometimes do straight HTTP.

In sum, if interoperability, flexibility and executable size are all simultaneous concerns, HTTP as-is will pose problems. While it may not be necessary for all devices to use a more space-efficient REST transfer protocol than HTTP, it may be necessary for some. If it is necessary to do a new transfer protocol, it should be possible to design a protocol an order of magnitude smaller just by cutting features and unneeded options from HTTP; and possibly smaller yet by being very careful about parsing and limiting extensibility.

More later on REST itself scaling down.

2010-01-03T15:14:00.000-08:00

A year in knitting: I knitted 6 miles of yarn in 2009.

Considering how much I spun in 2009, I'm pretty pleased. My stash definitely grew due to the spinning & gifts and purchases (5 miles purchased at Stitches West Feb 2009 alone, mostly not knit up yet).

1. Alpaca socks for Laurie (modeled), 2. fit of pathways 2 socks, 3. Zombiegurumi, 4. Vilai sock, 5. Prairie glass scarf, 6. Trekking sock, 7. Replacement travel shawl, 8. First handspun sock, 9. Amsterdam socks, 10. YAPS, 11. Panda wool retro rib, 12. Lavender shimmer scarf, 13. Petroglyph hat, 14. "Dove wing" icarus closeup, 15. Cowl from handspun., 16. pnw shawl on hangar, 17. zigzag fold, 18. Hawt sweater 1, 19. Malabrigo shawl.JPG, 20. Auction scarf 3 closeup

2009-11-20T15:49:00.000-08:00

Defining a new URI or URN scheme properly turns out to be really difficult. I've been sponsoring drafts for such schemes for almost four years, and the same problems come up again and again. The major themes:

Comparisons
Uniqueness
Stability

Comparisons

Comparing one URI to another turns out to be a commonly desired feature. Browsers look up cached pages based on URI comparison. If I click a link to bookmark in delicious.com, I'd like it to be bookmarked only once and if I've already bookmarked it, bring up that page so I can see what I tagged it with and when. Outside of http URIs, I'd still like to know if I'm already subscribed to an XMPP user, etc.

Comparison is harder than it sounds, but you already know that if you've dealt with any code requiring canonicalization, conversion or encoding. If a link in a Web page contains a space, at some point my browser has to convert that space to %20 to use in an HTTP request. Should the browser do that conversion before or after looking up the URL in the cache? Should delicious.com bookmark the URL with the space or the one that's used in HTTP requests? This is the tip of a very large iceberg that potentially includes all of internationalization and Unicode. Be very clear on what character set is used by each part of a URI, and if it's all ASCII, say so.

Case sensitivity is a frequent issue. Be very clear on which parts of the URI are case sensitive.

For new URIs, giving options makes the comparison job much harder. Let's say a new URI scheme needed to include a country designation: it seems nice to let users put a two-letter country code, a three-letter country code, a TLD or a OID in there. Only now one needs a horrid table to convert and compare these, and string comparison is no longer enough.

Optional syntaxes are similarly difficult; even allowing for '/' vs '\' can lead to error.

When the URI form is an alternate form for an identifier that already exists, now the URI may have to be comparable to something that's not a URI. For example, both IRI and URN forms exist for ISO OIDs. Don't they need to be compared to each other?

Can the URI form have query syntax? Is that part of the comparison or is that stripped off first? In HTTP URIs if I stripped off the query syntax I'd retrieve quite a different resource, but in some URI forms, the query syntax is used to carry information other than resource-identifying information. For example, can I compare two mailto URIs that have the same mail address, even if one of them has a query part with "?subject=Hey%20There"?

Can the URI form contain multiple values? The SMS URI definition had to include text on comparison when multiple SMS addresses were packed into the same URI. Does order matter to comparison?

Uniqueness

Frequently URI schemes need to avoid collisions, so that there isn't an attempt to give two different things the same identifier. The problem here is delegating the ability to create new URIs, while still avoiding collisions. The major fallback here is the DNS: URIs that contain a domain name, where the resource being named belongs to that domain, effectively delegate the uniqueness concern to that domain holder.

For example, we don't need to worry that "xmpp:lisa@jabber.org" will conflict with other resources, because the domain 'jabber.org' assigns usernames uniquely within that domain and prevents collisions.

The other main option is to use registries. For example, all OIDs, defined by ISO, use the ISO process to register numerical values and string values for use in the parts of an OID. Other times, IANA is the registry (e.g. for port numbers in HTTP URIs). If there is a new registry needed by the URI, this is more work and more to get right.

Some processes for ensuring uniqueness are quite heavyweight. Many IANA registries have processes which can take weeks or months to resolve. If the registrar is not IANA, who is going to actually run the registration process and under which rules? The OGC URN defined in RFC 5165 includes sub-namespaces issued by OGC itself . The first consequence of this is that the OGC organization must be referred to for any new OGC URN unless it explicitly delegates that part of the namespace. To reduce the burden of being a registrar in the case of non-permanent, test or experimental OGC URNs, the URN definition mentions the possibility of an experimental sub-namespace and the possibility of collisions within that namespace. Now implementors have to consider the possibility of leaked experimental names and dealing with collisions. The approval discussion of RFC5165 was lengthy, because of these nuances.

Stability

Some URIs need to refer to the same thing over only a short time, but typically the desired stable period is long or even longer. Domain names can be a problem here. Initially it might seem great to use HTTP URIs as XML namespaces, but consider whether the holder of the "example.org" domain will change over time, and whether the new holder will have the same policies regarding use and allocation of URIs in that namespace.

If a registry is used to achieve unique assignment, and the registrar is not IANA, then the stability of the registry must be considered. How long is the organization going to exist and maintain the registry publicly? We look for a public commitment, existing Web pages, a long-lived organization and so on. An explanation of the process and deciding factors for how names are assigned and how the organization ensures they are not reassigned, shows that they've thought about this commitment. See RFC 5328 for an example.

Random List of Gotchas

Does your URI scheme include use of fragment identifiers (like the #iri part of http://example.org/faq#iri)? Forget it; fragment identifiers relate to the media type of the *resource*, not the type of the URI. So if the URI "foo:bar:baz" retrieves a HTML page, then the fragment identifier would act like a HTML document fragment identifier.
ABNF is hard to get right. Get it reviewed by an expert. Use a ABNF generator or something like that to test your instincts. Refer to existing productions where possible. One common issue is to use a separator like "=" between two constructs, and then define one of those constructs in a way that it includes the separator character itself.
Another ABNF/syntax issue is to accidentally use a character that has an obscure meaning in URI syntax, or is simply reserved.
URIs that contain phone numbers include a whole barrel of troublesome monkeys. It's so hard to get telephone numbers right, with variable length and special encodings, '+' prefixes, dashes or spaces, and extension numbers, that it's worth trying very hard to use an existing phone-based URI instead of defining a new one.
If query parameters are used, can key values be extended? Can new key value be defined by anybody? Do they have global meaning (like mailto URIs) or purely local meaning (like HTTP URIs)?
The "community considerations" section required for URN registrations is frequently misunderstood. What the IETF looks for in this section is an indication that the work done to standardize this scheme and allocate a new scheme or URN type will add value; that there is benefit to the Internet community and not only to a private consortium or private company.
If any reviewer blithely says "whyever invent a new URI scheme, use HTTP for everything!" just ignore them until they provide actual reasoning for this proposal.
Embedding URIs within URIs, or any syntax that is infinitely extensible, is asking for trouble.

Final advice; provide more examples of actual URIs or URNs than you think people will need. Along with an example, explain how that example would be assigned, derived, and if applicable, dereferenced.

References

Here are the documents and registries that govern the registration and syntax of new URI schemes and new URNs.

RFC3406:How to define a new URN namespace or "NID" or "Namespace Identifier".
RFC2141: URN syntax, or the syntax of URIs that begin with 'urn:'.
RFC3986: URI syntax, how to parse all URIs and URNs regardless of scheme.
RFC4395: Guidelines and Registration Procedures for new URI schemes.
IANA Scheme Registry: Existing registered URI schemes
IANA URN-NID registry: Existing URN Namespace registrations

2009-06-22T16:06:00.000-07:00

I remember reading Neal Stephenson's Snow Crash and his description of the Metaverse, his conception of virtual reality and online communication, thrilled me. I knew in many ways it was more realistic than Gibson's cyberspace. For instance, Stephenson described how people can choose their own avatars and it's a sign of a newbie or at least a non-programmer to have an "off-the-shelf" avatar, and indeed we see this in places from static online forums all the way to Second Life

One thing nagged at me back then: Stephenson realized that there's no reason not to teleport in virtual reality, but explained that the programming rules forbid it.

You can't just materialize anywhere in the Metaverse, like Captain Kirk beaming down from on high. This would be confusing and irritating to the people around you. It would break the metaphor... Once you have materialized in a Port, you can walk down the Street or hop on the monorail or whatever.

This is unrealistic in a virtual reality which is supposed to be the predominant way hackers like Hiro interact with each other online. Today, online gamers tolerate some limitations on teleporting in game environments like World of Warcraft or Puzzle Pirates, but even there, friction caused by do-nothing travel time is minimized. And in a more general communication milieu -- Web forums, Facebook, Twitter -- there isn't a single, limiting place. I can "be" on two forums at the same time on Ravelry, open two or more Facebook windows and chat with multiple people and I'm "there" with them all for some value of "there". Not only is there the ability to go immediately where I want to be in most online fora, but it doesn't even involve leaving the other "places" I already am.

Ok, here's another piece of the picture that didn't bother me in 1993 but does today:

Most avatars nowadays are anatomically correct, and naked as a babe when they are first created, so in any case, you have to make yourself decent before you emerge onto the Street... [Hiro sees] A liberal sprinkling of black-and-white people -- persons who are accessing the Metaverse through cheap public terminals, and who are rendered in jerky, grainy black-and-white.

This assumes an architecture where the client renders their own avatar. Even in that architecture, a proxy for a public terminal could render a classier avatar. Low-res displays would more likely affect the receiver than the sender -- somebody accessing the online universe through a poor public terminal might see every other avatar equally low-res, but their own avatar could still appear fantastic to people on good computers. It's complicated.

I guess the lessons are that today's online fora are less like the real-world than we could imagine fifteen years ago, and future online fora are less like the real-world than we are yet capable of imagining. We're still sending messages that look like paper mail and have envelope icons, and we still think of "bulletin boards" as a real model. We haven't integrated IM or twitter-like experiences fully into other experiences. Today, I'm downloading the Adium beta to see how twitter "what I'm doing" messages and community are integrated with IM and whether that improves on the old IM concept of presence in a significant way. Trivial interface changes in these sites and software can be significant in how people use them.

To borrow Ted's analogy when we touched on this over coffee today, we're in the same phase cinema was in when a movie camera was pointed at a stage, and a stage play acted upon it: the unique affordances of cinema weren't discovered immediately and are still being discovered even today. With online interaction, we're only beginning to discover how different it is from experience in the physically-limited real world.

2009-06-17T09:43:00.000-07:00

I have a bunch of baking books, or cookbooks that include serious sections on baking. The Joy of Cooking and The New Best Recipe are my favourites by a long shot, and often I enjoy making the "best" scone even if the recipe is basically white sugar, white flour and a pound of butter.

However, sometimes I'm looking for a healthier scone, muffin or coffee cake -- something that I can eat for breakfast without too much guilt, or offer to health-conscious friends -- and I don't have resources that are just right for me. Ideally, a book on healthy baking would balance out a number of factors without being fanatical on any one of them:

How is the whole grain content? Can some of the white flour be replaced with wheat, or can the recipe handle an optional addition of wheat germ, ground flax seeds, oats or so on?
Can the sugar be cut down and/or replaced with honey or maple syrup?
Can the fat be cut down without sacrificing moistness, shelf life, texture and flavour?
Is the protein ratio good? Is substituting soy flour an option? Adding nuts?
Are the ingredients readily available or can rare ingredients be optional?
Is the taste pumped up? I eat less of pure, dark chocolate or tongue-tingling ginger sweets because my palate is satisfied earlier.

I understand some people get fanatical about one thing, just the sugar or fat or whole wheat content to a recipe, but I rather think balance is important and certainly taste is.

Along these lines, here's an adapted recipe for Mango Chutney coffee cake, derived from Light and Easy Baking. That book focuses only on reducing fat content, which I brought up a little again, but the pumped-up taste is there and the hot pepper is surprisingly good.

2-1/2 cups all-purpose flour
3 t. baking powder
1 t. salt
1/2 c. sugar
1/2 c. brown sugar
1 c. milk
1/3 c. canola oil
1 egg, slightly beaten
2 T. orange marmelade
1/3 c. raisins
3/4 c. chopped mango chutney
Additional pepper, cinnamon or cardamom, particularly if chutney is mild

Mix the dry ingredients together then mix the rest in. Bake in a loaf pan at 350 for 65 minutes.

2009-05-20T09:37:00.000-07:00

Quick mommy blogging, just to say I'm still here.

A couple days ago I hear a scream and "Get it off me!" from the two year old in the next room. I go running. It's a piece of sticky fluff on his index finger from him poking under the furniture. Sarcasm kicks in but doesn't work:

Me: "Oh noes! It's a disaster!"

Him: "I has a zaster on my finger!!"

2009-03-23T22:00:00.000-07:00

We had a really great IETF APPs area meeting today. We invited a whole bunch of people to talk about their topics and moved through the presentations quickly. These topics were:
- HTTP Resource Discovery
- Service/server Discovery
- Timezone publication
- SCRAM (Salted Challenge Response Authentication in SASL)
- Bayeux and cometd: JSON pubsub over HTTP
- BOSH for tunneling XMPP over HTTP
- rHTTP: reverse REST
- Analysis of all these server-initiated HTTP schemes plus Web Sockets
- Massive Multiparticipant Online Experience: the Overview

The slides are all already available here. Here's my favorite, a slide from Mark Lentczner's deck, for tying a whole bunch of things together. Even though Tufte would probably cry as Mark said.

2009-02-05T16:23:00.000-08:00

I'm trying to read some code in Objective-C. This is hard, but it's solidifying my abstract understanding of programming languages. I'm not that hardcore a programmer, but I guess I've picked up a few things over the years (gawd that makes me sound old).

One of the neat things about Objective-C is that using a class or instance method involves sending a message. C++, in contrast, calls those methods. A C++ object has a fixed number of methods that can knowably be called. An Objective-C object might be able to handle arbitrary messages. This makes some things harder and somethings easier: polymorphism is easier; finding cases of using the wrong type of object or having a null object are harder to detect (must be done at runtime, not compile time).

The thing is, this is very familiar to me because this is how wire protocols work. In fact Objective-C has "protocols" which are interfaces, or a set of messages, that an object claims to be able to handle, so the terminology overlaps quite a bit. Anyway, in a wire protocol the client sends a message, and because anything can happen to that message, the client has to be able to handle a large number of outcomes. Polymorphism? You bet; a server that appears to be a HTTP server (implements the HTTP protocol) can also be a WebDAV server, a CalDAV server and an FTP server.

Designing protocols can be hard for people who think in terms of fixed interfaces à la C++. RPC-style protocols embody this thinking, making Remote Procedure Calls and expecting predictable, limited results. It makes more sense to me now, that RPC-style protocols are so brittle: designers and implementors are acting as if there's compile-time checking of the remote interface, whereas since the remote interface is on somebody else's computer that may have been upgraded or may just have a different implementation, of course there's no compile-time checking.

2009-02-01T20:24:00.000-08:00

Mommy blogging today: Natasha said I should post about toddler sleeping stuff.

I have a kid that naps and goes to bed willingly and easily. Clearly there is a huge part of sheer luck in this because I've seen little correlation between loving, wise, firm parents and perfect kids, and I'm not always wise and firm. But we did luck out on a few things that have added to his personality to make for the easiest bedtimes ever.

We introduced an attachment toy early on. This toy, known here as "sleepy bear" because his eyes appear closed, is a blanket-with-head style minky toy.
We taught the sign for bear early on (bear hug yourself with arms crossed, and scratch your upper arms with each opposite hand) so he could ask for the toy pre-speech.

We attached a pacifier to sleepy bear with a folded strip of fabric. The fabric loop goes through the pacifier loop and around the pacifier, so it can come off for easy machine laundering of the toy.
We bought a second identical sleepy bear (and attached another fabric loop) when it became clear this was the favourite toy. Usually one remains hidden to be brought out very conveniently when there's contamination events or simply the bear has gotten too dirty from grubby hands.
Since sleepy bear is always "sleepy", he has to stay in the kid's bedroom most of the time. We make exceptions for when he's sick or at difficult times like coming home in the car after bedtime.

So now, when we say "It's bedtime" his response is (whining) "Nooooo...." but the next phase is "Let's go find sleepy bear" and he responds "OK" and follows us to the bedroom. Sleepy bear is closely associated with sleeping and triggers him to lie down and relax.

He is too young to say recognize fatigue and say "I'm tired, I'm ready for a little nap" but he's easily old enough to ask for bear. If we're at home and his hands are clean, we ask him to go into his bedroom and cuddle with bear until he's ready to come out without bear (and if he comes out with bear we bring him back to the bedroom and ask him to say bye to bear before he can go out and play again). So yesterday morning he did this and actually fell asleep, getting a bonus morning nap which he doesn't usually need any more.

When we want him to fall asleep in a new place (traveling or spending an evening at friends') we just bring sleepy bear. We pull out the toy and any old blanket, and that's enough for the kid to sleep in a new room fairly easily.

Finally, this somewhat limits pacifier use without ruling it out entirely. I don't care too deeply, but there's something annoying about a kid that talks through a pacifier all the time. Having the main pacifier attached to a toy limited to the bedroom means that most of the time when he's playing he doesn't have one. We do have a couple extras on leashes for carseat or stroller travel where it appears to seriously improve patience levels.

I don't mean to give advice because all parents are different and all kids are different, but I did agree it was worth explaining how this works for us. Good luck!

2009-01-27T15:14:00.000-08:00

Messaging Architects put out a very nice press release about my joining the company. I used it as a bit of a soapbox to talk about what makes a fully Open Standard: free to read, free to implement and free to participate in.

2009-01-22T11:38:00.000-08:00

I now work for Messaging Architects. I started a couple weeks ago but it's been busy; I traveled to Montreal last week to visit HQ and meet the management team.

It's going to be a fun job. The company is smallish (small enough for everybody to be on IM and see each other) but growing and building its product line. The M+Guardian product does spam control and other policy enforcement on email in transit, and I can definitely get behind spam control. The M+Archive is a bread-and-butter product for any company that has to follow regulations on email retention, which is a growing number, and I like the focus on swift retrieval. The M+NetMail email server was aquired from Novell a year ago and the company is now putting its stamp on the product (the team in Utah, who I met last October). In addition there's calendar integration, which you know I'm interested in, and possibly some file-sharing technology.

There's a lot of attention to customer needs at Messaging Architects, and a lot of enthusiasm and dedication. It's not hard for that to rub off on me even working from my own home! I'm in the midst of establishing a more fixed and attractive working spot at home, getting on IM with all my co-workers, joining regular meetings and getting the products running myself. Of course, I'm context swapping this new stuff with ongoing IETF work as I continue to handle the Applications Area Director responsibilities.

Thank-you to everybody who was looking out for me during the job hunt and if you're hunting, may you be as lucky as I.

2009-01-09T10:01:00.000-08:00

My fellowship at CommerceNet just came to its expected end -- CommerceNet hires fellows for limited periods, to encourage them, rightly, to get cracking. I figure the fellowship was 75% successful.

Half my work was to continue as Applications Area Director for the IETF, which I continue to do. I got more efficient, I continue to learn a lot, and I even managed to publish a few documents (on consensus processes, interoperability testing and reporting and HTTP Email) that might lead somewhere. The HTTPBIS, IDNABIS and ALTO WGs all got launched. I helped 40 documents become RFCs. It's hard to measure success because there's always more one could do.

The other half my time was even more nebulously defined: I was to build a new venture or otherwise do stuff that would either reflect very well on CommerceNet or be a good investment. I wrote a paper with Rachna Dhamija, another Fellow, before she moved on to launch Usable. I worked with Phil Crosby to build a prototype of a sleep tracker Web application, and with Jeff Lindsay to build a prototype Web site for making public health information more accessible.

By the end of 2007, I focused on the public health information accessibility site. I built a short-term development plan and budget (or actually, several) and a long-term vision. The vision included supporting research by letting scientists share information with each other, benefiting from the same visualizations and data transformations that are required to make data accessible to ordinary people. The site was also intended to be highly contributory in the long-run, so that data collectors (whether government departments, academic or industry researchers or private organizations) could publish their data in this accessible forum.

In early 2008 I still had very little budget so I decided to build the live site myself. I brushed off my Python, taught myself Django, and wrangled with databases, graphing packages and CSS stylesheets, then with running a Linux server. I even did a logo myself. In the summer I launched openfindings.org.

During all this time, I presented and demonstrated to everybody I could meet: at over sixty people, that was more than one demo a week. I was looking for partners (I actually hate building stuff alone), investment, grants or a home for the project. This is where I signally failed; although I often saw enthusiasm and expressions of support, I didn't manage to get enough concrete support to keep the project going.

I hope the ideas make it out there, though. There's no excuse for taking public money to create vast collections of public data, and then make the public interface as bad as this, or this, or this. Even CDC has daunting Web forms and codes to know. Users need to be able to discover data in an exploratory way, learning about it as they go, rather than be forced to know all about the data in order to know how to query it before they can even see any of it. Can you do that? Yes. Users can browse topics and see thumbnails of data visualizations, filtering as they go, never having to fill out a Web form or learn an ICD code. Faceted browsing and rich interlinking of related topics/graphs (more than I was able to implement on openfindings.org) would make data browsing even easier and richer.

So that's the summary of what I've been doing "at work" for the last two years. I expect the next few to be interesting too of course!

2009-01-05T14:12:00.001-08:00

Crafting in 2008

This mosaic only shows finished items, though the photo wasn't necessarily taken of the finished object in all cases.

1. Dumbledore top in progress, 2. Baby's initial on baby toy, 3. Outer stripes, 4. chinese yarn socks, 5. Anna socks modeled in mirror, 6. purple Fair Isle socks, 7. crochet fruit, 8. bigass spider, 9. limited-edition-rib-2, 10. color-chevron-scarf, 11. angora-parallelograms, 12. Souvenir of Ireland almost done, 13. Logan River Scarf, 14. Cell-automaton-full, 15. Cable-trim-neckline, 16. Primero scarf detail, 17. Riverbed sole, 18. Essential Stripe from kit, 19. Duet mohair wool beret, 20. Duet mohair wool mittens, 21. Diasilkombre progress, 22. Diakeito Diadomino yarn, nearly two balls already knit up, 23. Rainbow tie, 24. Faux laceup foxglove socks, 25. Rainbow spiral socks

# started in 08: 27
# finished in 08: 26
# given away: 16
# quilts: 3
# sweaters: 1
# pairs socks: 7
# scarves and shawls: 9
# stuffed toys: 3
# things for Darwin: 2
# made from old (>2 yrs) stash: 10

Based on the list of completed projects, I'm inspired by new yarn, old fabric and by having a deadline or recipient!

2008-12-14T19:05:00.000-08:00

This little triptych makes me happy... I love wearing these vivid purple socks, but they're nearly five years old. Luckily I still had the yarn leftover from making them, and the soles of handmade socks consist of "stockinette stitch" which is easy to darn. Done! I plan to get at least another five years out of these.

2008-12-08T13:48:00.000-08:00

The HTTPMail proposal I mentioned here now has a discussion list. I've already received a few good comments, building a list of changes to make when I go back into editing the draft. However, I'm in no hurry to do so. People are still discussing¹ what they think it's for; which use cases they believe in or not, and which use cases I forgot to mention. That's a good discussion to have before nailing down what features it has to have.

[1] discussion so far on the general Apps Discuss mailing list; discussion really hasn't started up on the special purpose list yet.

2008-12-05T14:36:00.001-08:00

This is pretty exciting news from Amazon: they're launching a service to freely host public data sets. Part of building openfindings.org was about that problem, but Amazon can do a better job of storage than a smaller organization can. Problems still to be solved:

finding the right dataset -- somebody needs to do data-focused searches on terms like "lyme disease" or "lung cancer mortality", and this probably needs a bit of ontology work
automatically generated visualizations appropriate to well-known kinds of data
Wikipedia-style annotation, comments, highlighting: people not only mining and analyzing the dataset in private, but also in public and benefiting from each others work

Add this to the long list of data-oriented Web efforts in the last two years (palantir, swivel, flu trends, flowing data to which I owe a hat tip, and more), and it's the hot thing calendaring was a couple years ago when eventful and meetup were launching. Financial services were the first I saw to do sophisticated dynamic online visualizations of data, followed by Google Analytics, and it's only starting from there.

2008-12-01T16:12:00.000-08:00

I'm following up on my last rant on multiple transports. It's a topic with nuance, but I still think it's a worthwhile caution to avoid multiple transports in non-realtime standard protocols.

Yngve talked about multiple transports used by Opera Mini for cell phone Web browsing. This is in the context of what appears to be a proprietary protocol; at least it's not standardized. Opera Mini talks to a transcoder run by Opera, so interoperability is mostly handled by not having multiple vendors of the client or the server. Skype was another example, but again Skype does not use a standard. More generally, Yngve talked about TCP-based applications that need to bypass firewalls and can use HTTP as a tunnel. Yngve, you're right about the general world of protocols, but what do you think about broadly standardized protocols?

Ralph and Joe pointed out that there are multiple implementations of BOSH, the HTTP binding for XMPP. This is interesting; I was aware of the proposal but had no idea how widely adopted it is. If it really becomes universally supported in servers, then we could write any new application on top of XMPP and it will work over HTTP as well as natively over TCP. Now that's interoperability!

2008-11-24T14:35:00.000-08:00

Herewith, a rant: multiple transport bindings considered harmful.

I still hear this and it's getting increasingly annoying. People consider it a good thing for a standard to be able to run over HTTP, BEEP and something else. Has this ever proven to be a good idea? Layering is good for other reasons, but not because it gives implementors a choice that leads to interoperability failure in many cases.

Is it a failure on the part of the designer to understand the usage characteristics of their protocol, and successfully map that onto TCP (connection-oriented), HTTP (stateless respond-and-forget) or something else?

SOAP is supposed to be transport-independent and offer choice, but as I overheard last week, there's a reason they call it web services. And the ultimate in multiple-transport wankery: I once heard somebody propose a schema which they said would run over SOAP or HTTP.

Are there use cases I'm unaware of, where this has been a really good thing for some standard?

2008-11-12T10:44:00.000-08:00

I'd like to be able to explain the important confluence of several ad-hoc Web standards to various people, so I paid attention Monday at IIW. This is the diagram I saw:

I can't call this a stack. I just can't. It's not a protocol stack or an API stack or a library stack. The best I can say is that "Some people call it a stack". There's no layering relationship between these things; there's not even a sequence or an order.

PortableContacts extends Open Social and the rest are independent proposals that work well together. It's a suite, a cluster or a collection of protocols/APIs. Here's a first stab at a moderately simple diagram that does not mislead:

Comments?

2008-11-09T12:50:00.000-08:00

The idea of using native HTTP resources to RESTfully access an email store is not only an old idea, it's been implemented many times. Some Web mail architectures are even somewhat RESTful although the purest implementations are not themselves Web UIs -- these RESTful email interfaces have typically been built to support Web UI frontends in a classic tiered architecture.

I have been talking about this for so long that it seems self-explanatory. Emails can be HTTP resources with persistent URLs and machine-readable representations. Mailboxes can also be HTTP resources with persistent URLs and machine-readable listings of contents, we just have to agree how to represent those listings. Here's my proposal.

Atom feeds is my choice for organizing those listings. I think it's worth explaining the two overriding reasons why.

Atom allows clients to just GET a representation of a feed, and the feed can contain an arbitrarily long list of items, but paged according to the server's needs. This is a great allocation of responsibility, making client logic simple and putting server performance in the server's hands.
The default model for feeds is that the same object can be in more than one feed. This is very important for the usability of email going forward. I know a few people who organize their mail into strict hierarchical collections, typically using IMAP, and find old email by jumping to the right place in the hierarchy -- but I know far more who rely on giant inboxes, saved searches, flagged email or tagging. The default model for feeds directly accomodates all those usage models.

I was very happy to find once I used these principles and applied the feed model, that it fits very well so far. It's possible to implement the spec as a proxy between an IMAP server and a client. It's possible to access a mail feed with an unmodified Atom-capable newsbrowser. This bodes well for implementing and deploying.

2008-10-17T07:50:00.000-07:00

There's lots of ways to amuse even a toddler under 2 with a computer. Browse through cuteoverload or icanhascheezburger. My favourite is to find a Flickr group like Animal Parent and Offspring and browse through saying what kind of animal and baby we're looking at.

On Youtube, one of the earliest interesting things, to a baby, is videos of other babies laughing or babbling, and other parents have uploaded a bunch of those. Of course there's more cute animal videos. Elsewhere, there's kids music videos by TMBG and others.

I'm sure there's lots more but I'm not usually so strapped for ideas that I turn to a computer. I have a sore throat and don't want to get up from the couch!

2008-10-06T13:50:00.000-07:00

Two IETF tidbits to pass on.

1. An OAuth BOF was approved, and OAuth was submitted as an IETF I-D as fodder for that BOF. I'm excited about this; we at the IETF have been rather floundering around, without much consensus or sustained progress. Two years ago (already!) at the WAE BoF we identified some possible steps forward but there was still much disagreement over basic approaches, both technical and architectural (integrate SASL into HTTP, make requirements for TLS Client auth in HTTP, invent something new for HTTP). Since OAuth is already implemented in several places, perhaps this will leapfrog us past the open-ended "what shall we build" discussions.

2. ALTO is a proposed WG for "Application Layer Traffic Optimization": focusing on P2P file sharing, how can the network provide information to peers to help them pick peers in a way that impacts other users the least? The ALTO charter was sent out for IETF-wide review, and I'm optimistic about its formation as a WG. I've requested a BOF meeting slot, but hoping to turn that into a WG meeting slot.

2008-09-30T21:15:00.000-07:00

This was roughly my bike route today to get to a meeting in Redwood City and back. Apparently about 17 miles, on top of my 6 mile bike between home and the office.

View Larger Map

2008-09-30T08:42:00.000-07:00

One of the best articles on DRM I've read in a while appears on Penny Arcade, and contains the words "DRM takes a big poo on your best customers". Actually, the full sentence is worth including:

DRM takes a big poo on your best customers -- the ones who've given you money -- whilst doing nothing practical to prevent others from 'stealing' your precious content juices.

Yeah, that's right, punish your friends and give everything away to your enemies. As artists and creators, you do deserve to make money. Perhaps there are other ways than punitive DRM, however.

I'm not into protesting games that use DRM for protest's sake, although I did avoid buying Spore because of its license checking. I just happen to like games that I can play when I like and pay for when I like. I've given money to Puzzle Pirates, which the author of the article is associated with, because the game let me have fun first before paying. Ikariam is similar but taunted me too much with the for-pay goodies so I found that pressure off-putting. Kingdom of Loathing has the best laid-back attitude towards paying and playing. Not only can I take a break without getting punished, they treat my contributions as valued donations, so I feel good when I toss them some meat.

2008-09-25T16:06:00.001-07:00

I've been asked a lot about use of SOAP in IETF standards. Most of the time, I think it's overkill and poor for interoperability, because it has a bunch of features that probably aren't needed, and if they are used, will not work unless everybody uses them the same way.

While using HTTP directly suffers from the same problem, at least the problem is not doubled by using SOAP over HTTP. And HTTP benefits from fantastic tool support and connectivity. So without further debate about whether HTTP itself is a bad or a good thing to use, here's how to use it in a standard.

Define your objects, these will be HTTP resources. These are the objects people operate on. In Atom, these objects are entries, feeds and service documents. In CalDAV, objects are calendars and events. In a protocol for transferring karma between people, resources might represent individual karma accounts.
If you can't think of objects but only of services, you may still be able to use HTTP POST to send requests to the service handler and get a response. I guess you're treating your service handler as a kind of object but it's only minimally RESTful to do this.
Don't bother defining the URLs to resources. They'll be HTTP URLs of some kind. Let servers choose and clients discover.
Define operations on your objects/resources. To create a new one, you should be using PUT (if the client can reasonably construct a new URL that the server will find acceptable) or POST if the server will pick a URL. To remove an object, use HTTP DELETE. To update an object by replacing its content, use PUT.
Some commonoperations are defined in other HTTP extensions already. To update an object by making a change to it, use PATCH. To move or copy an object, use MOVE or COPY.
Define how clients will find resources. E.g. Atom has service documents which locate feeds, and feed documents which locate blog entries. Because of Atom's strongly document-oriented approach, this means Atom uses GET requests for navigation and resource discover. CalDAV, by contrast, uses WebDAV collections for containers, therefore uses PROPFIND for the same purpose.
You may find in this part of the exercise that you've invented new objects to help navigation and resource discovery work well. Reexamine your object model.
If you need permissions, use the WebDAV ACL model if possible. Hold your nose if you need to. It's better than building one yourself. Leave permissions out entirely if you can wait a revision.
Figure out if you've got any magic or use cases that aren't handled. E.g. CalDAV needed some magic to query calendars to find out "what events occur or reoccur this week". That wasn't solved with a RESTful PROPFIND request, so CalDAV defined some magic and used the REPORT method to carry the request for magic. Define your own new method for this one if you're dying to exert some creativity.
Figure out the details. There are a fair amount. For example, define whether resources have fixed MIME types; whether caching is appropriate, whether ETags are required. It's probably a good exercise to walk through the section headings of the entire HTTP spec and say "Is this required to support, required not to use, or does it interact with our new protocol in some funny way."

---

Ok, I wrote this quickly to get it out and I'd love feedback. There are probably other recipes; this is the one I use. Exceptions exist. Void where prohibited.

2008-09-08T14:49:00.000-07:00

DjangoCon 1.0 was great, and now I've got the T-shirt, as shown at left (one of 100 printed -- in a real female size, no less, thanks to Leah.) The photo is from my lightning talk wherein I introduced myself and gave 10 reasons why people who use Django might be interested in IETF and how I could help. Also I'm tickled because the photo was taken by Simon Willison, whose blog I've been reading ever since I started using Django.

It's always nice to meet people with common interests, people I've gotten some awareness of online, but it was especially lovely this time. As Cal Henderson (shown left, pic by webmonkey_photos on flickr) pretty much said in the funny roast-style keynote entitled "Why I Hate Django", the core Django people are just too damn nice.

I also had moment after moment of future shock, just thinking about how many more tools we have for self-organizing as groups of people. Blogs, forums, voting widgets, comment fields, shared calendars and shared docs, collaborative bookmarking and tagging. Yes, none of these are new to Django but Django makes sites with these features very easy to create. I got started on dial-in text-only bulletin boards, people -- we're talking 28 kb modem -- consciously moving my reference point further back than it normally is. And on the other side of the coin, Django is such a useful project because of the online collaboration of loosely connected individuals. That's some heavy bootstrapping there.

2008-08-27T11:23:00.000-07:00

I am going to attend the first DjangoCon in a couple weeks. Sorry if you have heard about this too late, all the spots are taken. The Django 1.0 release party is still open, however.

Django has been great to use. It's a huge leap from how I used to have to do data-backed Web sites. It does the same things Rails does but for a Python environment, which I prefer.

I look forward to meeting other django-heads for the first time since I started using it. I'm sure I can learn a lot.

2008-08-20T09:49:00.001-07:00

Warning: Knitting content has returned, because I need a place to post this picture.

Claudia asked for solutions to keeping sock knitting on the sock needles while carrying it around, and my favorite solution is a little fabric pocket. I measured the fabric an inch wider than the length of the sock needles in question, on the bias so that there would be stretch. Then the pocket is sewn up as shown in the diagram, with extra seams to keep the sock needles from popping out. The knitting hangs out the opening of the pocket, with the flap to fold over (probably optional).

I did my own two pockets in cotton and silk, finishing the raw edgings with a little rolled hem. A super fast hemless solution would be to use a small square of slightly stretchy fleece fabric, which needs no edge finishing.

The reason this works as well as the point protectors is that the hole on the pocket is smaller than the needles are, so it holds the knitting -- remember, the knitting remains mostly outside of the pocket -- narrowly on the center of the needles. It's the same principle as the sock knitting tubes sold here but cheaper.

2008-08-19T14:44:00.000-07:00

Google announced support for the CalDAV standard, and explained how to view a Google calendar in Apple's iCal client. There are a few glitches but it's a developer-focused release and they've solicited bug reports.

I would be very interested to know how their WebDAV logic (underlying CalDAV) plays with their Atom logic (underlying the GData API). I'm sure Google Calendar is not the only implementation to support both protocols to view and modify the same data, and there will be more.

2008-06-25T09:21:00.000-07:00

Biking around and trailing the toddler is all very well, until you hurt your lower back pushing those pedals, and then have to pick up said toddler. And when I say "you" here, I mean "me" (don't you love the vagueness of the impersonal second person, or "generic you").

The resident expert advised me to pedal faster at lower gears, and to spin the pedals around rather than push them down. Apparently RPMs of more than one per second are common for real cyclists. I started saying "Left, right, left right" to myself to try to get a feel for this rhythm, and it seemed reasonable until I actually got on the bike again and tried to pedal that fast. It feels too fast! Will definitely take some getting used to.

While this may all be very logical and reasonable, it's evidently not what I'm used to. When we were kids we used to have durable, stiff bikes and we'd stand up on the pedals alternately to get moving, putting our full weight on each slow stroke. Besides, the seats were torn up anyway by mistreatment, so why even sit on the seat. I think when I was thirteen, or maybe even twelve, my parents got me a full-size men's mountain bike, figuring it was the last bike they'd buy me before I moved out. It lasted me through university too as I recall. Of course it didn't fit me -- frames for men are too long for most female torsos -- but we didn't know any better about that either.

2008-06-23T11:35:00.000-07:00

I recently scanned through Flow, by Mihàly Csíkszentmihályi, a book I first read a decade ago. It didn't improve on rereading -- it's not a bad book, but it has the kind of overblown promises I've come to recognize in a certain category of books by obsessed people. Greater world peace, less illness, better personal finances and fewer unwed teen mothers would result if more people learned to experience more flow. Another book in this category is The Promise of Sleep, by Dr. William C. Dement: if only more people got better sleep we'd see better health, less unemployment and many fewer traffic accidents. Any others?

My husband has been ranting about another category of books: books about a particular invention that "changed the world". The smaller the subject the funnier these are, and these are easy to find with an Amazon search:

The Machine That Changed the World -- about "lean manufacturing" as introduced by Toyota, basically a PR exercise for Toyota according to reviews.

Rome 1960: The Olympics that Changed the World

The Night of the Long Knives: Forty-Eight Hours That Changed the History of the World

Nixon and Mao: The Week That Changed the World

The Map That Changed the World: William Smith and the Birth of Modern Geology

The Gun that Changed the World -- about the Kalashnikov

Mersey the River That Changed the World

The Fibre that Changed the World: The Cotton Industry in International Perspective, 1600-1990s

The Box That Changed the World: Fifty Years of Container Shipping - An Illustrated History

Tea: The Drink That Changed the World

Banana: The Fate of the Fruit That Changed the World

Cod: A Biography of the Fish That Changed the World

2008-06-21T10:41:00.001-07:00

I have a yoga rant today. Why are yoga classes so seldom both strong and safe? By "strong" I mean including poses that take strength to hold, and that are held for five or more long breaths, such that the class overall makes one sweat and improves muscle tone. Load-bearing exercise is important for women of all ages for late-life bone strength, and men seem to like these types of classes better because they include some poses that men are good at.

By "safe" I mean that dangerous poses, ones that can harm the wrists or back or knees, are not as frequent and come with careful instruction and warning, and the teacher explains how to do the pose safely and also explains who should not do the pose. Some of the things I hear from yoga teachers I consider "safe":

Push the base of the fingers down into the mat in poses that put weight on the wrist; this engages the muscles of the hand to protect the wrist.

Remember to flex the foot in poses that open the hip by using the whole leg as a lever (e.g. the "knee to ankle" pose). Again this activates the muscles that hold the joint solid, rather than stress the ligaments.

"If this hurts in this way, stop."

Instead, the normal tendency when teachers focus on yoga that doesn't hurt, is to make the whole class easier; while the tendency when building a class for advanced or strong students is to do dangerous poses without teaching how to do them safely.

Anyway, I do know a teacher, currently teaching, who excels at combining strength and safety, although I don't have a schedule that allows me to take her classes right now.

2008-06-18T13:20:00.000-07:00

I am not experiencing flow doing Django Web form programming. Don't get me wrong, it's powerful and good, but there is something seriously messing with my ability to keep working. As usual, I need to look up Python and Django and HTML forms details every so often, but this is normal and I've been able to maintain focus doing other kinds of programming.

To build Web forms, I create a new HTML page and see that it loads, add a form element and a form field, see if it shows up, add a submit button, test that it works without being wired up, wire it up to a form submission URL and write a dummy handler, see that the dummy handler is called, replace the dummy handler code with real handler code. Each of these steps takes only a minute or two and after many of them I find I have to stop working rather than go on to the next step. It's worst when I have to figure out "what next" when I'm done with one whole form, at least for the time being. Perhaps it's the constant swapping between deciding "What am I doing" and "How do I do what I decided to do".

2008-06-09T11:46:00.000-07:00

This was my route and itinerary by bike last Friday:

I biked three miles to the Palo Alto train station, caught an earlier train than I intended, and arrived 45 minutes early for my 10am meeting at point B. I left there just before 11 am and saw two bridges and much waterfront on my way to point C, where I was 30 minutes early for my noon meeting.

I left C at 1:30 and was 20 minutes early for my 2:30 appointment at point D. Finally I left there at 3:30 and had a last appointment near A before biking back to Caltrain. This SF loop is 12.5 miles according to Google, and I optimized it to have very little hill-climbing but this is SF folks. There were some hills.

The caltrain ride home was very much slowed by a pedestrian accident past Palo Alto, so I got off the train at the unscheduled Menlo Park stop and biked home: 4 miles.

A total of 19.5 miles on bike, all in a skirt with my demo laptop on my back. I loved it!!

2008-06-04T21:05:00.000-07:00

Jasmin ordered photos of my red hair and I have complied. It's weird having a conversation via blog posts and comments, you know? I had a little fun with the shooting but was on my own with D. wandering around the patio so I could only do so much.

2008-06-03T14:48:00.000-07:00

I visited the Asian Art museum of San Francisco a few years ago with a good friend. We were struck by all the buddhist iconography embodied in the hair of painted gods: peaceful gods had softly flowing blue hair, while wrathful deities had fiercely curling fiery red hair. Like Jasmin:

After seeing all these fierce hairdos, my friend and I immediately went home and dyed our hair bright red. But Jasmin really rocks that look!

UPDATED: I didn't mean to imply this photograph is one I took, but I forgot to give credit for it. 'gretch_dragon' took the photo.

2008-06-02T11:14:00.000-07:00

Some news related to my role as an Applications Area Director, for a change. Last week was the P2P Workshop organized by Jon Peterson and Cullen Jennings. I didn't take any notes so I won't have many names to drop, but I learned a lot. In the morning we heard talks from operations guys at Comcast and by Stanislav Shalunov who works on one of the BitTorrent clients. The basic summary of each of the talks was that despite their differences they'd like to work together and with the IETF or for the good of the Internet. Comcast talked about their commitment to net neutrality and the costs/challenges of providing cable service -- neat stuff about how upload bandwidth is completely separate from download bandwidth, and BitTorrent is unusual among applications in using more upload bandwidth than most. Stanislav talked about all the logic they have in the client he works on to be careful about overloading the network despite having very little information available on how to ration network usage.

In the afternoon, I recall a talk on "Localization". I'm used to this meaning to translate software into another language, but in this case the meaning was to find a server that's "local" in network terms before attempting to do something. Leslie Daigle and Henning Shulzrinne also gave talks.

Some of the takeaways for the Applications area could be:
- Documenting the BitTorrent protocol in a stable document
- Standard on how to find an appropriate mirror or location for a resource that's available on multiple servers -- this is re-implemented all the time in the wild, with different confusing HTML interfaces on each implementation
- Standard on how to find a local server -- this could be a subproblem of the basic service discovery problem (given a particular domain), or it could be more sophisticated (not just one domain but based instead on network topology)
- More ways to find out network information -- this could include information on bandwidth available, bandwidth quota, local servers, caches etc.

I don't know exactly how followup from this meeting may turn into Internet-Drafts, WGs or other, so that's all the information I have for now and no predictions of whether people will actually volunteer to do any of the above. But even as is, I thought the workshop was useful for bringing people closer to the same page.

2008-05-21T15:40:00.001-07:00

About a year ago, I worked with Jeff Lindsay for a couple months. He worked on the demo for the project I'm still working on, and he set me up with a source code repository, wiki and ticket system: all an integrated part of his devjavu.com work. Since I hate maintaining these development services but must have them, it's a perfect solution for me. I recommend it.

BTW, today Jeff showed me how to give anonymous permissions to the source code repository, so I did. Here's what I'm working on :)

2008-05-15T17:12:00.001-07:00

I've been writing import tools (not quite screen scraping, but close). Today I used those import tools to grab a bunch of CDC data on causes of death. This can get a little depressing at times. Sometimes I notice a large number of people died of something I didn't even realize was a cause of death. Look at the skew to age in death attributed directly to dementia:

Other times, it's nice to note that hardly anybody dies of something (e.g. cannabis usage, such low numbers I didn't bother importing). Here's deaths due to all mental and behavioural disorders from use of psychoactive substance (and nearly all of them are alcohol):

2008-05-11T09:53:00.001-07:00

I took Zack Arias' OneLight Workshop on Friday. It was, on the whole, awesome, as many others have reported, with the personal exception explained below.

We spent several hours reviewing the science and the analysis of light, shutter speed and aperture. Being an engineer, I can pretty much memorize this level of technical stuff when it's presented as clearly as Zack does: words in my eyes (slides), words in my ears (him speaking), a couple diagrams, quite a few examples and definite humour. So far so good.

The next part of the workshop I was totally unprepared for. This was the part where Zack set up a lighting situation and a model, explained the setup, and then let everybody in turn use their own camera (but his light) to shoot the model. I was a nervous wreck and confused and embarrassed and disfunctional.

Half of my problem -- and both halves were my problem, not the workshop's -- was that I didn't know my camera well enough. I thought I knew how to shoot in manual mode, but while nervously fiddling to get that to actually work, I screwed up several other settings on my camera. It took me another hour after that to recover proper focus, because I could not relax and look and see what was wrong. I've used other fully manual cameras before, and I've shot with the Rebel in aperture priority or shutter priority mode, but I needed to know *this* camera well enough to be able to focus on other things besides the camera's button's and dials and screen. Ouch.

The other half of my problem was sudden incapacitating social awkwardness. I was completely bashful about working with live, volunteer models I'd never met before. I was intimidated by going before and after professional photographers who knew their cameras and who were joking comfortably with the models. I was embarrassed at wasting everybody's time, the models and the other photographers waiting to use the same light setup. I was too wrapped up in my own awkwardness to ask the models to turn their head, or move closer to the light, or up or down. Stupid, huh? They were lovely, friendly, patient models but I was completely flustered. I think I babbled incoherent apologies at them at the end of each "shoot".

As this part of the workshop continued into our third and fourth light setups, or about two hours after quietly flipping out, I started pulling things together. Several people definitely tried to be helpful and were in fact helpful (thanks Tara and of course Zack) but I was at that point where I was too self-conscious to be very effective at accepting help. I've seen other people in that position but haven't experienced it myself recently. It's a humbling reminder for when I'm in "helper" role of saying "Oh, just do it this way". I got my focus straightened out. I started adjusting my aperture in the right direction instead of consistently doing it in the wrong direction. I got to that moment where I decided to try to alter a shot by taking down the light on the background compared to the model. I worked it through: I remembered I had to increase the shutter speed to pick up less ambient light, while maintaining an aperture that picked up the flash on the subject at the same brightness. The result, while in no way a good photo -- I mean, not at all -- was at least the product of an intention that I formulated to make use of what I'd learned.

It all went uphill from there. I started to be able to make a plan and imagine a look and adjust what part of the background was in the photos and how much light it had. I really didn't take a lot of good photos but increasingly I was able to make changes that I thought of. I took notes of what I was doing to cement the principles.

At the end, I finally got comfortable photographing one of the photographers. This was in the part of the workshop where we split into groups and set up our own lighting. Tara setup a shoot-through umbrella near a wood fence, and Bärbel was the fellow photographer who sat for my shooting, as I sat through hers:

I was now comfortable enough to ask Bärbel to move away from the fence, to give more separation between the light on her and the light on the fence, thus more control at dialing one up and one down. I was comfortable enough to move the umbrella/flash stand closer to her to her. I was still trying to be quick and didn't take a zillion shots with different facial expressions or different compositions, but since I was there to learn the lighting part that was OK. And I got this shot, and I was happy.

Thanks to our hosts who provided excellent food and a location, and the models and fellow photographers and Zack. I was too dorky to act normally but if you happen to stumble on this I want you to know I enjoyed myself and learned a lot.

2008-05-05T12:47:00.000-07:00

I'm getting into geographical distributions and shapefiles now for the health data accessibility project. Ekr and Cullen got the basic R and maptools extension commands working for me. I tweaked it a bit and now have thumbnails and detail views (shown here, deaths attributed to HIV in white males):

One thing I've learned about freely-available shapefiles is that they have a lot of crap in them. Crap is, of course, in the eye of the beholder. In my case, I want the shapes of the regions only, and not the population, households, whites, blacks, males, females, density, age distributions, divorced, married, never married, single-parent household, household units, vacancies, mobile homes, farms, crop acres and more counts/rates, for every region.

I can probably get a shapefile editor and remove that stuff but it will only slim down the files slightly. Even bigger than all that data are the latitude and longitude of every vertex required to draw a state outline. Think coastline and all those islands in Alaska, Puget Sound, etc. Since my map sizes are limited anyway, I wonder:

how many vertexes I could afford to lose without losing any
resolution in the larger size, and how I could find a shapefile appropriate for that
how much smaller the resulting shapefiles would be
whether that would be any faster anyway

Performance is a problem right now! I can trivially do caching of drawn maps but it's at the point of slowing development down at times I do not want to cache.

2008-05-01T09:57:00.001-07:00

The code review I got from Grant last weekend was really great, but I'm back to programming solo. Luckily, one of the nice things about code reviews is that when you explain something, you are likely to see problems with it before the person you're explaining to even finishes parsing your tech-tech-tech. And you don't even need the other person to make this work if you're disciplined enough to read code aloud all by your lonesome. (I can read code to the cat at home, but maybe I need a teddy bear at work.)

So here's how I read some code to myself yesterday:


   Okay, first step here is we get a list of graphs to draw, it's a generator.
   We go through each graph figuring out which datafiles we need to load so we
   only load them once. 

   This we know is working so far because of print statements, showing two
   datafiles in this case. 

   Now the next step, we're going through the generator again,... oh.

Yeah, you can't use a generator twice in Python and I knew it, but I didn't remember it until I heard myself speaking out loud. What's up with that?

2008-04-25T11:40:00.000-07:00

Paul commissioned me to knit a tie-dye, Grateful-Dead-inspired, tie. I made it, he gave it to the UCSC Chancellor at a press announcement. Cool!

2008-04-23T10:41:00.000-07:00

Kids car seats are getting bigger (apart from the ultra-portable infant seats). Three reasons why, all related to new safety requirements:

LATCH (Lower Anchors and Tethers for CHildren) are anchor points in cars that make installing carseats much safer since roughly 2003 when they became standard in all passenger cars. Manufacturers of car seats have made them bigger as a result. The seat needs to accommodate a strap between LATCH points as well as routing for old-fashioned seat-belts. The tether for the top of the seat, which goes over the back of the car's seat back and fastens there, has caused the manufacturers to make the back of the carseat taller and have a little more room for hardware above the kid's head.
Around the same time as LATCH, the US has seen more stringent regulations for when car seats must be forward facing or rear facing. To save parents from buying multiple seats, there are convertible seats that can be turned around somewhere between infancy and toddler/preschool age, and still be used up to 60 pounds in some cases if the kid is not too tall. Of course, seats that can be converted from rear-facing and reclined positions to front-facing and upright positions have bulkier bases to accommodate the different LATCH points, seat belt slots and tilt options.
New requirements and recommendations for booster seats for kids up to 6 years old and 60 pounds (or even 8 years old and 80 pounds) have been enacted in many states in the last three or four years. A pure booster seat, to raise the kid and position the belt safely, is currently thought to be best for school-age kids once they're too big to sit in a full carseat. However, in order to provide an option that parents can buy to last from pre-school through school age, many forward-facing car seats are now convertible to pure booster seats.

In my recent research, I found it's hard to buy a mainstream car seat that is not convertible either from forward to rear facing, or from forward-facing (with a seat back) to booster seat. I was looking for a seat for a kid over 22 pounds and over 1 year old, that is 16 inches wide and lightweight. And the main special requirement? FAA approval.

I'm going on five more trips with my kid in the rest of 2008. I usually need a car seat when I get to my destination. I also need something to keep my kid in the airplane seat more securely. Even if you believe airplane crashes are so low probability that it's not worth worrying about, I can vouch for the sanity that comes from a wriggly kid well-secured in an airplane seat. I held somebody else's baby during turbulence on a recent flight (while the other mom vomited repeatedly) and I was glad my own kid was asleep and strapped into a car seat with a five-point harness. He didn't even wake up during the turbulence, because I didn't have to wake him up to hold him down during the stomach-inverting drops of altitude.

Using the car seat in the airplane has become a real problem at his current height, however. It's so bulky (thick back, thick seat) that in coach he runs out of leg room and can already powerfully kick the seat in front of him. Time to start investigating alternatives, and as I found out, the alternatives do not reasonably include lightweight, narrow, FAA-approved seats that are also legal in cars in most states.

Here's how airplanes are different from cars, as far as child restraints are concerned:

No LATCH points. I don't know if airplane seats will ever have LATCH points.
Airplane seats are designed to recline/fold. I think this is why booster seats are forbidden by the FAA. So are seats that can convert to booster seats.
Not as much room, obviously. It can even be hard to bring a toddler-size car seat down the aisle, let alone fit into the width and depth limitations of coach seating.

FAA did recently approve a harness product (CARES) that has no seat or back parts at all. It attaches to the airplane seat back and to the seat belt and looks like it will hold a wriggly kid and handle turbulence. It's the only legal option for booster-seat aged kids on planes, aside from just using the regular lap belt. I've purchased one and will certainly mention if it doesn't work well.

Since I still need* a car seat at my destination, I'll probably start putting the Britax Roundabout into checked luggage. *Big sigh* -- it will be years before I'm likely to travel without checked luggage again. If it weren't for carseats, I can pack my own clothing and my kid's for 3-6 days into one carry-on.

* As a final note, let me define "need" in the last paragraph. It turns out the law in BC, for example, is different for residents and visitors. Residents must use car seats for kids up to 40 pounds, but visitors can omit car seats with children over 20 pounds. On average that weight is reached at nine months! A slightly heavy baby might reach 20 pounds at an earlier age and still not be able to sit up on his own. So although the law doesn't require me to have a car seat on my upcoming trip to Vancouver, I have decided I need one.

2008-04-21T22:28:00.000-07:00

I'm making good progress on the public health statistics visualization site. I just added the ability to use pycha's multi-line line graphs:

This shows the incidence rates of kidney cancer in the Northern California population, according to the NCCC.

2008-04-16T21:36:00.000-07:00

Whew, I finally got the right minimal equipment to do off-camera flashes. With my Canon Digital Rebel, this meant

Canon Off Camera Shoe Cord OC-E3
Canon 430EX Speedlite Flash for Canon Pro1

I meant to do this months ago but got the wrong parts -- a flash synchro cable and hot shoe adaptor that just don't all fit together in a way that connect the camera to the flash (a $25 mistake at worst, and I still haven't had time to find out if they might be useful anyway).

Anyway, here are my first photos with this setup, with our cat as a model because she stays still better than other live models in the household (!). The one with whitish carpet and wall as background was really the first photo, slightly cropped but otherwise unedited, just connected the flash pieces and snapped the camera in automatic portrait mode. The one with the dark background is not even in an unlit room, it just had the nice effect that the flash lit the kitty and stuff you can't see, while overwhelming the less-lit background into black.

Only three weeks until Zack Arias' One Light Workshop. Hat tip to Ted Leung for letting me know about that workshop and inspiring me to try all this.

2008-04-16T11:12:00.000-07:00

The site I'm building will make public health data more accessible. This means both findable and visualizable. Kai just came by my office and demonstrated a use case for both aspects of accessibility for this data.

Kai's task this morning is to put together a presentation which involves talking about the "long tail" of diseases: for every common disease that pharmaceutical companies target, there are a hundred orphan diseases. Pharmaceuticals typically ignore these because the revenues from a drug targeting an orphan disease are necessarily small.

A good graphic for Kai's presentation would be a classic long tail graph, with specific incidence rates filled in: possibly lung cancer on the left with a high incidence, tuberculosis in the middle with a vastly lower incidence, and Ebola at the real tail end. But how do you fill in the numbers for incidence of these diseases -- say, for the US, for a given year?

National Cancer registries cover only cancer incidence and mortality
CDC Mortality data covers all diseases, but mortality instead of incidence (e.g. prostate cancer is very common but not a common cause of death)
CDC tracks infectious disease incidence, but not incidence of most non-infectious diseases
The National Heart, Lung and Blood Institute publishes incidence of hemophilia
The National Health Interview Survey counts things like anemia, and while you can find that page on the CDC site, the data are presented very different from data on cancer or infectious disease...

So pulling together incidence rates even for a presentation is hard, and if you wanted to do science it would be even harder to ensure that the same population was looked at. Googling this stuff is harder than you would think. It's fun doing something about this kind of use case.

2008-04-15T12:54:00.000-07:00

In order to keep my life and work organized, the tool I use to track tasks has to be streamlined and seamless. Do you know what I mean? Any hassle or delay, and I'm likely to get distracted or postpone the bookkeeping of adding a new task or updating the status of a task. If I'm hampered in bookkeeping, I fall behind and the system becomes less trustworthy so I can even fall out of the habit of keeping it up-to-date simply because it's already not up-to-date. It requires discipline as well as ease-of-use.

My system is not just one single task list. That would be too long. Instead, it's organized roughly as "Getting Things Done" suggests, into contexts. This is crucial, for reasons I won't go into now. So one of my requirements is that I have several groups of tasks, and a task can only be in one group/context. Another requirement is that I be able to mark tasks as "done" to make them disappear (but not deleted).

For three years now I've been using iOrganize. It's a small and reliable note-taking program that I don't mind running constantly in order to do bookkeeping at a moment's notice. It has folders for "notes", and lists notes by title within a folder. Click on a note and a view pane shows the body text. I have a couple conventions to make this function as a task manager: I mark the tasks that have top priority with a ** so they show up in a quick search, and I drag a note to the _Done folder (which sorts last with the underscore) to mark as done.

The thing is, I unthinkingly upgraded to a new version when prompted, and a few things are driving me crazy, and worse, driving me towards dis-use.

The search function now only searches in the current folder. This breaks my ability to quickly use search for ** to find top tasks. Even though the functionality to search all folders is still there, it now won't show me which folder the search results are in.
The drag to the _Done folder now changes the view to the _Done folder, when it used to (and I really want to) keep the view on the source folder, the context where I'm getting stuff done.

... *sigh*

I have been experimenting this morning with TextMate's awesome extensibility to see if I could use it for a task manager since I run it constantly anyway, but am running into barriers. Can I downgrade my running version of iOrganize instead?

2008-04-14T09:09:00.000-07:00

Shortest recognizable quotes

Three words

I'll be back
Feed me, Seymour
Yippee Kiyay, M*F*
Et tu, Brute
Phone's ringing, Dude
Run, Forrest! Run!
As you wish.
Clatto... Verata... Necktie?
God is dead.

Two words

A Handbag!

One word

Inconceivable!
Stella!
Doh!

Well, recognizable to me, anyway. I'm surprised I can't think of any more two-word quotes.

2008-04-12T11:49:00.000-07:00

Three thumbs up for the Chariot Cougar 2 I bought a couple weeks ago. When you use something day to day the smoothness of use matters a lot:

One thumb for the easy way it attaches to the bike. It's got a ball joint to slip into a socket, held with a pin, and the pin is held with a rubber cover, and then a safety strap in case all that fails, but it's still fast to put together. Somehow with the Burley I never got snappy at getting everything all at the right angles.
One thumb for the carrying capacity and organization thereof. I went to the grocery store today with D. and came back with two big (cloth) bags plus a watermelon. There's two exterior pouches in back, one of them very big yet it folds up when not in use, and small side pouches inside the cab for the spare yoke or Darwin's stuff.
One thumb (Darwin's) for the comfortable, secure safety restraints. The chest yoke comes down over his head and snaps to a strap between his legs, and the seat belt snaps over that. Snap, snap, done.

When you use something day to day the smoothness of use matters a lot. I'm enjoying this.

You may ask if I'm turning into a diligent environmentalist, abandoning even my Prius in order to bike to work and the grocery store. Ok, well even if you don't ask, I'll tell you, there's are different things mixed into the satisfaction return of this practice. There's saving gas, yes, and knowing where I can find a spot to park without frustration. There's the physical well-being from exercise and feeling sun on my face or at least fresh air. I like to think I look pretty cool on the sharp red bike with red trailer, so yes there's a bit of pride too! Whatever keeps me pedaling.

2008-04-08T14:12:00.000-07:00

Programmers sometimes shave yaks. But fiber artists sometimes engage in yak [hair] spinning, plying, winding and fulling -- and best of all, yak whacking. Yea, verily. Not sure which I'd rather do or not do right now.

2008-04-08T06:12:00.000-07:00

Unit testing and doc testing are working in the Django-based project. A couple wrinkles:

1. In the unit tests, I created some model instances in the setUp() method easily. I then discovered that the setUp() method is called before every method in the test class, not just once when the test class is instantiated. Ah yes, I remember this from Java days too. Is there some deep reason why tests (in this case functional tests) shouldn't share the exact same setup but need to recreate it? The fix that seemed to meld most smoothly with what the unit testing framework offered, was to tearDown() all the model instances after every test method, even if I was going to use the same work for the next test.

2. Django only seems to pick up doctests in the models module. I found a hint online (can't even find it anymore) about creating a "__test__" dictionary and putting this in models.py worked:

__test__ =  {"1": Source, "2": DataFrame}

In this case Source is a class in models.py, and DataFrame is a class imported from a file called query.py. The labels "1" and "2" are arbitrary and I don't yet know how they're used. What worries me about this is that I don't want to import everything into the models module in order to test it.

2008-04-07T10:57:00.000-07:00

Finally fixed the archives link. The only way I could figure out to do it within the limited time I gave myself was to throw away my custom template and use a Blogger layout. Less control over other things, but it is better having inaccessible archives.

2008-04-03T16:41:00.000-07:00

Sometimes programming goes too fast, and I actually have to stop and think.

2008-03-28T15:28:00.001-07:00

I miss working closely with other developers; that's part of why I'm blogging right now.
Python exacerbates that, because there's so many ways to do things, particularly the kinds of things I'm doing right now, like parsing text files with tables in human-readable but not-particularly-computer-readable formats.

Here's a tiny example: I needed to remove commas from strings like "1,207" before converting them to integers to graph them. I thought of using slicing, which is a powerful Python feature on lists, strings and more:

>>> string="1,207"
>>> string[:string.find(',')] + string [string.find(',')+1:]
'1207'

It got ugly fast so I looked for something else:

>>> string="1,207"
>>> string.replace(',','')
'1207'

Splits are also powerful:

>>> string = "1,207"
>>> ''.join(string.split(','))
'1207'

Of course I could also define a simple "removefrom(char, string)" function:

>>> string="1,207"
>>> def removefrom(char, string):
...     return string.replace(',','')
... 
>>> removefrom(',', string)
'1207'

To find good ways of doing things, I end up browsing a lot of Python code online. That's frustrating because some sites have started hosting "sample code" mostly as a way to put advertisements on-screen and in pop-ups. And I'm sure the most trivial code review would identify plenty of areas my code could be much more Python-clever.

2008-03-20T17:21:00.000-07:00

Internships for CommerceNet this summer: I'll be the one hiring and managing. If we get a great proposal for a project from the right intern candidate, CommerceNet might simply hire that intern to work on that project. Otherwise, here's what I have in mind.

CommerceNet is an entrepreneurial research institute, dedicated to fulfill the promise of the Internet. We are currently seeking Software Engineer interns to implement a data visualization Web application for public health information. Involves JavaScript and Python, both data access and graphics. CommerceNet may also accept proposals for internships to work on well-specified projects of the intern's own design.

What you'll do
Develop open source libraries or widgets for graphing and data visualization
Build public service, community oriented Web site
Be part of a small team or work nearly independently
Develop with minimal guidance, using rapid iteration and feedback loop and with leeway in choices of tools.
Borrow, create or collaborate on visual design and visual elements
Required Skills:
Web Applications development, including CSS and JavaScript
Python or demonstrated ability to pick up languages
MySQL or similar data management experience
Great ability to extrapolate from raw ideas to realistic implementations.
Demonstrated initiative pulling a project forward
Some experience using graphics libraries
Familiarity with Cleveland or Tufte principles would be a bonus
Email ldusseault@commerce.net with questions or cover letter and resume.

2008-03-18T14:14:00.000-07:00

There is so much IETF work on email these days. There are five separate WGs (one in the Security area), a Research Group and a couple informal efforts. I tend to have to summarize the work to pull it together in my head. Here's the post-71st-IETF sitrep. Also see Barry's post for more detail on SIEVE, DKIM, LEMONADE and ASRG. Document links collected at bottom.

The IMAPEXT WG is so close to shutting down, it did not meet last week. One of its work items was to internationalize parts of IMAP (including mailbox names, and how to sorting strings like subjects) and those documents were delayed but finally got approved.
The LEMONADE WG met, and seems to be winding down. Although its extensions are all linked by being useful to mobile email clients, there are some extensions there of general interest.
The SIEVE WG just finished publishing a whack of documents around its new core spec, RFC5228. At its meeting, the group discussed whether to recharter to do another round on the core SIEVE documents and standardize some more filtering extensions.
EAI WG has requested publication for most of its documents. These are Experimental Standards for using non-ASCII characters in email addresses, which affects IMAP, POP3, SMTP in interconnected and complicated ways.
A design team is nearly done updating SMTP (RFC2821) and the Internet Message Format standard (RFC2822). They're handling "last call" issues on the list.
DKIM -- having previously published its core signatures doc (RFC4871) and requirements for signing practices (RFC5016) -- is now working on the signing practices standard itself.
ASRG, the Anti-Spam Research Group, is documenting various anti-spam techniques.
In informal discussions, several of us keep talking about rearchitecting email access. However, nobody's ready to predict, let alone commit, that their company will implement something new. Does that mean that there's not really enough pain around using IMAP? Or that the pain is the user's and not the software vendor's?

Documents:

IMAP Sorting and Threading (in IESG evaluation): draft-ietf-imapext-sort-20
IMAP i18n (Approved as Proposed Standard): draft-ietf-imapext-i18n-15
IMAP Annotations on messages (Approved but waiting on completion of SORT): draft-ietf-imapext-annotate-16
IMAP LIST Extensions (Approved but waiting on completion of i18n): draft-ietf-imapext-list-extensions-18
LEMONADE Profile, to replace RFC 4550: draft-ietf-lemonade-profile-bis-08
... other LEMONADE docs in progress listed on WG page
New SIEVE base spec: RFC5228
Other new SIEVE specs: RFC5229, RFC5230, RFC5231, RFC5232, RFC5233 and RFC5235.
EAI/I18n Delivery and Status Notifications (in Last Call): draft-ietf-eai-dsn-06
EAI/I18n Addresses in SMTP (in Last Call): draft-ietf-eai-smtpext-11
EAI/I18n Addresses in Mail Headers (in Last Call): draft-ietf-eai-utf8headers-09
Other EAI documents yet to come include documents on the mailto URI, mailing list considerations, downgrade mechanisms, IMAP support and POP3 support.
SMTP Revision (in IETF Last Call): draft-klensin-rfc2821bis-08
Internet Message Format Revision: draft-resnick-2822upd-06
DKIM Sender Signing Practices: draft-ietf-dkim-ssp-03.txt
ASRG: DNS Blacklists and Whitelists: draft-irtf-asrg-dnsbl-04

2008-03-17T11:58:00.001-07:00

One of my favourite tools for keeping on track is doing a regular report to my management.
Ever since Max Dunn, a former co-worker, told me about the "four P's" (progress, plans, problems, personnel) mnemonic, that's been a useful concept. Writing down some notes in each of those categories makes me remember stuff better, catch up, and plan better. I get almost all the value even without sending the report.

Right now I send a somewhat different monthly activity report to the Apps Area Discuss mailing list and to the Working Group chairs of the WG's I advise. It serves some of the same purpose to me, and other people have found it useful. I indicate what I think is the status of each document I'm sponsoring to become an RFC, and if the authors think the status is different (e.g. one of those common "I was waiting for you while you were waiting for me" traps) they tell me. I mention a couple notable things at the top of the report, and try to summarize what state each WG I advise is in.

I try to send these in the first week of the month, which means I'm two weeks late for March due to being sick and having the IETF meet last week. Maybe later today??

2008-03-11T11:47:00.000-07:00

Although I'm mostly doing IETF stuff at IETF this week, I took advantage of Andrew's expertise while he's here too, to help me get a Python library running and working. I had gotten blocked last week and we worked through several more problems this afternoon. Some notes follow for the record, to add to the couple blog posts out there that help one out on these problems.

I was trying to install pycha and play with its simple charting. To do this, I needed to have py-cairo and cairo version 1.4.12 installed on python 2.5.

The easiest way to get cairo installed is using MacPorts. This is fine, although it installs cairo into the MacPorts version of Python, not the Mac OS X version of Python. The Mac OS X version of Python already has cairo installed, but it's apparently an older version. I didn't know how to upgrade the system cairo library so I focused on doing this with the Python installed by MacPorts. This led to later complication.

After apparently successfully installing cairo version 1.4.14, attempts to install py-cairo failed. It kept on insisting that only cairo 1.4.10 was available. Andrew tracked this down, and we're pretty conviced that the information that the MacPorts cairo package applies to pkg-config is false: it says version 1.4.10 right inside the 1.4.14 cairo.pc file. We edited this file directly and went on.

Next we tried to install py-cairo using the regular "python setup.py install", but that failed -- it is not set up to work on Mac OS/X and there were numerous errors. MacPorts rescued me again, thanks to a hint we ran across from KenKeiter -- this port package has a bunch of hacks (string replacements in directory paths it looks like) to make py-cairo work on Mac.

Finally, back to installing pycha itself. This has a nice handy Python Egg available, so I installed "easy_install" to install that. I goofed and installed that in the default System version of Python, so while installing easy_install worked, installing the pycha egg failed. It turns out there is a MacPorts port of the same thing, called py-setuptools. Only you don't want that one, you want py25-setuptools if you have Python 2.5. I installed that.

OK, back to installing pycha for real again and it finally worked. One gotcha in following the main documentation page example, which is otherwise very nice -- I needed to "import pycha.bar" for the sample code to work (Update: Lorenzo Gil Sànchez fixed this already). And it did work.

Proof:

2008-03-04T20:30:00.000-08:00

Mike from work pointed me to EpiSpider, which does dynamic graphs and uses in part an in-browser library for faceted browsing called Exhibit, from the SIMILE project. This project also has a beautiful Timeplot library, again in-browser.

The basic approach here is minimalist: all a site developer needs is a place to host Web pages, some of which are data files. No database, no content management system. Still, those have their places too, and I can already see how to make great use of Django features as well as Exhibit and Timeplot features, and I know databases. To tie these together, one just needs a bit of AJAX code to call a server-side process to pull the appropriate subset of data from the database in JSON or other simple format, and I know that exists too.

I had been ready to try generation of static chart images in order to save time -- that was why I was trying to use R (also the data import, manipulation, analysis and export features could be useful). But ultimately good online charts need features like mousing over to see point values, scrolling, and zooming; static charts can't provide that but in-browser canvas code like that used in Timeplot can.

Sample here.

Update: removed annoying iframe

2008-02-29T16:45:00.000-08:00

Nearly two years ago, I made a prediction that Atom would replace WebDAV. At the time I was even working on revising WebDAV and just beginning my involvement with Atom standardization. I told this prediction to Cullen Jennings and documented it in a note dated 5/9/06. "Replace" is a fuzzy term, because they're not equivalents, but here's what I see today.

Google turns Atom, RSS and AtomPub into GData and uses that for blogs, calendars and task lists for starters. Google does not use WebDAV as far as I know.
Microsoft is unifying its developer platform protocols on Atom and AtomPub, and using those protocols for unstructured application storage, e.g. photo albums to blogs. (h/t James Snell. Hey, I used to work with David Treadwell, haven't talked to him in years.) WebDAV support in projects like Exchange is downplayed.
IBM has a bunch of projects using Atom but it has a bunch of projects period and I haven't seen strategy announcements about either standard.
Apple uses Atom and RSS in quite a few applications, although it's also using WebDAV and CalDAV on .mac and in its calendar server.
Blog service sites are ubiquitous and all use Atom or at least RSS. File sharing sites or other WebDAV public services are rare. Photo sharing sites are more likely to use Atom than WebDAV by a long shot.

Were I to propose CalDAV today it would probably be CalAtom -- some things would be easier, some harder, but it would catch a wave instead of drifting in the tail of something that was never much of a popular wave. Oh well, we needed something then, and WebDAV gave the most leverage at the time.

2008-02-29T16:24:00.001-08:00

The Django stuff was going so well, I switched over to learning R a couple days ago just to depress myself. It's tricky to get into. The documentation is very scattered (kudos to those who are improving it at the wiki but that's a work in progress). I haven't yet found a way to get documentation for arbitrary libaries. The cute name 'R' makes it very hard to Google for R libraries. E.g. I haven't yet found a core graph or a library function that creates horizontal bar graphs (though maybe somebody will tell me to do vertical bar graphs with all the labels turned, then turn the whole image).

One headache in particular was trying to work with display strings that needed to be converted to numbers. I had a table containing this tiny fraction of data (out of 8 columns and 15000 rows):

ICD.Chapter    Crude.Rate
A00-B99        22.0
C00-D48        193.2
D50-D89        3.2 (Unreliable)
E00-E88        33.7
F01-F99        21.0
...

To graph the Crude Rate without the occasional "Unreliable" strings, I needed to break the information about which measurements were unreliable into a separate column (not losing the information), reduce the Crude Rate to be a number and convert it into an R 'numeric' type. I spent about 8 hours. The first four I managed to solve the first problem: I looked for a "contains(string, substring)" style function but there was none; "grep" produced errors that were really hanging me up whenever the input cell did not have the string I was looking for. Finally I wrote my own little contains function using regexpr:

contains <- function(pattern, string) { 
  regexpr(pattern, string) != -1 
}

It looks simple now, but it was tricky to get a function that worked properly when converting 15000 cells to 15000 new cells.

The next four hours were spent trying to remove the "(Unreliable)" string from the cells that had it. Again poring through documentation, I tried various approaches and got the closest with this one:

first.word <- function(x) { 
  substr(x, 0, regexpr(" ", x)[1]-1)
}

The problem was that although this one worked fine on single strings when I tested it, it didn't work on turning a whole column into a whole new column.

Today at 11:00 am I turned to Excel to see how quickly I could do it in Excel. It's been a long time since I've used Excel formulas, but I knew this kind of thing could be done. In many ways the Excel formula documentation is worse, but it's a smaller set of controls (or at least a clear subset) and it was comprehensive and organized.

In Excel the two new columns are created like this from the H column:

=NOT(ISERROR(SEARCH("Unreliable", H2)))
=IF(ISTEXT(H2), VALUE(REPLACE(H2, SEARCH(" ",H2), 13, "")), H2)

I'm sure this could still be done in R but making those two formulas work for this particular table took me 20 min in Excel instead of four hours.

2008-02-26T17:03:00.000-08:00

Development work went much smoother today. I set aside installation of Apache and MovableType and decided I could do something easier at least for now. Believe it or not it was easier -- for me at least -- to implement a minimalist blog in Django, from the database on up to the display pages, than it was to install and hook together software.

I also wrote some of the core data model for the site using Django's standard model tools. It's a site that will improve the accessibility of health-related statistics (costs, health surveys, incidence registries and other epidemiological data) by offering good search and lookup and simple clean visualizations. So I started by creating data models for data sources, report names and conditions that are tracked, and hooking those data models primitively into the Web pages where I want them to show up.

Of course development involves writing new code, but that's not nearly all of it. Even when writing new code, part of development is debugging: when you know what you want to do and have chosen a way to do it, but it's not working. Another chunk of new development time is decision-making, e.g. deciding how objects will relate or how interfaces will look. It's interesting how fast this style of development goes from decision-making to debugging and back, without a lot of time spent writing new code.

Any quick pointers on
- unittesting Django sites?
- Integrating C libraries into Django projects?
- Departing from Django DB models -- when to create one's own tables and how?

2008-02-25T15:57:00.000-08:00

Since Ravelry ate my blog, I'm going to see if I can't restart this with a different idea. I originally started a blog to try to say clever (and subtly snarky) things about news, particularly science news. Then this turned into a lame knitting blog. Now I feel like I don't know what to post here. So how about I post what I like, eh?

I am trying to learn how to use a camera and a flash better. I bought a Canon Speedlite 480ex so that I could point my flash in different directions, dial up or down the light intensity, and use a diffuser. I also tried to buy a hot shoe adapter and cord so that I can use the flash at a distance from the camera, but it looks like I might have bought the wrong model of something. The cord has a little plug just like a single earphone plug, and there's a dock space on the Speedlite, but it's empty. Behind the hole there's no socket, just plastic.

Still, the Speedlite is better than nothing. I used the diffuser for some of the shots taken of people in a big, poorly lit conference room at the Ravelry meetup.

I think it turned out better than with a standard Rebel built-in flash. I had fun anyway.

At work, I'm immersing myself in Django and oh-my-god real setup of Web, database and other server software. This intimidates me. When I get something working (like an index.html page with working stylesheet and logo links in Django) I take a little "success" break to let the feeling last.

I hate how easily this kind of work can go wrong. There have got to be common ways of doing things that just work. But I don't know what those are, and software for developers errs on the side of flexibility and power, rather than on the side of making choices for you so that it just works. And some choices work but are awkward. Where do I install Apache? Where do I locate the static files served by Apache? Where do I install MySql so that it can be used by MovableType as well as Django? Do I upgrade to Python 2.5 inplace or have it co-exist with Python 2.3? Invariably on the Mac there are permissions issues: do I change the permission of the file so I can do what I'm trying to do with my login account or do I switch to 'sudo' the command? I normally get things working eventually but it's sooo slow.

2008-01-27T15:50:00.000-08:00

What pictures of curious and fast-crawling babies often look like.

2008-01-24T13:40:00.000-08:00

Clearly my lack of posting is explained by the fact that I have died in a knitting accident. It was the size 1 lace Addi turbos I've been knitting the cellular automaton shawl with, those are some sharp needles.

Maybe it's just a metaphor for this blog: I haven't been posting much since I joined Ravelry, so the blog was what died in the knitting accident.

Have I anything else to say? I've been thinking about email software, and I concluded that a really great email-focused program would include GUI design contributions from a game designer. Email management needs to have the same smoothness and responsiveness that a good console game does. I deal with hundreds of emails a day; dealing with them should be so smoothly software-assisted that I'm eager to click to kill the next one. I mean file it. Instead my flow gets interrupted constantly by tasks that draw me away from getting through my email.

2007-11-25T22:02:00.000-08:00

I've been pretty busy but just had time to learn a bit of JavaScript and the canvas element. Since I'd been playing with cellular automata ideas in knitting, I put together one of those. Pretty!

2007-10-02T12:51:00.001-07:00

Blogging here is going to continue to be light for the foreseeable future. I am now a member of Ravelry, so that's a more obvious place for me to post knitted projects -- the whole point of doing that was so that other knitters could find my projects as they research what project to do next, as I do. Ravelry was built for that purpose and makes it easier for knitters to post their own projects, and to find other projects done with a certain yarn, or with a certan pattern, etc.

But I did want to post a link to this Telegraph article by Liz Hunt, and that doesn't fit naturally either on Ravelry or on the CommerceNet blog. The article bitterly critiques a book called The Gentle Art of Domesticity (not available in US yet), by Jane Brocket. I wish I could figure out what has Liz Hunt so bitter, because I can't see anything intrinsic in the work of this enthusiastic cook and craftster to inspire the tone. The comments on the Telegraph article vary from wondering how Jane Brocket could be so "perfect" to angry reactions to the article's tone and criticism of a domestic lifestyle.

I've been following Jane's blog since she started it in early 2005, staying as she progressed her writing style and particularly her photographic abilities (from this to this , for example) even as her posts on knitting became more and more infrequent. Most of what she documents and extols I have no desire to imitate. I'd rather bake oatmeal squares and butter tarts than elaborately decorated cakes and "fairy buns". I hate gardening and do not want to live in a remote town or in the country. I do knit and sew but I've been knitting for five times as long as her blog has been up so I already have my own style and inspirations. But it's a gentle, beautiful picture she paints, and it's like visiting an English village house and garden for a quiet moment when I read a new post on Jane's blog.

The only way I can reconcile this with the Telegraph article is the tension over feminism vs. domesticity again. Liz Hunt brings up careers and Shirley Conran in opposition to the domestic values. I haven't got a copy of Jane's book yet but I suppose Jane even started the argument by extolling domesticity as an antidote to ambition and stress. But I'm pretty sure that if a woman wrote a book extolling jogging, swimming, line dancing, playing cello, online gaming, sculpture, reading, bird-watching, photography or dirt-biking as an antidote to ambition or stress, it would be seen as trite at worst rather than threatening.

2007-08-07T15:36:00.000-07:00

I frequently try Web applications out just for the sake of seeing what they can do, and usually abandon using them. But I like keeping up with the field, particularly for email, calendaring and task-related applications.

Today I tried one that does all three, that I heard about originally from Rael Dornfest (back when I worked at OSAF; it wasn't available then and it is now). I might abandon this online app too because I like to manage my time offline better than online, but there was one point when testing the application sent a frisson of pleasure down my spine. Usually that reaction is reserved for either more private pleasures, or when fondling yarn.

The site is stikkit.com, launched last year. The site is very smart at telling what kind of note you're creating, detecting todos and events from linguistic cues. For example, it detects Quiltathon on September 30 as an event and puts it on the correct date on the calendar.

The 'frisson' moment was when I created a task to email somebody. It was recognized as a task by the phrase do send mail to So Andso. Then I noticed that the new task had a linked stikkit for a person: "So Andso". When I clicked on that, the automatically-created stikkit had a line reading name: So Andso. I intuited that if I added another line to the stikkit with email: soandso@example.com would make the new stikkit into a contact, and voilà, it did!

It was a little disappointing to find that while Quiltathon on September 30 was detected seamlessly as an event, September 30: Quiltathon was not. Also when I had a space between the "name: ..." and the "email: ...." lines in my second attempt at creating a contact stikkit, it didn't work until I removed the empty line.

Oh, and I love, love love applications with hotkeys so that I don't have to mouse around. Even better when they can be discovered.

2007-07-08T19:35:00.000-07:00

These cute kid items were knit by Uta Hayward for her son. I'm completely tickled to see the blanket because it's the pattern I published on For the Love of Yarn. It looks very different in the multicoloured yarn chosen by Uta's son, but it looks terrific. I like the way the colour pooling forms little arcs because of the decreases, and that the little arcs are separated by the yarn-over columns. Maybe I'll knit the pattern again myself in multicoloured yarn!

2007-07-04T14:38:00.000-07:00

Happy Independence day. Then again, I'm working today -- I've got IETF stuff scheduled tomorrow with people who aren't in the US and they'll be prepared.

I would have said Happy Canada Day three days ago, but I was travelling home from Canada on that day. There were nice people on the plane with maple leaf body paint or temporary tattoos or something. Even nicer, the man who switched seats to give me more room for the baby, and his wife who held the baby while we switched.

I am having fun despite the work (hey, work's fun too). I just discovered the FireFox plugin Tab Mix PLus, which I have used to set it up so that each new tab opens to my local home page which has quick links to stuff I use a lot. Yay smoother work processes!

I was also sucked into Facebook last week by my cousins and siblings. I initially felt too old but then a bunch of people in my class in grade school and high school contacted me immediately so I already feel like I belong a little. We were making fun of Facebook at work recently. One jibe was that it's a classic force multiplier: it allows you to form cliques and shun people at great efficiency! Before you could only shun people in sight, while out socially -- but now you can shun people anywhere, anytime!

2007-06-12T21:38:00.000-07:00

Reading Tukey's Exploratory Data Analysis:

If we want to see what our plots ought to tell us, there is no substitute for the use of tracing paper (or acetate). If we slip a well-printed sheet of graph paper just below the top sheet of a pad of tracing paper, we can plot on the top sheet of tracing paper almost as easily as if it were itself ruled. Then, when we have the points plotted, some boundary or reference lines drawn, and a few scale points ticked, we can take away the graph sheet and look at the points undisturbed by a grid. We often gain noticeably in insight by doing this. (And we have had to pay for a sheet of tracing paper rather than for a sheet of graph paper.) [...]

An alternative that:
can be even mre effective,

is no more expensive,

takes a little more trouble to prepare for,

replaces the tracing paper by the thin sheets of transparent plastic (acetate) made for use in overhead projectors. Two cautions are important:
You can only use markers specially made for the purpose. [...]

It is important to keep one's fingers off the plastic until the picture is completed. (A piece of thin graph paper, placed upside down, works very well as a hand shield.)

Much of this classic textbook is obsolete tips and tricks -- e.g. for collecting data on paper and checking that you've copied it accurately to another piece of paper. The notes on cost of tracing paper vs. graph paper vs. acetate strikes me as quite funny now!

I think I learned to graph by hand in school but never had to except as an academic exercise -- I was using graphing software before entering University (summer job in a water testing lab). I wonder what kids today do in school, if they even bother with the academic exercise. I hope so, because it was kind of fun playing with numbers on paper. For some values of fun :)

2007-05-31T15:28:00.001-07:00

Random Stuff

If you know me, you know I've been pretty busy and no surprise that I've been neglecting my blog! Rather than let the blog languish for even longer waiting for the perfect post, I'll just be random today.

1. The Atom Publishing Protocol (APP) is up for approval by the IESG this week and I'm the sponsoring Area Director. I'm excited about APP because it will serve as a model or basis for much future work in accessing and editing structured content on the Web. (Will the IETF APP area one day be assumed to stand for Atom Publishing Protocol rather than APPlications?) I observed some of the Atom Interop event last month (summarized by Tim Bray here) and it was great.

2. I am apparently a sucker for some kinds of advertising copy, not necessarily what one would call good copy. Witness: here is the entire Product Description for the ice cube trays I bought online when my freezer was mysteriously not making ice cubes automatically, before it mysteriously started to again.

Ice Cold Fun! These wild and wacky Flexible Ice Trays are just plain crazy. The flexible designs allow ice to just pop out! Its quick and easy. And fun! Better still are the nutty shapes the ice forms into Stars or Hearts! Wild! Fun! Each tray container contains twelve individual molds perfect for all the fun ice youll be making. Blue star tray and pink heart tray. Made of high quality rubber.

Nutty shapes! Wild and wacky! Oh the fun we'll have with ice cubes!

3. I continue to knit stuff. I just don't have time to post about it any more. Here's a shot of me with other knitters at a meetup at Nine Rubies, and the project at my feet is now a completed scarf (with an estimated 40,000 stitches, which is why I normally don't calculate stitch counts).

4. I also made a mei-tai baby carrier. Again, this photo is obviously not taken by me, otherwise I probably wouldn't have a photo of the carrier at all. I don't even have photos of the sweater or bag I finished a while back.

5. Looks like IETF Chicago will be exciting: the Apps area will have BOF meetings on HTTP, notifications from email stores, and personal address book access.

2007-04-11T12:25:00.000-07:00

"For the Love of Yarn", an online knitting magazine, has now published my Cielo baby blanket pattern in their spring '07 issue. Oh the fame!

Seriously I did get a query already about yarn substitutions. I knit this with Elann.com "Den-m-nit" but that appears to be discontinued. There are two special effects inherent in this yarn: one is that it will fade like blue jeans over time as it's washed and used, and another is that it will shrink %10 to %20 the first time it's washed. Rowan Denim yarn behaves the same way and is widely available, thus it's a perfect substitution -- but expensive. Another less-well-known option is Twilleys Freedom Denim Yarn. I found a couple online vendors for this:

Texere in the UK
sells this for 2£/50g, which is currently about $3 US. Even with the shipping, for 13 balls (what the pattern suggests) this is cheaper than US sources.

Erica's Yarn sells and ships in the US for $5/50g.

It should be easy to substitute other worsted weight yarns to get other colours besides indigo, although of course the faded jeans effect won't happen. Note that there are two main kinds of 100% cotton yarns, mercerized and unmercerized. Mercerized cotton yarns do not shrink, therefore behave differently than the denim yarns. Still, some of these can be wonderfully cheap and colourful, like Elann.com Sonata, which has great stitch definition and sheen and is $1.98/50g. It's also possible to use cotton blend or non-cotton worsted weight yarn for another kind of look. To substitute a yarn that doesn't shrink, the knitter just has to stop knitting the main body when the blanket is big enough, and knit the edging on as written.

Some unmercerized cotton yarns do shrink and others don't. Worse, unmercerized cotton yarn isn't always advertised as such. For example, I've used Mission Falls 1824 cotton and Blue Sky Organic cotton without any noticeable shrinking (and they're both great yarns though with the Blue Sky you have to live with some pilling and fuzzing). Knitter's Review confirms that Mission Falls 1824 cotton is unmercerized and various blogs say that the Blue Sky cotton is unmercerized. So just getting an unmercerized yarn doesn't guarantee that it will or won't shrink.

2007-04-10T14:13:00.000-07:00

I hadn't yet seen the talk by Hans Rosling, last year at Ted 2006. It's fascinating from a technology perspective but even more so from a political or social welfare perspective.

I browsed a little further into other Gapminder presentations, including one given by Ola Rosling at Google a year ago, where he talks about the significant differences made by left vs. right politics in achieving health advances vs. wealth advances -- a point which didn't appear in Hans' talk.

What is news is that Google bought the Trendanalyser software and hired its team from Gapminder, announced last month. The Gapminder foundation continues its mission separately although I'm not entirely clear what that is now.

2007-04-04T13:06:00.000-07:00

Needed: Web 2.0 Hackers

CommerceNet (my employer) would like to start a bunch of Web 2.0 projects in the areas of calendaring, security, schemas/standards and health information. We're looking for some hot Web 2.0 developers to help us prototype some of these sites. The job descriptions and address to apply to are here but you can ask me directly for information if you'd like -- I'm a hiring manager.

2007-03-03T10:19:00.000-08:00

I am part of a panel hosted and organized by Scott Rosenberg on why Software is Hard at the Hillside Club tomorrow evening. Scott's got more exposure to this than any other journalist I can think of, perhaps you've seen his book Dreaming in Code. Eric Allman and Chad Dickerson will be part of the panel too.

2007-02-23T12:24:00.000-08:00

How to shop at Ikea with a Prius:

This is packed with $860 worth of furniture and storage boxes from Ikea. The big-ticket items were a full height and width Bonde shelf unit with doors, and a four-unit-tall Effectiv unit with doors. Even bulkier, however, were the two storage baskets and various plastic bins. The amazing thing was that we still fit two adults and a baby in his carseat in the car, too. The adult passenger had to hold plastic boxes on her lap for that to work.

Other pictures: another view of the packed car, and the purchases spread out on the driveway.

2007-02-19T14:27:00.000-08:00

Just posted pictures from yet another baby project -- a lace angora/silk handspun baby hat -- to my current knitting page.

2007-01-16T17:55:00.000-08:00

Perogies from Lisa's Test Kitchen!

I was inspired by the America's Test Kitchen approach from the Cook's Illustrated folks, which results in the full-disclosure style of "The New Best Recipe", and from there, my newly-reliable baking skills. So for my yearly Ukrainian 12-meatless-dishes feast, since I scaled down to making only perogies and kutya this year and my lovely friends brought the rest of the food, I decided to do some testing. Like Cook's Illustrated, I began by collecting a range of perogy dough recipes that differed significantly, and tried three of them. Here are the results of testing three different base recipes, with the main criteria being handling (rolling out dough and sealing the filling in side is hard to do properly with my mom's recipe) and tenderness.

1. Growing Alberta's recipe: in one of the many, MANY ways of combining just oil, flour, water, salt and eggs, this recipe is very simple and similar to many pasta recipes (e.g. ravioli dough). Most other perogy dough recipes I found online had the same basic ingredients in only slightly different quantities. Results: the dough itself is easy to make up, and then sufficiently easy to work. However, when boiled, this ended up making mutant-looking perogies: oddly lumpy, with a surface texture that looked a little more shiny and bubbly. The texture, more importantly, was a little more rubbery than ideal, and was not anybody's favourite in taste-testing.

2. Epicurean's recipe: this had a cup of sour cream instead of any water, and it also adds some baking soda (the rest is flour, egg, salt). This seemed sufficiently different to be worth trying. Result: this made a fine dough that rolled out reasonably well and sealed well. This was the most robust and even dough: every single perogy made using this dough looked like a perfect, even, golden half-moon, even after boiling and frying. At least one person decided this also produced the best texture or tooth.

3. My usual recipe, from my mom, given to her by a Polish friend in Edmonton who was a fabulous cook. This dough is my standard to compare to because I've made it before: it's very tender and everybody loves it, but it's a pain to roll out and use for perogies as it's so fragile. There are usually a few casualties during boiling, where the dough opens and water gets inside the perogy even if the filling doesn't spill out. It's also unreliable: it uses 1/2 cup of mashed potatos, which is hard to standardize (varies by type of potato, size, how long boiled, how carefully mashed, and how long left to stand to accumulate or release moisture after boiling/mashing). I couldn't find any recipes on the Web that had mashed potato in the dough, so it appears to be a unique recipe. Taste-testing results: most people found this was the best texture.

Here's that "heirloom" recipe for comparison:

1 and 3/4 cup flour 2 egg yolks
1/2 cup mashed potatos 2 tablespoons shortening
1 teaspoon cream of tartar 1/2 cup warm water
1 teaspoon salt

Mix flour, cream of tartar and salt together. In separate bowl, mix mashed potatoes, egg yolks, shortening and water together. Now mix wet and dry ingredients together, adding flour as necessary for a workable dough [this always comes up too wet, I need to adjust that]. Let sit 10 minutes before rolling dough. Roll a portion of the dough to a couple millimeters thick. Cut circles [I use a water glass inverted]. Put a spoonful of filling on each circle. Fold over circle and seal with water.

The next step in this testing process appears to be to start varying the best recipe found so far, to see if its most important advantages can be improved, its drawbacks minimized, or its technique simplified. This is helped along by hypotheses about what makes this the best (or by what makes it difficult). These are the variations on my usual recipe that I've tried so far.

Try omitting the cream of tartar as its role is not apparent in the absence of egg whites. Result: the exact same dough made with and without cream of tartar had apparently the same texture, but turned quite grey while standing, as the potato starch interacted with air. The grey blotchy dough was so unappealing that I threw it out without attempting to roll it out, cook it, etc. Conclusion: keep using cream of tartar as long as the recipe contains potato.

Try using a beater on wet ingredients, both to save labour and to improve consistency and robustness of dough by making sure the potatos are well-mashed. My regular dough always has a few visible bits of potato once I roll it thin, and sometimes this creates a weak spot as I form or cook a perogy. Results: using the beater made the dough turn out too gluey and it stuck to everything, making the rolling/folding even harder than normal and without making the dough any more robust. This result could have been guessed if you knew not to over-mash potatoes.

If letting it sit 10 minutes is good, is letting it sit longer even better? I tried a couple variations here. Results: Overnight wasn't good (though the results may be conflated with having used a beater on that batch). I also didn't see any difference waiting 10 minutes or an hour when the recipe was otherwise done the same way.

That was enough experimenting for one person for one year. What are the promising avenues for testing next year?

Sometimes the Cook's Illustrated people form a hypothesis about what makes a recipe good, and try to achieve the same goodness with a different approach. My hypothesis here is that the point of adding mashed potato to the dough is probably to increase the amount of starch relative to the amount of gluten. This should make the dough a little less rubbery or more tender. This hypothesis suggests there might be a more reliable way of reducing gluten ratios than using mashed potatos -- for example, using more cake flour and less all-purpose flour. Thus, I should look into how to reduce gluten ratio without using mashed potatoes (or using less of them). How much all-purpose flour should be replaced with cake flour?

Try to simplify technique and increase reliability by using instant mashed potatos. This approach would not suffer from variability in the starch content or waxiness of the potatoes, or from changes in the details of cooking and mashing.

Try the sour cream dough again. It was definitely easier to work with and people were not unhappy with the result. I want a bit more direct comparison (with the exact same filling, side by side) and evaluation to be confident that this is as tender, or just about as tender, to see if it's worthwhile switching from my traditional recipe.

Combine the two most successful recipes: what about using the mashed potato recipe with sour cream instead of water?

Also, next year I need to take pictures. As I write this blog post two days later, it's too late: they're all gone. Very popular, I tell you. My guests take home leftovers if there are any (last year there weren't) although I insist they leave leftovers for me too.

2007-01-10T08:58:00.000-08:00

I guess the new year is a good time for me to finish projects. I finished a sweater that I've had on the needles for months, as well as a period baby hat (a rare non-knitting project). I'm almost done a Noro Kureyon felted purse, too.

1 and 3/4 cup flour	2 egg yolks
1/2 cup mashed potatos	2 tablespoons shortening
1 teaspoon cream of tartar	1/2 cup warm water
1 teaspoon salt