It's not entirely intuitive, but the same architectural style that allows Web servers to handle large scale usage and iterate so successfully, may also help constrained devices (like sub-10$-material-cost sensors) to handle small scale loads and power usage and avoid being upgraded for many years.

A lot of discussions of using REST in constrained devices have glossed over whether it's really interoperable HTTP that's being used (e.g. compressed HTTP) and whether the low-power device would be in the role of the client or the server. My opinion at this point is that for truly constrained devices, compressing HTTP itself would be the wrong choice, and that the device must be in the role of the server. The rest of this series explores why I believe that and should lay open the assumptions that might allow somebody to correct my information and revise my opinion.

The first part of the series attempts to explain is why HTTP is the wrong transfer protocol choice for some constrained devices. In theory it's a massively extensible protocol, with ways to extend the operation set, upgrade versions, add status responses, add headers, add resource content types and so on. Again in theory, almost all headers can be combined with each other and with different methods unless forbidden. In practice, the deployed base makes some HTTP extensions, and particularly the combinatorial expansion of extensions and optional features working together, quite difficult, and much special-case code must be written to handle the cases where widely-deployed software fails to implement some detail correctly. Here's a few examples of problems we've had over the years.

1. In developing WebDAV, we tried to use the standard "OPTIONS *" request (the special-case URI '*' means "give me the options or capabilities of the server"). However, we found that this is poorly implemented. Java servlets, for example, did not support the '*' URI. Other HTTP frameworks supported the '*' URI but did not allow the response to OPTIONS * to be extended by software extending the HTTP framework. Ugh.

2. HTTP has several different ways to indicate the content-length (historical reasons as well as for different use cases). One is the MIME type "multipart/byteranges". Another is chunked transfer encoding, where the server (and thus the client) does not need to know the total length until the last chunk is transferred. A third is "Content-Length". A fourth is for the server to terminate the connection, although this feature opens up the possibility of truncation attacks. Some of these methods work poorly with TCP connection continuation. Both clients and servers have to support almost all of these.

3. Both absolute and relative URIs are allowed in several locations, even though they're not a good idea in all locations. The resolution of a relative URI can be tricky and complicate client software. Implementations of this make it worse; e.g. the "Location" header was defined to allow only absolute URIs, but many implementations have been found which use their generic URI parsing or generating code to allow relative URIs in that field as well.

4. Parsing headers with quoting, separator (comma and semi-colon), whitespace and continuation rules is difficult. See http://greenbytes.de/tech/httpbis/issue-14.xhtml, http://greenbytes.de/tech/httpbis/issue-30.xhtml, http://greenbytes.de/tech/httpbis/issue-62.xhtml, http://greenbytes.de/tech/httpbis/issue-77.xhtml etc. Some headers defined in other specifications besides RFC2616 even used the syntax rules incorrectly and have to be special-cased.

5. The support for the Expect: header and 100 Continue response has never been good. In theory it's required, so there's no way for a client to ask if a server supports it, thus the client could end up waiting quite a while before giving up on the server initial response. Instead quite a few clients ignore the specification text and start sending their request bodies right after the headers including the "Expect" header, without waiting for the server 100 Continue response. This kind of feature also makes it harder to integrate authentication (what happens if the client uses this header when an authentication handshake needs to be initiated? ) and connection/transport features, as well as to implement intermediaries.

There are many more examples in a long issues list (http://greenbytes.de/tech/httpbis/index.xhtml), and some of the discussions of these issues get quite lengthy in considering how features are actually implemented and how they work combined with other features.

Developing a very simple HTTP client is simple, particularly if the client is talking to a single known HTTP server or is making only a small number of kinds of requests. Developing a limited-use HTTP server can be fairly simple too, although it gets to be significantly more complicated if the developer actually tries to handle all RFC2616 features and all unknown headers and methods properly. What turns out to be very hard is building an HTTP client library, server library or extensible server, because these are used in unexpected ways. It's easy for a developer using one of these libraries to break things, e.g. by manually adding a header that doesn't work with the headers added by the library. The library has to support using TLS and not; several kinds of authentication, several extensibility mechanisms and many failure modes.

The HTTP Components project at Apache talks about the many flaws and excessive simplicity of most HTTP client libraries, and states that "This is one reason why Jakarta, and other free and commercial vendors, have implemented independent HTTP clients". In other words, code re-use is seriously reduced by the way HTTP must be implemented. Some software companies are still selling HTTP client libraries to fill implementation gaps.

General-purpose HTTP servers -- ones that work for a large number of resources, a large number of requests, support TLS, content-negotiation, cache-control, redirect, authentication and other major features -- are even harder. When well implemented, HTTP server farms scale tremendously well. But much, much effort has gone into making those work.

When we look specifically at constrained devices, we see a much more limited set of use cases than the overall Web offers.

• Documents are not large! Thus, document range and response continuation features are not desired. Only one transfer-encoding should be necessary, and conditional request features won't be worthwhile.
• Documents from constrained devices are not intended for direct user consumption. There is no need for language and charset negotiation.
• Even negotiating content type may be rare. A constrained device will state what it supports and not negotiate.
• A constrained device will never act as a proxy, so need not support for Via headers and a bunch of status codes like 504. Further, if we define a non-HTTP, non-proxied transfer protocol that can be converted to HTTP at the first proxy step (or converted from HTTP at the last step), then the non-proxied transfer protocol doesn't need any proxy features at all.
• Redirects are not necessary or could at a minimum be drastically simplified, making most of the 300 status responses unnecessary.
• All of the 100 level informative status responses are unnecessary.
• The authorization model for constrained devices is quite different than the authentication model assumed in HTTP (and associated logic like the 401 status code behavior is therefore unnecessary). Access control will be simpler than in current Web servers.

Imagine instead of a messy HTTP client library or server stack, we had a protocol with about a third the features, less extreme extensibility and simplified parsing. If well-specified and interop-tested early and often, I imagine such a protocol could be implemented in framework form in a tenth the size of an HTTP server framework. In even more constrained cases, a very specific constrained transfer protocol implementation (e.g. one which supported a subset of CRUD operations and did no optional features) could be 1/100th the size of a simple HTTP Web server. That starts to be an appropriate size for burning to a chip and shipping in a $1.00 to $10.00 device.

I tried to get some sanity checking on my estimates, and it's not easy. A Web server can sometimes be a wrapper on a file system, in which case most of the logic involved comes "free" from the file system. For example, a simple implementation of one style of redirect might be a trivial number of lines of code if the underlying file system supports symlinks. Andrew McGregor pointed me to the Contiki operating system for embedded objects, which has a Web server already. So that's a proof that embedded and limited devices can sometimes do straight HTTP.

In sum, if interoperability, flexibility and executable size are all simultaneous concerns, HTTP as-is will pose problems. While it may not be necessary for all devices to use a more space-efficient REST transfer protocol than HTTP, it may be necessary for some. If it is necessary to do a new transfer protocol, it should be possible to design a protocol an order of magnitude smaller just by cutting features and unneeded options from HTTP; and possibly smaller yet by being very careful about parsing and limiting extensibility.

More later on REST itself scaling down.

A year in knitting: I knitted 6 miles of yarn in 2009.


Considering how much I spun in 2009, I'm pretty pleased. My stash definitely grew due to the spinning & gifts and purchases (5 miles purchased at Stitches West Feb 2009 alone, mostly not knit up yet).

1. Alpaca socks for Laurie (modeled), 2. fit of pathways 2 socks, 3. Zombiegurumi, 4. Vilai sock, 5. Prairie glass scarf, 6. Trekking sock, 7. Replacement travel shawl, 8. First handspun sock, 9. Amsterdam socks, 10. YAPS, 11. Panda wool retro rib, 12. Lavender shimmer scarf, 13. Petroglyph hat, 14. "Dove wing" icarus closeup, 15. Cowl from handspun., 16. pnw shawl on hangar, 17. zigzag fold, 18. Hawt sweater 1, 19. Malabrigo shawl.JPG, 20. Auction scarf 3 closeup

Defining a new URI or URN scheme properly turns out to be really difficult. I've been sponsoring drafts for such schemes for almost four years, and the same problems come up again and again. The major themes:

• Comparisons
• Uniqueness
• Stability

Comparisons

Comparing one URI to another turns out to be a commonly desired feature. Browsers look up cached pages based on URI comparison. If I click a link to bookmark in delicious.com, I'd like it to be bookmarked only once and if I've already bookmarked it, bring up that page so I can see what I tagged it with and when. Outside of http URIs, I'd still like to know if I'm already subscribed to an XMPP user, etc.

Comparison is harder than it sounds, but you already know that if you've dealt with any code requiring canonicalization, conversion or encoding. If a link in a Web page contains a space, at some point my browser has to convert that space to %20 to use in an HTTP request. Should the browser do that conversion before or after looking up the URL in the cache? Should delicious.com bookmark the URL with the space or the one that's used in HTTP requests? This is the tip of a very large iceberg that potentially includes all of internationalization and Unicode. Be very clear on what character set is used by each part of a URI, and if it's all ASCII, say so.

Case sensitivity is a frequent issue. Be very clear on which parts of the URI are case sensitive.

For new URIs, giving options makes the comparison job much harder. Let's say a new URI scheme needed to include a country designation: it seems nice to let users put a two-letter country code, a three-letter country code, a TLD or a OID in there. Only now one needs a horrid table to convert and compare these, and string comparison is no longer enough.

Optional syntaxes are similarly difficult; even allowing for '/' vs '\' can lead to error.

When the URI form is an alternate form for an identifier that already exists, now the URI may have to be comparable to something that's not a URI. For example, both IRI and URN forms exist for ISO OIDs. Don't they need to be compared to each other?

Can the URI form have query syntax? Is that part of the comparison or is that stripped off first? In HTTP URIs if I stripped off the query syntax I'd retrieve quite a different resource, but in some URI forms, the query syntax is used to carry information other than resource-identifying information. For example, can I compare two mailto URIs that have the same mail address, even if one of them has a query part with "?subject=Hey%20There"?

Can the URI form contain multiple values? The SMS URI definition had to include text on comparison when multiple SMS addresses were packed into the same URI. Does order matter to comparison?

Uniqueness

Frequently URI schemes need to avoid collisions, so that there isn't an attempt to give two different things the same identifier. The problem here is delegating the ability to create new URIs, while still avoiding collisions. The major fallback here is the DNS: URIs that contain a domain name, where the resource being named belongs to that domain, effectively delegate the uniqueness concern to that domain holder.

For example, we don't need to worry that "xmpp:lisa@jabber.org" will conflict with other resources, because the domain 'jabber.org' assigns usernames uniquely within that domain and prevents collisions.

The other main option is to use registries. For example, all OIDs, defined by ISO, use the ISO process to register numerical values and string values for use in the parts of an OID. Other times, IANA is the registry (e.g. for port numbers in HTTP URIs). If there is a new registry needed by the URI, this is more work and more to get right.

Some processes for ensuring uniqueness are quite heavyweight. Many IANA registries have processes which can take weeks or months to resolve. If the registrar is not IANA, who is going to actually run the registration process and under which rules? The OGC URN defined in RFC 5165 includes sub-namespaces issued by OGC itself . The first consequence of this is that the OGC organization must be referred to for any new OGC URN unless it explicitly delegates that part of the namespace. To reduce the burden of being a registrar in the case of non-permanent, test or experimental OGC URNs, the URN definition mentions the possibility of an experimental sub-namespace and the possibility of collisions within that namespace. Now implementors have to consider the possibility of leaked experimental names and dealing with collisions. The approval discussion of RFC5165 was lengthy, because of these nuances.


Stability

Some URIs need to refer to the same thing over only a short time, but typically the desired stable period is long or even longer. Domain names can be a problem here. Initially it might seem great to use HTTP URIs as XML namespaces, but consider whether the holder of the "example.org" domain will change over time, and whether the new holder will have the same policies regarding use and allocation of URIs in that namespace.

If a registry is used to achieve unique assignment, and the registrar is not IANA, then the stability of the registry must be considered. How long is the organization going to exist and maintain the registry publicly? We look for a public commitment, existing Web pages, a long-lived organization and so on. An explanation of the process and deciding factors for how names are assigned and how the organization ensures they are not reassigned, shows that they've thought about this commitment. See RFC 5328 for an example.


Random List of Gotchas

1. Does your URI scheme include use of fragment identifiers (like the #iri part of http://example.org/faq#iri)? Forget it; fragment identifiers relate to the media type of the *resource*, not the type of the URI. So if the URI "foo:bar:baz" retrieves a HTML page, then the fragment identifier would act like a HTML document fragment identifier.
2. ABNF is hard to get right. Get it reviewed by an expert. Use a ABNF generator or something like that to test your instincts. Refer to existing productions where possible. One common issue is to use a separator like "=" between two constructs, and then define one of those constructs in a way that it includes the separator character itself.
3. Another ABNF/syntax issue is to accidentally use a character that has an obscure meaning in URI syntax, or is simply reserved.
   
4. URIs that contain phone numbers include a whole barrel of troublesome monkeys. It's so hard to get telephone numbers right, with variable length and special encodings, '+' prefixes, dashes or spaces, and extension numbers, that it's worth trying very hard to use an existing phone-based URI instead of defining a new one.
5. If query parameters are used, can key values be extended? Can new key value be defined by anybody? Do they have global meaning (like mailto URIs) or purely local meaning (like HTTP URIs)?
6. The "community considerations" section required for URN registrations is frequently misunderstood. What the IETF looks for in this section is an indication that the work done to standardize this scheme and allocate a new scheme or URN type will add value; that there is benefit to the Internet community and not only to a private consortium or private company.
7. If any reviewer blithely says "whyever invent a new URI scheme, use HTTP for everything!" just ignore them until they provide actual reasoning for this proposal.
8. Embedding URIs within URIs, or any syntax that is infinitely extensible, is asking for trouble.

Final advice; provide more examples of actual URIs or URNs than you think people will need. Along with an example, explain how that example would be assigned, derived, and if applicable, dereferenced.

References

Here are the documents and registries that govern the registration and syntax of new URI schemes and new URNs.

• RFC3406:How to define a new URN namespace or "NID" or "Namespace Identifier".
• RFC2141: URN syntax, or the syntax of URIs that begin with 'urn:'.
• RFC3986: URI syntax, how to parse all URIs and URNs regardless of scheme.
• RFC4395: Guidelines and Registration Procedures for new URI schemes.
• IANA Scheme Registry: Existing registered URI schemes
  
• IANA URN-NID registry: Existing URN Namespace registrations
  

I remember reading Neal Stephenson's Snow Crash and his description of the Metaverse, his conception of virtual reality and online communication, thrilled me. I knew in many ways it was more realistic than Gibson's cyberspace. For instance, Stephenson described how people can choose their own avatars and it's a sign of a newbie or at least a non-programmer to have an "off-the-shelf" avatar, and indeed we see this in places from static online forums all the way to Second Life

One thing nagged at me back then: Stephenson realized that there's no reason not to teleport in virtual reality, but explained that the programming rules forbid it.

You can't just materialize anywhere in the Metaverse, like Captain Kirk beaming down from on high. This would be confusing and irritating to the people around you. It would break the metaphor... Once you have materialized in a Port, you can walk down the Street or hop on the monorail or whatever.

This is unrealistic in a virtual reality which is supposed to be the predominant way hackers like Hiro interact with each other online. Today, online gamers tolerate some limitations on teleporting in game environments like World of Warcraft or Puzzle Pirates, but even there, friction caused by do-nothing travel time is minimized. And in a more general communication milieu -- Web forums, Facebook, Twitter -- there isn't a single, limiting place. I can "be" on two forums at the same time on Ravelry, open two or more Facebook windows and chat with multiple people and I'm "there" with them all for some value of "there". Not only is there the ability to go immediately where I want to be in most online fora, but it doesn't even involve leaving the other "places" I already am.

Ok, here's another piece of the picture that didn't bother me in 1993 but does today:

Most avatars nowadays are anatomically correct, and naked as a babe when they are first created, so in any case, you have to make yourself decent before you emerge onto the Street... [Hiro sees] A liberal sprinkling of black-and-white people -- persons who are accessing the Metaverse through cheap public terminals, and who are rendered in jerky, grainy black-and-white.

This assumes an architecture where the client renders their own avatar. Even in that architecture, a proxy for a public terminal could render a classier avatar. Low-res displays would more likely affect the receiver than the sender -- somebody accessing the online universe through a poor public terminal might see every other avatar equally low-res, but their own avatar could still appear fantastic to people on good computers. It's complicated.

I guess the lessons are that today's online fora are less like the real-world than we could imagine fifteen years ago, and future online fora are less like the real-world than we are yet capable of imagining. We're still sending messages that look like paper mail and have envelope icons, and we still think of "bulletin boards" as a real model. We haven't integrated IM or twitter-like experiences fully into other experiences. Today, I'm downloading the Adium beta to see how twitter "what I'm doing" messages and community are integrated with IM and whether that improves on the old IM concept of presence in a significant way. Trivial interface changes in these sites and software can be significant in how people use them.

To borrow Ted's analogy when we touched on this over coffee today, we're in the same phase cinema was in when a movie camera was pointed at a stage, and a stage play acted upon it: the unique affordances of cinema weren't discovered immediately and are still being discovered even today. With online interaction, we're only beginning to discover how different it is from experience in the physically-limited real world.

I have a bunch of baking books, or cookbooks that include serious sections on baking. The Joy of Cooking and The New Best Recipe are my favourites by a long shot, and often I enjoy making the "best" scone even if the recipe is basically white sugar, white flour and a pound of butter.

However, sometimes I'm looking for a healthier scone, muffin or coffee cake -- something that I can eat for breakfast without too much guilt, or offer to health-conscious friends -- and I don't have resources that are just right for me. Ideally, a book on healthy baking would balance out a number of factors without being fanatical on any one of them:

• How is the whole grain content? Can some of the white flour be replaced with wheat, or can the recipe handle an optional addition of wheat germ, ground flax seeds, oats or so on?
• Can the sugar be cut down and/or replaced with honey or maple syrup?
• Can the fat be cut down without sacrificing moistness, shelf life, texture and flavour?
• Is the protein ratio good? Is substituting soy flour an option? Adding nuts?
• Are the ingredients readily available or can rare ingredients be optional?
• Is the taste pumped up? I eat less of pure, dark chocolate or tongue-tingling ginger sweets because my palate is satisfied earlier.
  

I understand some people get fanatical about one thing, just the sugar or fat or whole wheat content to a recipe, but I rather think balance is important and certainly taste is.

Along these lines, here's an adapted recipe for Mango Chutney coffee cake, derived from Light and Easy Baking. That book focuses only on reducing fat content, which I brought up a little again, but the pumped-up taste is there and the hot pepper is surprisingly good.

2-1/2 cups all-purpose flour
3 t. baking powder
1 t. salt
1/2 c. sugar
1/2 c. brown sugar
1 c. milk
1/3 c. canola oil
1 egg, slightly beaten
2 T. orange marmelade
1/3 c. raisins
3/4 c. chopped mango chutney
Additional pepper, cinnamon or cardamom, particularly if chutney is mild

Mix the dry ingredients together then mix the rest in. Bake in a loaf pan at 350 for 65 minutes.

Quick mommy blogging, just to say I'm still here.

A couple days ago I hear a scream and "Get it off me!" from the two year old in the next room. I go running. It's a piece of sticky fluff on his index finger from him poking under the furniture. Sarcasm kicks in but doesn't work:

Me: "Oh noes! It's a disaster!"

Him: "I has a zaster on my finger!!"

We had a really great IETF APPs area meeting today. We invited a whole bunch of people to talk about their topics and moved through the presentations quickly. These topics were:
- HTTP Resource Discovery
- Service/server Discovery
- Timezone publication
- SCRAM (Salted Challenge Response Authentication in SASL)
- Bayeux and cometd: JSON pubsub over HTTP
- BOSH for tunneling XMPP over HTTP
- rHTTP: reverse REST
- Analysis of all these server-initiated HTTP schemes plus Web Sockets
- Massive Multiparticipant Online Experience: the Overview

The slides are all already available here. Here's my favorite, a slide from Mark Lentczner's deck, for tying a whole bunch of things together. Even though Tufte would probably cry as Mark said.

I'm trying to read some code in Objective-C. This is hard, but it's solidifying my abstract understanding of programming languages. I'm not that hardcore a programmer, but I guess I've picked up a few things over the years (gawd that makes me sound old).

One of the neat things about Objective-C is that using a class or instance method involves sending a message. C++, in contrast, calls those methods. A C++ object has a fixed number of methods that can knowably be called. An Objective-C object might be able to handle arbitrary messages. This makes some things harder and somethings easier: polymorphism is easier; finding cases of using the wrong type of object or having a null object are harder to detect (must be done at runtime, not compile time).

The thing is, this is very familiar to me because this is how wire protocols work. In fact Objective-C has "protocols" which are interfaces, or a set of messages, that an object claims to be able to handle, so the terminology overlaps quite a bit. Anyway, in a wire protocol the client sends a message, and because anything can happen to that message, the client has to be able to handle a large number of outcomes. Polymorphism? You bet; a server that appears to be a HTTP server (implements the HTTP protocol) can also be a WebDAV server, a CalDAV server and an FTP server.

Designing protocols can be hard for people who think in terms of fixed interfaces à la C++. RPC-style protocols embody this thinking, making Remote Procedure Calls and expecting predictable, limited results. It makes more sense to me now, that RPC-style protocols are so brittle: designers and implementors are acting as if there's compile-time checking of the remote interface, whereas since the remote interface is on somebody else's computer that may have been upgraded or may just have a different implementation, of course there's no compile-time checking.

Mommy blogging today: Natasha said I should post about toddler sleeping stuff.

I have a kid that naps and goes to bed willingly and easily. Clearly there is a huge part of sheer luck in this because I've seen little correlation between loving, wise, firm parents and perfect kids, and I'm not always wise and firm. But we did luck out on a few things that have added to his personality to make for the easiest bedtimes ever.

1. We introduced an attachment toy early on. This toy, known here as "sleepy bear" because his eyes appear closed, is a blanket-with-head style minky toy.
2. We taught the sign for bear early on (bear hug yourself with arms crossed, and scratch your upper arms with each opposite hand) so he could ask for the toy pre-speech.
   

3. We attached a pacifier to sleepy bear with a folded strip of fabric. The fabric loop goes through the pacifier loop and around the pacifier, so it can come off for easy machine laundering of the toy.
4. We bought a second identical sleepy bear (and attached another fabric loop) when it became clear this was the favourite toy. Usually one remains hidden to be brought out very conveniently when there's contamination events or simply the bear has gotten too dirty from grubby hands.
   
5. Since sleepy bear is always "sleepy", he has to stay in the kid's bedroom most of the time. We make exceptions for when he's sick or at difficult times like coming home in the car after bedtime.
   

So now, when we say "It's bedtime" his response is (whining) "Nooooo...." but the next phase is "Let's go find sleepy bear" and he responds "OK" and follows us to the bedroom. Sleepy bear is closely associated with sleeping and triggers him to lie down and relax.

He is too young to say recognize fatigue and say "I'm tired, I'm ready for a little nap" but he's easily old enough to ask for bear. If we're at home and his hands are clean, we ask him to go into his bedroom and cuddle with bear until he's ready to come out without bear (and if he comes out with bear we bring him back to the bedroom and ask him to say bye to bear before he can go out and play again). So yesterday morning he did this and actually fell asleep, getting a bonus morning nap which he doesn't usually need any more.

When we want him to fall asleep in a new place (traveling or spending an evening at friends') we just bring sleepy bear. We pull out the toy and any old blanket, and that's enough for the kid to sleep in a new room fairly easily.

Finally, this somewhat limits pacifier use without ruling it out entirely. I don't care too deeply, but there's something annoying about a kid that talks through a pacifier all the time. Having the main pacifier attached to a toy limited to the bedroom means that most of the time when he's playing he doesn't have one. We do have a couple extras on leashes for carseat or stroller travel where it appears to seriously improve patience levels.


I don't mean to give advice because all parents are different and all kids are different, but I did agree it was worth explaining how this works for us. Good luck!

Messaging Architects put out a very nice press release about my joining the company. I used it as a bit of a soapbox to talk about what makes a fully Open Standard: free to read, free to implement and free to participate in.

I now work for Messaging Architects. I started a couple weeks ago but it's been busy; I traveled to Montreal last week to visit HQ and meet the management team.

It's going to be a fun job. The company is smallish (small enough for everybody to be on IM and see each other) but growing and building its product line. The M+Guardian product does spam control and other policy enforcement on email in transit, and I can definitely get behind spam control. The M+Archive is a bread-and-butter product for any company that has to follow regulations on email retention, which is a growing number, and I like the focus on swift retrieval. The M+NetMail email server was aquired from Novell a year ago and the company is now putting its stamp on the product (the team in Utah, who I met last October). In addition there's calendar integration, which you know I'm interested in, and possibly some file-sharing technology.

There's a lot of attention to customer needs at Messaging Architects, and a lot of enthusiasm and dedication. It's not hard for that to rub off on me even working from my own home! I'm in the midst of establishing a more fixed and attractive working spot at home, getting on IM with all my co-workers, joining regular meetings and getting the products running myself. Of course, I'm context swapping this new stuff with ongoing IETF work as I continue to handle the Applications Area Director responsibilities.

Thank-you to everybody who was looking out for me during the job hunt and if you're hunting, may you be as lucky as I.

My fellowship at CommerceNet just came to its expected end -- CommerceNet hires fellows for limited periods, to encourage them, rightly, to get cracking. I figure the fellowship was 75% successful.

Half my work was to continue as Applications Area Director for the IETF, which I continue to do. I got more efficient, I continue to learn a lot, and I even managed to publish a few documents (on consensus processes, interoperability testing and reporting and HTTP Email) that might lead somewhere. The HTTPBIS, IDNABIS and ALTO WGs all got launched. I helped 40 documents become RFCs. It's hard to measure success because there's always more one could do.

The other half my time was even more nebulously defined: I was to build a new venture or otherwise do stuff that would either reflect very well on CommerceNet or be a good investment. I wrote a paper with Rachna Dhamija, another Fellow, before she moved on to launch Usable. I worked with Phil Crosby to build a prototype of a sleep tracker Web application, and with Jeff Lindsay to build a prototype Web site for making public health information more accessible.

By the end of 2007, I focused on the public health information accessibility site. I built a short-term development plan and budget (or actually, several) and a long-term vision. The vision included supporting research by letting scientists share information with each other, benefiting from the same visualizations and data transformations that are required to make data accessible to ordinary people. The site was also intended to be highly contributory in the long-run, so that data collectors (whether government departments, academic or industry researchers or private organizations) could publish their data in this accessible forum.

In early 2008 I still had very little budget so I decided to build the live site myself. I brushed off my Python, taught myself Django, and wrangled with databases, graphing packages and CSS stylesheets, then with running a Linux server. I even did a logo myself. In the summer I launched openfindings.org.



During all this time, I presented and demonstrated to everybody I could meet: at over sixty people, that was more than one demo a week. I was looking for partners (I actually hate building stuff alone), investment, grants or a home for the project. This is where I signally failed; although I often saw enthusiasm and expressions of support, I didn't manage to get enough concrete support to keep the project going.

I hope the ideas make it out there, though. There's no excuse for taking public money to create vast collections of public data, and then make the public interface as bad as this, or this, or this. Even CDC has daunting Web forms and codes to know. Users need to be able to discover data in an exploratory way, learning about it as they go, rather than be forced to know all about the data in order to know how to query it before they can even see any of it. Can you do that? Yes. Users can browse topics and see thumbnails of data visualizations, filtering as they go, never having to fill out a Web form or learn an ICD code. Faceted browsing and rich interlinking of related topics/graphs (more than I was able to implement on openfindings.org) would make data browsing even easier and richer.


So that's the summary of what I've been doing "at work" for the last two years. I expect the next few to be interesting too of course!

Crafting in 2008

This mosaic only shows finished items, though the photo wasn't necessarily taken of the finished object in all cases.



1. Dumbledore top in progress, 2. Baby's initial on baby toy, 3. Outer stripes, 4. chinese yarn socks, 5. Anna socks modeled in mirror, 6. purple Fair Isle socks, 7. crochet fruit, 8. bigass spider, 9. limited-edition-rib-2, 10. color-chevron-scarf, 11. angora-parallelograms, 12. Souvenir of Ireland almost done, 13. Logan River Scarf, 14. Cell-automaton-full, 15. Cable-trim-neckline, 16. Primero scarf detail, 17. Riverbed sole, 18. Essential Stripe from kit, 19. Duet mohair wool beret, 20. Duet mohair wool mittens, 21. Diasilkombre progress, 22. Diakeito Diadomino yarn, nearly two balls already knit up, 23. Rainbow tie, 24. Faux laceup foxglove socks, 25. Rainbow spiral socks

# started in 08: 27
# finished in 08: 26
# given away: 16
# quilts: 3
# sweaters: 1
# pairs socks: 7
# scarves and shawls: 9
# stuffed toys: 3
# things for Darwin: 2
# made from old (>2 yrs) stash: 10

Based on the list of completed projects, I'm inspired by new yarn, old fabric and by having a deadline or recipient!

This little triptych makes me happy... I love wearing these vivid purple socks, but they're nearly five years old. Luckily I still had the yarn leftover from making them, and the soles of handmade socks consist of "stockinette stitch" which is easy to darn. Done! I plan to get at least another five years out of these.

The HTTPMail proposal I mentioned here now has a discussion list. I've already received a few good comments, building a list of changes to make when I go back into editing the draft. However, I'm in no hurry to do so. People are still discussing¹ what they think it's for; which use cases they believe in or not, and which use cases I forgot to mention. That's a good discussion to have before nailing down what features it has to have.

[1] discussion so far on the general Apps Discuss mailing list; discussion really hasn't started up on the special purpose list yet.

This is pretty exciting news from Amazon: they're launching a service to freely host public data sets. Part of building openfindings.org was about that problem, but Amazon can do a better job of storage than a smaller organization can. Problems still to be solved:

• finding the right dataset -- somebody needs to do data-focused searches on terms like "lyme disease" or "lung cancer mortality", and this probably needs a bit of ontology work
• automatically generated visualizations appropriate to well-known kinds of data
  
• Wikipedia-style annotation, comments, highlighting: people not only mining and analyzing the dataset in private, but also in public and benefiting from each others work

Add this to the long list of data-oriented Web efforts in the last two years (palantir, swivel, flu trends, flowing data to which I owe a hat tip, and more), and it's the hot thing calendaring was a couple years ago when eventful and meetup were launching. Financial services were the first I saw to do sophisticated dynamic online visualizations of data, followed by Google Analytics, and it's only starting from there.

I'm following up on my last rant on multiple transports. It's a topic with nuance, but I still think it's a worthwhile caution to avoid multiple transports in non-realtime standard protocols.

Yngve talked about multiple transports used by Opera Mini for cell phone Web browsing. This is in the context of what appears to be a proprietary protocol; at least it's not standardized. Opera Mini talks to a transcoder run by Opera, so interoperability is mostly handled by not having multiple vendors of the client or the server. Skype was another example, but again Skype does not use a standard. More generally, Yngve talked about TCP-based applications that need to bypass firewalls and can use HTTP as a tunnel. Yngve, you're right about the general world of protocols, but what do you think about broadly standardized protocols?

Ralph and Joe pointed out that there are multiple implementations of BOSH, the HTTP binding for XMPP. This is interesting; I was aware of the proposal but had no idea how widely adopted it is. If it really becomes universally supported in servers, then we could write any new application on top of XMPP and it will work over HTTP as well as natively over TCP. Now that's interoperability!

Herewith, a rant: multiple transport bindings considered harmful.

I still hear this and it's getting increasingly annoying. People consider it a good thing for a standard to be able to run over HTTP, BEEP and something else. Has this ever proven to be a good idea? Layering is good for other reasons, but not because it gives implementors a choice that leads to interoperability failure in many cases.

Is it a failure on the part of the designer to understand the usage characteristics of their protocol, and successfully map that onto TCP (connection-oriented), HTTP (stateless respond-and-forget) or something else?

SOAP is supposed to be transport-independent and offer choice, but as I overheard last week, there's a reason they call it web services. And the ultimate in multiple-transport wankery: I once heard somebody propose a schema which they said would run over SOAP or HTTP.

Are there use cases I'm unaware of, where this has been a really good thing for some standard?

I'd like to be able to explain the important confluence of several ad-hoc Web standards to various people, so I paid attention Monday at IIW. This is the diagram I saw:



I can't call this a stack. I just can't. It's not a protocol stack or an API stack or a library stack. The best I can say is that "Some people call it a stack". There's no layering relationship between these things; there's not even a sequence or an order.

PortableContacts extends Open Social and the rest are independent proposals that work well together. It's a suite, a cluster or a collection of protocols/APIs. Here's a first stab at a moderately simple diagram that does not mislead:

Comments?

The idea of using native HTTP resources to RESTfully access an email store is not only an old idea, it's been implemented many times. Some Web mail architectures are even somewhat RESTful although the purest implementations are not themselves Web UIs -- these RESTful email interfaces have typically been built to support Web UI frontends in a classic tiered architecture.

I have been talking about this for so long that it seems self-explanatory. Emails can be HTTP resources with persistent URLs and machine-readable representations. Mailboxes can also be HTTP resources with persistent URLs and machine-readable listings of contents, we just have to agree how to represent those listings. Here's my proposal.

Atom feeds is my choice for organizing those listings. I think it's worth explaining the two overriding reasons why.

1. Atom allows clients to just GET a representation of a feed, and the feed can contain an arbitrarily long list of items, but paged according to the server's needs. This is a great allocation of responsibility, making client logic simple and putting server performance in the server's hands.
2. The default model for feeds is that the same object can be in more than one feed. This is very important for the usability of email going forward. I know a few people who organize their mail into strict hierarchical collections, typically using IMAP, and find old email by jumping to the right place in the hierarchy -- but I know far more who rely on giant inboxes, saved searches, flagged email or tagging. The default model for feeds directly accomodates all those usage models.
   

I was very happy to find once I used these principles and applied the feed model, that it fits very well so far. It's possible to implement the spec as a proxy between an IMAP server and a client. It's possible to access a mail feed with an unmodified Atom-capable newsbrowser. This bodes well for implementing and deploying.