Sometimes programming goes too fast, and I actually have to stop and think.

I miss working closely with other developers; that's part of why I'm blogging right now.
Python exacerbates that, because there's so many ways to do things, particularly the kinds of things I'm doing right now, like parsing text files with tables in human-readable but not-particularly-computer-readable formats.

Here's a tiny example: I needed to remove commas from strings like "1,207" before converting them to integers to graph them. I thought of using slicing, which is a powerful Python feature on lists, strings and more:

>>> string="1,207"
>>> string[:string.find(',')] + string [string.find(',')+1:]
'1207'

It got ugly fast so I looked for something else:

>>> string="1,207"
>>> string.replace(',','')
'1207'

Splits are also powerful:

>>> string = "1,207"
>>> ''.join(string.split(','))
'1207'

Of course I could also define a simple "removefrom(char, string)" function:

>>> string="1,207"
>>> def removefrom(char, string):
...     return string.replace(',','')
... 
>>> removefrom(',', string)
'1207'

To find good ways of doing things, I end up browsing a lot of Python code online. That's frustrating because some sites have started hosting "sample code" mostly as a way to put advertisements on-screen and in pop-ups. And I'm sure the most trivial code review would identify plenty of areas my code could be much more Python-clever.

Internships for CommerceNet this summer: I'll be the one hiring and managing. If we get a great proposal for a project from the right intern candidate, CommerceNet might simply hire that intern to work on that project. Otherwise, here's what I have in mind.

CommerceNet is an entrepreneurial research institute, dedicated to fulfill the promise of the Internet. We are currently seeking Software Engineer interns to implement a data visualization Web application for public health information. Involves JavaScript and Python, both data access and graphics. CommerceNet may also accept proposals for internships to work on well-specified projects of the intern's own design.

What you'll do

• Develop open source libraries or widgets for graphing and data visualization
• Build public service, community oriented Web site
• Be part of a small team or work nearly independently
• Develop with minimal guidance, using rapid iteration and feedback loop and with leeway in choices of tools.
• Borrow, create or collaborate on visual design and visual elements
  

Required Skills:

• Web Applications development, including CSS and JavaScript
• Python or demonstrated ability to pick up languages
• MySQL or similar data management experience
• Great ability to extrapolate from raw ideas to realistic implementations.
• Demonstrated initiative pulling a project forward
• Some experience using graphics libraries
• Familiarity with Cleveland or Tufte principles would be a bonus
  

Email ldusseault@commerce.net with questions or cover letter and resume.

There is so much IETF work on email these days. There are five separate WGs (one in the Security area), a Research Group and a couple informal efforts. I tend to have to summarize the work to pull it together in my head. Here's the post-71st-IETF sitrep. Also see Barry's post for more detail on SIEVE, DKIM, LEMONADE and ASRG. Document links collected at bottom.

• The IMAPEXT WG is so close to shutting down, it did not meet last week. One of its work items was to internationalize parts of IMAP (including mailbox names, and how to sorting strings like subjects) and those documents were delayed but finally got approved.
• The LEMONADE WG met, and seems to be winding down. Although its extensions are all linked by being useful to mobile email clients, there are some extensions there of general interest.
• The SIEVE WG just finished publishing a whack of documents around its new core spec, RFC5228. At its meeting, the group discussed whether to recharter to do another round on the core SIEVE documents and standardize some more filtering extensions.
• EAI WG has requested publication for most of its documents. These are Experimental Standards for using non-ASCII characters in email addresses, which affects IMAP, POP3, SMTP in interconnected and complicated ways.
  
• A design team is nearly done updating SMTP (RFC2821) and the Internet Message Format standard (RFC2822). They're handling "last call" issues on the list.
• DKIM -- having previously published its core signatures doc (RFC4871) and requirements for signing practices (RFC5016) -- is now working on the signing practices standard itself.
• ASRG, the Anti-Spam Research Group, is documenting various anti-spam techniques.
  
• In informal discussions, several of us keep talking about rearchitecting email access. However, nobody's ready to predict, let alone commit, that their company will implement something new. Does that mean that there's not really enough pain around using IMAP? Or that the pain is the user's and not the software vendor's?
  

Documents:

• IMAP Sorting and Threading (in IESG evaluation): draft-ietf-imapext-sort-20
• IMAP i18n (Approved as Proposed Standard): draft-ietf-imapext-i18n-15
• IMAP Annotations on messages (Approved but waiting on completion of SORT): draft-ietf-imapext-annotate-16
• IMAP LIST Extensions (Approved but waiting on completion of i18n): draft-ietf-imapext-list-extensions-18
• LEMONADE Profile, to replace RFC 4550: draft-ietf-lemonade-profile-bis-08
• ... other LEMONADE docs in progress listed on WG page
• New SIEVE base spec: RFC5228
• Other new SIEVE specs: RFC5229, RFC5230, RFC5231, RFC5232, RFC5233 and RFC5235.
  
• EAI/I18n Delivery and Status Notifications (in Last Call): draft-ietf-eai-dsn-06
• EAI/I18n Addresses in SMTP (in Last Call): draft-ietf-eai-smtpext-11
• EAI/I18n Addresses in Mail Headers (in Last Call): draft-ietf-eai-utf8headers-09
• Other EAI documents yet to come include documents on the mailto URI, mailing list considerations, downgrade mechanisms, IMAP support and POP3 support.
• SMTP Revision (in IETF Last Call): draft-klensin-rfc2821bis-08
• Internet Message Format Revision: draft-resnick-2822upd-06
• DKIM Sender Signing Practices: draft-ietf-dkim-ssp-03.txt
• ASRG: DNS Blacklists and Whitelists: draft-irtf-asrg-dnsbl-04
  

One of my favourite tools for keeping on track is doing a regular report to my management.
Ever since Max Dunn, a former co-worker, told me about the "four P's" (progress, plans, problems, personnel) mnemonic, that's been a useful concept. Writing down some notes in each of those categories makes me remember stuff better, catch up, and plan better. I get almost all the value even without sending the report.

Right now I send a somewhat different monthly activity report to the Apps Area Discuss mailing list and to the Working Group chairs of the WG's I advise. It serves some of the same purpose to me, and other people have found it useful. I indicate what I think is the status of each document I'm sponsoring to become an RFC, and if the authors think the status is different (e.g. one of those common "I was waiting for you while you were waiting for me" traps) they tell me. I mention a couple notable things at the top of the report, and try to summarize what state each WG I advise is in.

I try to send these in the first week of the month, which means I'm two weeks late for March due to being sick and having the IETF meet last week. Maybe later today??

Although I'm mostly doing IETF stuff at IETF this week, I took advantage of Andrew's expertise while he's here too, to help me get a Python library running and working. I had gotten blocked last week and we worked through several more problems this afternoon. Some notes follow for the record, to add to the couple blog posts out there that help one out on these problems.

I was trying to install pycha and play with its simple charting. To do this, I needed to have py-cairo and cairo version 1.4.12 installed on python 2.5.

The easiest way to get cairo installed is using MacPorts. This is fine, although it installs cairo into the MacPorts version of Python, not the Mac OS X version of Python. The Mac OS X version of Python already has cairo installed, but it's apparently an older version. I didn't know how to upgrade the system cairo library so I focused on doing this with the Python installed by MacPorts. This led to later complication.

After apparently successfully installing cairo version 1.4.14, attempts to install py-cairo failed. It kept on insisting that only cairo 1.4.10 was available. Andrew tracked this down, and we're pretty conviced that the information that the MacPorts cairo package applies to pkg-config is false: it says version 1.4.10 right inside the 1.4.14 cairo.pc file. We edited this file directly and went on.

Next we tried to install py-cairo using the regular "python setup.py install", but that failed -- it is not set up to work on Mac OS/X and there were numerous errors. MacPorts rescued me again, thanks to a hint we ran across from KenKeiter -- this port package has a bunch of hacks (string replacements in directory paths it looks like) to make py-cairo work on Mac.

Finally, back to installing pycha itself. This has a nice handy Python Egg available, so I installed "easy_install" to install that. I goofed and installed that in the default System version of Python, so while installing easy_install worked, installing the pycha egg failed. It turns out there is a MacPorts port of the same thing, called py-setuptools. Only you don't want that one, you want py25-setuptools if you have Python 2.5. I installed that.

OK, back to installing pycha for real again and it finally worked. One gotcha in following the main documentation page example, which is otherwise very nice -- I needed to "import pycha.bar" for the sample code to work (Update: Lorenzo Gil Sànchez fixed this already). And it did work.

Proof:

Mike from work pointed me to EpiSpider, which does dynamic graphs and uses in part an in-browser library for faceted browsing called Exhibit, from the SIMILE project. This project also has a beautiful Timeplot library, again in-browser.

The basic approach here is minimalist: all a site developer needs is a place to host Web pages, some of which are data files. No database, no content management system. Still, those have their places too, and I can already see how to make great use of Django features as well as Exhibit and Timeplot features, and I know databases. To tie these together, one just needs a bit of AJAX code to call a server-side process to pull the appropriate subset of data from the database in JSON or other simple format, and I know that exists too.

I had been ready to try generation of static chart images in order to save time -- that was why I was trying to use R (also the data import, manipulation, analysis and export features could be useful). But ultimately good online charts need features like mousing over to see point values, scrolling, and zooming; static charts can't provide that but in-browser canvas code like that used in Timeplot can.

Sample here.

Update: removed annoying iframe

Nearly two years ago, I made a prediction that Atom would replace WebDAV. At the time I was even working on revising WebDAV and just beginning my involvement with Atom standardization. I told this prediction to Cullen Jennings and documented it in a note dated 5/9/06. "Replace" is a fuzzy term, because they're not equivalents, but here's what I see today.

• Google turns Atom, RSS and AtomPub into GData and uses that for blogs, calendars and task lists for starters. Google does not use WebDAV as far as I know.
  
• Microsoft is unifying its developer platform protocols on Atom and AtomPub, and using those protocols for unstructured application storage, e.g. photo albums to blogs. (h/t James Snell. Hey, I used to work with David Treadwell, haven't talked to him in years.) WebDAV support in projects like Exchange is downplayed.
  
• IBM has a bunch of projects using Atom but it has a bunch of projects period and I haven't seen strategy announcements about either standard.
  
• Apple uses Atom and RSS in quite a few applications, although it's also using WebDAV and CalDAV on .mac and in its calendar server.
  
• Blog service sites are ubiquitous and all use Atom or at least RSS. File sharing sites or other WebDAV public services are rare. Photo sharing sites are more likely to use Atom than WebDAV by a long shot.
  

Were I to propose CalDAV today it would probably be CalAtom -- some things would be easier, some harder, but it would catch a wave instead of drifting in the tail of something that was never much of a popular wave. Oh well, we needed something then, and WebDAV gave the most leverage at the time.

The Django stuff was going so well, I switched over to learning R a couple days ago just to depress myself. It's tricky to get into. The documentation is very scattered (kudos to those who are improving it at the wiki but that's a work in progress). I haven't yet found a way to get documentation for arbitrary libaries. The cute name 'R' makes it very hard to Google for R libraries. E.g. I haven't yet found a core graph or a library function that creates horizontal bar graphs (though maybe somebody will tell me to do vertical bar graphs with all the labels turned, then turn the whole image).

One headache in particular was trying to work with display strings that needed to be converted to numbers. I had a table containing this tiny fraction of data (out of 8 columns and 15000 rows):

ICD.Chapter    Crude.Rate
A00-B99        22.0
C00-D48        193.2
D50-D89        3.2 (Unreliable)
E00-E88        33.7
F01-F99        21.0
...

To graph the Crude Rate without the occasional "Unreliable" strings, I needed to break the information about which measurements were unreliable into a separate column (not losing the information), reduce the Crude Rate to be a number and convert it into an R 'numeric' type. I spent about 8 hours. The first four I managed to solve the first problem: I looked for a "contains(string, substring)" style function but there was none; "grep" produced errors that were really hanging me up whenever the input cell did not have the string I was looking for. Finally I wrote my own little contains function using regexpr:

contains <- function(pattern, string) { 
  regexpr(pattern, string) != -1 
}

It looks simple now, but it was tricky to get a function that worked properly when converting 15000 cells to 15000 new cells.

The next four hours were spent trying to remove the "(Unreliable)" string from the cells that had it. Again poring through documentation, I tried various approaches and got the closest with this one:

first.word <- function(x) { 
  substr(x, 0, regexpr(" ", x)[1]-1)
}

The problem was that although this one worked fine on single strings when I tested it, it didn't work on turning a whole column into a whole new column.

Today at 11:00 am I turned to Excel to see how quickly I could do it in Excel. It's been a long time since I've used Excel formulas, but I knew this kind of thing could be done. In many ways the Excel formula documentation is worse, but it's a smaller set of controls (or at least a clear subset) and it was comprehensive and organized.

In Excel the two new columns are created like this from the H column:

=NOT(ISERROR(SEARCH("Unreliable", H2)))
=IF(ISTEXT(H2), VALUE(REPLACE(H2, SEARCH(" ",H2), 13, "")), H2)

I'm sure this could still be done in R but making those two formulas work for this particular table took me 20 min in Excel instead of four hours.

Development work went much smoother today. I set aside installation of Apache and MovableType and decided I could do something easier at least for now. Believe it or not it was easier -- for me at least -- to implement a minimalist blog in Django, from the database on up to the display pages, than it was to install and hook together software.

I also wrote some of the core data model for the site using Django's standard model tools. It's a site that will improve the accessibility of health-related statistics (costs, health surveys, incidence registries and other epidemiological data) by offering good search and lookup and simple clean visualizations. So I started by creating data models for data sources, report names and conditions that are tracked, and hooking those data models primitively into the Web pages where I want them to show up.

Of course development involves writing new code, but that's not nearly all of it. Even when writing new code, part of development is debugging: when you know what you want to do and have chosen a way to do it, but it's not working. Another chunk of new development time is decision-making, e.g. deciding how objects will relate or how interfaces will look. It's interesting how fast this style of development goes from decision-making to debugging and back, without a lot of time spent writing new code.

Any quick pointers on
- unittesting Django sites?
- Integrating C libraries into Django projects?
- Departing from Django DB models -- when to create one's own tables and how?

Since Ravelry ate my blog, I'm going to see if I can't restart this with a different idea. I originally started a blog to try to say clever (and subtly snarky) things about news, particularly science news. Then this turned into a lame knitting blog. Now I feel like I don't know what to post here. So how about I post what I like, eh?

I am trying to learn how to use a camera and a flash better. I bought a Canon Speedlite 480ex so that I could point my flash in different directions, dial up or down the light intensity, and use a diffuser. I also tried to buy a hot shoe adapter and cord so that I can use the flash at a distance from the camera, but it looks like I might have bought the wrong model of something. The cord has a little plug just like a single earphone plug, and there's a dock space on the Speedlite, but it's empty. Behind the hole there's no socket, just plastic.

Still, the Speedlite is better than nothing. I used the diffuser for some of the shots taken of people in a big, poorly lit conference room at the Ravelry meetup.


I think it turned out better than with a standard Rebel built-in flash. I had fun anyway.

At work, I'm immersing myself in Django and oh-my-god real setup of Web, database and other server software. This intimidates me. When I get something working (like an index.html page with working stylesheet and logo links in Django) I take a little "success" break to let the feeling last.

I hate how easily this kind of work can go wrong. There have got to be common ways of doing things that just work. But I don't know what those are, and software for developers errs on the side of flexibility and power, rather than on the side of making choices for you so that it just works. And some choices work but are awkward. Where do I install Apache? Where do I locate the static files served by Apache? Where do I install MySql so that it can be used by MovableType as well as Django? Do I upgrade to Python 2.5 inplace or have it co-exist with Python 2.3? Invariably on the Mac there are permissions issues: do I change the permission of the file so I can do what I'm trying to do with my login account or do I switch to 'sudo' the command? I normally get things working eventually but it's sooo slow.

What pictures of curious and fast-crawling babies often look like.

Clearly my lack of posting is explained by the fact that I have died in a knitting accident. It was the size 1 lace Addi turbos I've been knitting the cellular automaton shawl with, those are some sharp needles.

Maybe it's just a metaphor for this blog: I haven't been posting much since I joined Ravelry, so the blog was what died in the knitting accident.

Have I anything else to say? I've been thinking about email software, and I concluded that a really great email-focused program would include GUI design contributions from a game designer. Email management needs to have the same smoothness and responsiveness that a good console game does. I deal with hundreds of emails a day; dealing with them should be so smoothly software-assisted that I'm eager to click to kill the next one. I mean file it. Instead my flow gets interrupted constantly by tasks that draw me away from getting through my email.

I've been pretty busy but just had time to learn a bit of JavaScript and the canvas element. Since I'd been playing with cellular automata ideas in knitting, I put together one of those. Pretty!

Blogging here is going to continue to be light for the foreseeable future. I am now a member of Ravelry, so that's a more obvious place for me to post knitted projects -- the whole point of doing that was so that other knitters could find my projects as they research what project to do next, as I do. Ravelry was built for that purpose and makes it easier for knitters to post their own projects, and to find other projects done with a certain yarn, or with a certan pattern, etc.

But I did want to post a link to this Telegraph article by Liz Hunt, and that doesn't fit naturally either on Ravelry or on the CommerceNet blog. The article bitterly critiques a book called The Gentle Art of Domesticity (not available in US yet), by Jane Brocket. I wish I could figure out what has Liz Hunt so bitter, because I can't see anything intrinsic in the work of this enthusiastic cook and craftster to inspire the tone. The comments on the Telegraph article vary from wondering how Jane Brocket could be so "perfect" to angry reactions to the article's tone and criticism of a domestic lifestyle.

I've been following Jane's blog since she started it in early 2005, staying as she progressed her writing style and particularly her photographic abilities (from this to this, for example) even as her posts on knitting became more and more infrequent. Most of what she documents and extols I have no desire to imitate. I'd rather bake oatmeal squares and butter tarts than elaborately decorated cakes and "fairy buns". I hate gardening and do not want to live in a remote town or in the country. I do knit and sew but I've been knitting for five times as long as her blog has been up so I already have my own style and inspirations. But it's a gentle, beautiful picture she paints, and it's like visiting an English village house and garden for a quiet moment when I read a new post on Jane's blog.

The only way I can reconcile this with the Telegraph article is the tension over feminism vs. domesticity again. Liz Hunt brings up careers and Shirley Conran in opposition to the domestic values. I haven't got a copy of Jane's book yet but I suppose Jane even started the argument by extolling domesticity as an antidote to ambition and stress. But I'm pretty sure that if a woman wrote a book extolling jogging, swimming, line dancing, playing cello, online gaming, sculpture, reading, bird-watching, photography or dirt-biking as an antidote to ambition or stress, it would be seen as trite at worst rather than threatening.

I frequently try Web applications out just for the sake of seeing what they can do, and usually abandon using them. But I like keeping up with the field, particularly for email, calendaring and task-related applications.

Today I tried one that does all three, that I heard about originally from Rael Dornfest (back when I worked at OSAF; it wasn't available then and it is now). I might abandon this online app too because I like to manage my time offline better than online, but there was one point when testing the application sent a frisson of pleasure down my spine. Usually that reaction is reserved for either more private pleasures, or when fondling yarn.

The site is stikkit.com, launched last year. The site is very smart at telling what kind of note you're creating, detecting todos and events from linguistic cues. For example, it detects Quiltathon on September 30 as an event and puts it on the correct date on the calendar.

The 'frisson' moment was when I created a task to email somebody. It was recognized as a task by the phrase do send mail to So Andso. Then I noticed that the new task had a linked stikkit for a person: "So Andso". When I clicked on that, the automatically-created stikkit had a line reading name: So Andso. I intuited that if I added another line to the stikkit with email: soandso@example.com would make the new stikkit into a contact, and voilà, it did!

It was a little disappointing to find that while Quiltathon on September 30 was detected seamlessly as an event, September 30: Quiltathon was not. Also when I had a space between the "name: ..." and the "email: ...." lines in my second attempt at creating a contact stikkit, it didn't work until I removed the empty line.

Oh, and I love, love love applications with hotkeys so that I don't have to mouse around. Even better when they can be discovered.

These cute kid items were knit by Uta Hayward for her son. I'm completely tickled to see the blanket because it's the pattern I published on For the Love of Yarn. It looks very different in the multicoloured yarn chosen by Uta's son, but it looks terrific. I like the way the colour pooling forms little arcs because of the decreases, and that the little arcs are separated by the yarn-over columns. Maybe I'll knit the pattern again myself in multicoloured yarn!

Happy Independence day. Then again, I'm working today -- I've got IETF stuff scheduled tomorrow with people who aren't in the US and they'll be prepared.

I would have said Happy Canada Day three days ago, but I was travelling home from Canada on that day. There were nice people on the plane with maple leaf body paint or temporary tattoos or something. Even nicer, the man who switched seats to give me more room for the baby, and his wife who held the baby while we switched.

I am having fun despite the work (hey, work's fun too). I just discovered the FireFox plugin Tab Mix PLus, which I have used to set it up so that each new tab opens to my local home page which has quick links to stuff I use a lot. Yay smoother work processes!

I was also sucked into Facebook last week by my cousins and siblings. I initially felt too old but then a bunch of people in my class in grade school and high school contacted me immediately so I already feel like I belong a little. We were making fun of Facebook at work recently. One jibe was that it's a classic force multiplier: it allows you to form cliques and shun people at great efficiency! Before you could only shun people in sight, while out socially -- but now you can shun people anywhere, anytime!

Reading Tukey's Exploratory Data Analysis:

If we want to see what our plots ought to tell us, there is no substitute for the use of tracing paper (or acetate). If we slip a well-printed sheet of graph paper just below the top sheet of a pad of tracing paper, we can plot on the top sheet of tracing paper almost as easily as if it were itself ruled. Then, when we have the points plotted, some boundary or reference lines drawn, and a few scale points ticked, we can take away the graph sheet and look at the points undisturbed by a grid. We often gain noticeably in insight by doing this. (And we have had to pay for a sheet of tracing paper rather than for a sheet of graph paper.) [...]

An alternative that:

• can be even mre effective,


• is no more expensive,


• takes a little more trouble to prepare for,

replaces the tracing paper by the thin sheets of transparent plastic (acetate) made for use in overhead projectors. Two cautions are important:

1. You can only use markers specially made for the purpose. [...]


2. It is important to keep one's fingers off the plastic until the picture is completed. (A piece of thin graph paper, placed upside down, works very well as a hand shield.)

Much of this classic textbook is obsolete tips and tricks -- e.g. for collecting data on paper and checking that you've copied it accurately to another piece of paper. The notes on cost of tracing paper vs. graph paper vs. acetate strikes me as quite funny now!

I think I learned to graph by hand in school but never had to except as an academic exercise -- I was using graphing software before entering University (summer job in a water testing lab). I wonder what kids today do in school, if they even bother with the academic exercise. I hope so, because it was kind of fun playing with numbers on paper. For some values of fun :)

Random Stuff

If you know me, you know I've been pretty busy and no surprise that I've been neglecting my blog! Rather than let the blog languish for even longer waiting for the perfect post, I'll just be random today.

1. The Atom Publishing Protocol (APP) is up for approval by the IESG this week and I'm the sponsoring Area Director. I'm excited about APP because it will serve as a model or basis for much future work in accessing and editing structured content on the Web. (Will the IETF APP area one day be assumed to stand for Atom Publishing Protocol rather than APPlications?) I observed some of the Atom Interop event last month (summarized by Tim Bray here) and it was great.

2. I am apparently a sucker for some kinds of advertising copy, not necessarily what one would call good copy. Witness: here is the entire Product Description for the ice cube trays I bought online when my freezer was mysteriously not making ice cubes automatically, before it mysteriously started to again.

Ice Cold Fun! These wild and wacky Flexible Ice Trays are just plain crazy. The flexible designs allow ice to just pop out! Its quick and easy. And fun! Better still are the nutty shapes the ice forms into Stars or Hearts! Wild! Fun! Each tray container contains twelve individual molds perfect for all the fun ice youll be making. Blue star tray and pink heart tray. Made of high quality rubber.

Nutty shapes! Wild and wacky! Oh the fun we'll have with ice cubes!

3. I continue to knit stuff. I just don't have time to post about it any more. Here's a shot of me with other knitters at a meetup at Nine Rubies, and the project at my feet is now a completed scarf (with an estimated 40,000 stitches, which is why I normally don't calculate stitch counts).

4. I also made a mei-tai baby carrier. Again, this photo is obviously not taken by me, otherwise I probably wouldn't have a photo of the carrier at all. I don't even have photos of the sweater or bag I finished a while back.

5. Looks like IETF Chicago will be exciting: the Apps area will have BOF meetings on HTTP, notifications from email stores, and personal address book access.