Using DynamoDB, work in progress

At work we're using Amazon Web Services' DynamoDB for a backend.  This is early days and a work in progress, but I thought I'd post about what we're doing so far because I've seen so little elsewhere about it.

Our Web framework is Ruby on Rails.  Rails is a system that favours convention over configuration.  Most RoR developers use ActiveRecord, Rails' built-in system for object modeling and abstracting away SQL database access.  If you stay on the rails, this works fantastically.  Rails automates or partially automates many tasks and systems, from migrating your data when the model changes, to setting up unit tests that conveniently setup and instantiate the things you want to test.  Building on top of this, many Ruby gems extend your Rails functionality in powerful ways (Web UI test automation, authentication and user management, access to social network sites).

As soon as a project starts to diverge from Rails conventions, trouble begins.  Trouble may be contained if the difference can be contained and made as conformant as possible to the default components.  For example, when writing a API that serves RESTful JSON resources instead of HTML, it's best to figure out how to use views to serve the JSON in the same way that HTML views are generated (a topic of a few posts I did a couple years ago).

Which brings me to Dynamoid.  Amazon's Ruby gem for access to DynamoDB is very basic and exposes DynamoDB architecture directly.  That can be useful but it doesn't behave anything like ActiveRecord, and in order to use Rails' powerful tools and extensions, we need something that behaves as much like ActiveRecord as possible.  The only ActiveRecord replacement for DynamoDB that I could find, that was at all active, was Dynamoid.  So I'm pinning my hopes on it.  AFAICT so far, it is incomplete but has "good bones".  I've already fixed one tiny thing and submitted a pull request, and intend to continue contributing.

Next post will be about testing in this setup.

Correctness impedes expression

In kindergarten and grade one these days, teachers encourage kids to get their thoughts onto paper any old way.  They don't explain how to spell every word and they certainly don't stop kids and correct their spelling.  For a beginning writer, it will likely be all caps, and the teacher may not even suggest spaces between words.  Here's my first grader's recent work:

wanda-yi wan-t toa school 

and i wat to room 

four it was nis 

iasst the tishr wat 

harnamwas martha She 

was nisdat war day sH- 

Em

This means "One day I went to a school and I went to room four.  It was nice.  I asked the teacher what her name was, Martha.  She was nice that(?) were day she (unfinished?)"   Don't you love the phonetic "wandayi" for "One day I" ?  I do.   Note that the letter "I" is used for long sounds like in "nice", because that makes sense before one learns that that sound can be spelled many ways including "ie", "i?e", "aye", or "y".

Okay, cuteness aside, I flashed to thinking about XCode while Martha explained why they teach writing this way:  it's hard enough for a kid, writing slowly and awkwardly, to get three words out onto paper, let alone a whole page of writing.  Many kids get intimidated by corrections and worry of mistakes.  Instead of answers, she gives them a whole bunch of resources: try sounding it out, think of a similar word, try looking somewhere else in your own writing, see if the word is somewhere else in the room.  Above all, she encourages practice and resourcefulness rather than perfection.

Unlike Martha, XCode is like the stereotypical teacher from 60 years ago who would stand over you constantly and warn if she even thinks you're about to make a mistake.  "That's wrong."  "No, it's still wrong".  "That's somewhat better, but still not good."  "Now that's right, but now this is wrong."

Maybe that's why I still use TextMate for Ruby.  If the code doesn't have the right syntax, I'll learn about it later.  (I write tests.)  But for getting an algorithm out of my head and onto the screen, I much prefer not to be corrected and warned constantly while I'm doing it.

AWS Persistence for Core Data

I like DynamoDB, and I like architecture that reduces the amount of backend engineering one needs to do in a company whose product is an app.  So I was quite interested to investigate AWS Persistence for Core Data (APCD, for lack of a better short name) in practice.

APCD Overview according to me

APCD is a framework you can install on an iOS app such that when the app wants to save an object to the cloud, it can set APCD to do so silently in the background.  Not only does it save the object to the cloud, but changes made in the cloud can be magically synched back to the app's Core Data.  There's a parallel framework for Android which is promising for supporting that platform with the same architecture.

On the server end, if the server needs to do some logic based on client data, the server can access the DynamoDB tables and view or modify objects created by the applications.  In theory one doesn't have to design a REST/other interface for synchronizing client data to the server or to other clients. That's a significant savings and acceleration of development, so we read up on APCD around the Web and implemented it.

While there were a bunch of minor problems that we could have overcome, the primarily one was: nowhere does Amazon seem to document how to architect to use AWS Persistence, or explain what is it for. In the main article linked above, the sample code and objects are "Checkin" and "Location".  But where's the context?  Are these Checkin and Location objects in the same table?  Is there one giant table for all data?  Does each client have its own private table for a total of N tables?  Or are there two tables? Or 2N?   It really helps if new technology documentation  includes some fully fleshed out applications to give context.  Full source code isn't even what I'm talking about, but at least tell us what the application does, why it's architected the way it is use the new technology, and some other examples of what the new technology is for.

What I think APCD is for

Well we recently put together a couple facts which suggest what APCD is for.

• You can't have more than 256 tables in a DynamoDB for an account, even when using APCD.  This limitation is very relevant to architectural choices made with APCD.*
• If an installed app has the key to access any part of a table, the app can access the whole table, all objects.  There's no object-level permissions yet, and because the app access the data on DynamoDB through APCD, the server can't intercede to add permissions checking.

All right, so that tells us we can't architect the application so that each app instance saves its own table separate from other apps' tables.  We run out of table space at 256 installed users if not sooner.  It also tells us that if apps are going to share larger tables, the information in those tables has to be public information.  

So that suggests to me that APCD is for apps to synchronize shared public data.  For example, an application that crowd-sources information on safe, clean public bathrooms.

How my sample app would work

The crowd-sourced bathroom app could have all the bathrooms' data objects in one big table, and each instance of the application can contribute a new bathroom data object or modify an existing one.  A server can access the bathrooms data too, so Web developers could build a Web front-end that interoperates smoothly as long as the data model is stable.  

Now to use the service, even if the whole dataset is too large to download and keep, an app could query for bathrooms within a range of X kilometers or in a city limit, and even synchronize local data for offline use.  When the app boots up it doesn't have to download local bathroom data if it has done so before, instead APCD is supposed to fill in new data objects matching the query, and update the client with changes. 

For security, we have to trust each app to identify users so we can identify and block simple bad actors (somebody using the app interface to insert false information), and we have to have some backup for dealing with the contingency where the app is completely hacked, its key is used to access the bathroom data, and somebody quite malicious destroys all the useful bathroom data.  

What we did

We ended up not using APCD because what we're building does not involve a shared public database. We have semi-private data objects shared among a small set of trusted users.  Doing that with APCDs limitations seemed too far off APCD's garden path of easy progress.

Is there a better way to use APCD? 

*   Yes, you can have the 256 table count lifted, but not by much.  Not, say, to 1 million. That's not how DynamoDB is architected to work well.

Opportunities arising in fall 2013

Working on a new project using cutting-edge AWS stuff and iOS 7, I note some opportunities.

1.  A really good Ruby library for working with AWS.

Although Amazon really should hire more Rubyists, this could also be done by outsiders.  AWS is powerful and Ruby is powerful.  Hooking them together properly would be sooooo nice.

2.  An iOS module or framework for higher-level use of Amazon Persistence for Core Data (APCD)

APCD is intriguing but Amazon has a lot more work to do.  For example, objects can only have one properly-persisted relationship between them.  You can't synch an Event object with an "organizer" relationship to Users as well as a "creator" relationship to Users.

Whether Amazon does more work here or not, there's opportunities for people to build on top of this service, because it doesn't address problems like version skew between mobile app instances.  For that matter, I'd like some explanation what it's for -- a real-time background table synch service is good for something but what exactly did the architects have in mind?  Without knowing what it was built for and tested for, it's hard to know whether the service will work smoothly for the applications I'm thinking of.

3.  Documentation and examples for the new XCode unit testing

There's vast amounts of information out there on unit testing with Java, Python and Ruby.  There's blog posts upon blog posts of best practices, and many great questions and answers on Stack Overflow.  But when it comes to XCode, I can't successfully google for "unit test method for filling in a text field in ios".  Apple, why do you hate Web searches?

Ok.

Would somebody get on these please?

Thank you.

Systems thinking vs algorithm thinking

I was chatting with another programmer about our different styles.  He's an incredible algorithm solver.  He's done compression and encryption algorithms in school, and codecs and video processing and GUI animation effects since (he's 23).  I tried to explain the kind of problem that I'm attracted to, which none of those are, and used the word "systems problems".

"But isn't everything a system?".   Only in the most trivial sense.

What I was trying to distinguish by talking about systems problems and systems thinking in programming is modeling independent and interconnected agents.  I'm not the only one with this kind of definition.  In an interesting publication on managing social change, I saw the definition "Systems characterised by interconnected and interdependent elements and dimensions are a key starting point for understanding complexity science."  Very close.  Is there a better phrase for system-style solutions as opposed to algorithm-style solutions?

Another way I explain this approach when I'm being self-mockingly post-modern is to say "I'm interested in the liminal spaces in computer architecture", which is an arty/jargony way of saying I'm interested in the interfaces between agents: APIs and protocols, typically.   I also hear the words of a British-accented UWaterloo professor from 20 years ago, saying "Modularization!" and "Information hiding!" over and over.  (Systems thinking supports object-oriented design.)

I've worked with a ton of people who have the same mental models because I worked a lot in the IETF for ten years.  It's a necessary part of communications security in particular, because in addition to thinking of each client and each server as ideal agents, one must always think of bad actors and flawed agents.

Coming back to startups, I'm always surprised in a way when I talk to a programmer who doesn't design and implement protocols and APIs because they think so differently.  It's more justifiably shocking when I meet people who know about and implement REST and aren't used to systems thinking!

I'm reading Don't Make Me Think by Steve Krug, and just read the section on home page messages.  That's why I laughed out loud (surprising my cat) when I saw this home page:

Amazing, huh?  It's got simple pricing!  Free, pro or on-premise!  What does it do?  Not important!

If you scroll below the fold -- I swear this is exactly what I see above the fold my very first visit to the site -- there's a "Full Feature List".  Now (and you really can't make this stuff up) I can see that Teambox supports

• Capacity (this is a header)
• Users
• Projects
• Storage
• Organizations
• Hosting
• Support
• Premium Features (this is the next header)
• Group Chat

In other words, I still have no clue except that Group Chat is apparently a premium feature.  Wow.  At this point I don't even want to know.  I feel like knowing what the service or product is would only detract from the delicious infinite possibilities of Teambox.  I don't want the service, mind you -- I'm happy with the inscrutable mystery of Teambox.  Some things are not meant to be known.

Kevin Liddle makes a case against cucumber in his blog post.  Since he doesn't have comments, I'll basically comment here.

I agree with Kevin that the idea that product managers will write cucumber tests is pretty weak.  They might well read them and understand them however.

I think, however, that the value of cucumber and its gherkin syntax are, as with many things, not exactly in the place where the designer thought the value would be.  The value is in using anything but ruby to describe what you're trying to accomplish with ruby.  Every so often I'll write or encounter a test written with the same narrow view with which the programmer wrote the code. Such a test verifies that the code reliably does the wrong thing in the larger sense.  Using ruby to test ruby also encourages trivial tests where the implementation detail is verified, not the application logic (see Don't Unit Test Trivial Code  )

My science-based (but not scientific) theory is based on how low-level thinking impedes high-level thinking.  As an example, this happens when you're driving, thinking about something low-level like adding distances, and get distracted and miss your freeway exit.  Cognitively,  when you're in the middle of writing Ruby code and thinking about how to write Ruby, it's harder than normal to think "what should the code do".  Switching to another language, gherkin or anything else, prompts the programmer to go meta. Going meta means repeatedly re-loading the mental model of how what i'm doing is fitting into a larger system and goals.

This effect is known in a couple different fields:

•  Learning math: "Good problem solvers possess metacognitive skill, the ability to monitor and assess their thinking" (ref Support for Learning)
• Corporate strategy: "A strategic thinker has a mental model of the complete end-to-end system of value creation, his or her role within it" (wikipedia Strategic THinking)

So besides just changing to another language to avoid ruby-testing-ruby circularities, Gherkin is designed to make the programer think in terms of wants and fulfilling user expectations.  The syntax "Given I am a new user, When I go to the home page, Then I should see the zero-content display" helps the developer pop from what she's trying to do to how she's trying to do it, and back, without losing the big picture.

An information coordinator is useful

An information coordinator is a useful person, and after two years of working with a partner or alone, this weekend was a nice reminder of that.

The classroom camping trip was Saturday night, and since I haven't been working full time this summer, I volunteered to organize it.  What I did:

• Printed out last year's camping duty list, announced that I was posting it two weeks ago.
• Collected the duty list and sent out email for the last few needed duties.
• Advised the shopping volunteer what to buy
• Sent out a public email answering the questions individuals had asked
• Kept track of which spots were free and directed arrivals (sounds like work, but wasn't really, more like socializing)
• Triggered volunteers when to begin lighting the grill, and the campfire
• Approved people's suggestions (people wanted to know if a suggestion would interfere with other plans so wanted some kind of coordination check, not really approval)

This didn't seem like work to me but it met with widespread gratitude.  An organized point of information exchange is really what I was, timekeeper, and a maker of trivial decisions.  Software projects call these people project managers, but they exist in all kinds of domains, sometimes with different names, sometimes with specialized expertise.  Sometimes the project manager is just an organized person holding the clock, the task list, and the notebook.

Order of Operations

I recently taught my son to ride his bike without training wheels.  He was very resistant and afraid of falling down.  I discovered that the order of learning skills was very important.  Before he could get confidence balancing and moving, he needed to be confident that he could brake and put his feet down at any time. This is a coordinated movement between hands and feet as well as body balance (which foot? which side? when?)  so not as simple as it seems.

Over twenty minutes on two days, we practiced braking dozens of times: I would hold his bike up while he put his feet on the pedals, help him move forward pushing the pedals, and then either tell him to brake, or let go and he would wobble and brake on his own.  Eventually one time he forgot to brake and just kept going: breakthrough!  So the order of learning was

1. Learn how to stop
2. Learn how to go straight on his own
3. Learn how to turn
4. Learn how to start on his own

Despite having a sore lower back from holding up his bike so much, I thought this worked well.  But what a strange order to learn in!  Then I remembered how knitting is most often taught.  The teacher will cast on a bunch of stitches and do a few rows, so that the knitter can (1) go straight, then learn to (2) turn at the end of the row, then (3) bind off at the end, and finally someday (4) cast on a new beginning.

UX for Lean Startups required reading

UX for Lean Startups: Faster, Smarter User Experience Research and Design by Laura Klein
My rating: 5 of 5 stars

I loved Laura's book.  As I read it I kept on putting it down thinking "I need to put this down and go follow her advice IMMEDIATELY" and then I would pick it up because I wanted to learn more and hear more of her voice.   Since I know Laura I could hear her voice advising, explaining, and gently mocking commonly-held falsehoods.  The tone combines with the topic matter to break down pre-conceptions, to convince and teach.

Laura's advice is incredibly practical. Having just been through a startup I could immediately see what I could have applied, and working with other startups now I do get an opportunity to apply more ideas.  Many ideas are only obvious in retrospect (like testing fake features when you're operating on a shoestring budget) and then even once the idea is obvious, there's great advice for making the most of the idea.


View all my reviews

Bleak House

Bleak House by Charles Dickens
My rating: 5 of 5 stars

I love Bleak House.  Dickens uses incredibly lush and complex metaphorical descriptions for London, estate houses, and especially, the Chancery Court.  What makes this novel one of my favourite Dickens books is the sweet and good unreliable narrator, Esther Summerson.  She is not perfect (which is even better than perfect).  Her imperfections lie in her lack of self-confidence and too-trusting nature.  Even as she tells in the nicest and most trusting tone about some other character's actions, the reader can tell that other character's truer nature.  And eventually Esther comes around to the reader's opinion, saddened if the other character turns out to be less than her ideal.


View all my reviews

Two meetings was I invited to, 
And sorry I could not both attend, 
at two o'clock, what could I do,
the time was firm for each, I knew,
so I composed regrets to send.

I chose the meeting Paul had called 
because my thoughts were needed there,
and progress on V2 was stalled 
because the techs were so enthralled
with a framework with no market share.

And while Paul's engineers discussed
Their plans for building on Hadoop, 
And writing all their code in Rust, 
And if the crowdsource trend is bust;
I thought about that other group.

I'll have to read the notes they took,
No matter how obtuse or dense,
and at their data models look,
and find a time I can rebook,
If I am to make a difference.

.... apologies to Robert Frost.

Modern Web architecture, cookie-cutter, view from the trenches

Web architecture has come together quite nicely of late.  The components work together better than ever before, and it's easier to put together a great Web site or service -- putting more emphasis on the design and user experience, of course, which remain important and hard.

This list of steps is an overview of the cookie cutter approach.  It works for most new Web sites and services.  Many people I've worked with believe they have something different, a special wonderful unique thing that is not right for the cookie cutter approach, and to they extent that they're right, their differences should be minimized (focus on making your value differentiated, not your architecture).  So, if you're getting started on a new site or even a new service within a service, try to put off being special and different as long and as far as you can.  To minimize early development costs while remaining flexible and scalable (new hires as well as increased usage), try to make the default choice for every step below until it's proven you need to be different.

I came up with this list by using these components myself in situations both early and at scale, by talking with lots of other developers about what actually works and is easy, and by keeping abreast of developments by following a few blogs and twitterers I admire.

1. Pick a modern Web framework

The popular favourites are Python and Django, and Ruby on Rails.  If you are or have a technical person already, personal preference is a fine way to make this choice.  These both have data abstractions, views, templates, lots of inherent flexibility, and a rich set of free extensions (e.g. Ruby 'gems').  If you're thinking about node.js, see "Qualms" below.


2. Pick a data store

Hosted MySql or PostgreSQL both work great for a long time and a lot of scaling.  Use hosted databases for online test, staging and production and either the same database or SQLite for development and local testing.  It would be fine to just go with whatever is cheapest and easiest, which might mean making this decision together with "pick a hosting service" (E.g. Heroku uses PostgreSQL by default).  NoSQL solutions go under "Qualms" below.


3. Pick a hosting service

Look.  Just don't run servers yourself, OK?  Not until you're the size where your hosting costs annually are approaching the annual cost (roughly twice salary) of a dev/ops employee.  Don't use EC2 either, that's still a lot of manual work.  Startup efforts don't have the manpower or processes to handle an early new customer and a vulnerability disclosure somewhere in the Web stack, in the same week.  So leave the operation and upkeep of the Web stack up to the hosting service. I like Heroku.


4. Design and implement landing page

This is where you'll need to pick a CSS layout library like Skeleton.  Use this library's columns and rows to lay out your pages.


5. Plug in a user login module if you need login

Either pick one that supports passwords, email verification, forgetting passwords etc, or pick one that supports using Facebook, Twitter, Google, etc identities.  Plugging this in should be very quick if you're still on the cookie-cutter path.  Bootstrappers can configure their service to send those validation and forgotten-password emails via a free GMail account initially.  Django comes with authentication, whereas for Rails I've used both AuthLogic and Sorcery.


6. Add JQuery to your Web site if you didn't already do it for the login feature

A combination of other Javascript libraries could be used to replace JQuery, but JQuery does both common widgets and functions (like handsome buttons) and AJAX -- the AJAX part is necessary for accessing data from your service in the next two steps.


7. Start with some kind of site template 


You probably need Web page headers, footers, and common colours and fonts.  A combination of Web framework page templates, including partial pages (e.g. the Web page footer is commonly a partial Web page that is included on every full page in your site) and CSS templates solve this.


8. Work out your core data model, using your Web framework and migrations

Your core data model is that special wonderful thing you're building.  It could be real estate listings, restaurant reviews, task lists or BBQ recipes.  Frame it as objects and collections.


9. Expose your data as a Web API with JSON, and show in the Web page with AJAX. 

Read up on how your chosen Web framework supports REST and apply it to your data model.  Read up on how your chosen Web framework supports AJAX.  Often the way to expose the data model is a direct extension of your data model. However, sometimes it's driven more by the view.  If the UI always shows collections of comments, you don't need a way to get just a single comment; you jump straight to "GET all the comments for this item."  Use JQuery or another Javascript library to make those calls and load data dynamically into pages.

This step and the previous step will happen over and over again as you add functionality.

Additional steps

• While in development, use free email; soon move to a hosted email provider.

To begin with use your Web framework's default email manager (ActionMailer in Rails), and just use any IMAP account to send mail, such as a free GMail account.  This is one of the things you'll need to upgrade sooner rather than later, by moving to a paid, hosted email provider.  Sendgrid is working OK for me and took less than half a day to sign up, set up and switch all test/demo/production servers to.  Sending SMS is similar -- in addition to choosing a SMS service you'll need to plug in a module.


• When or if you need native mobile or 3rd party app support, firm up your Web API

It's pretty cheap to get started with mobile functionality via HTTP/HTML/AJAX, at least for prototype, demo or even MVP.  There are probably cases where a native app is needed right away, but that's not too common.   The mobile client can use the same Web API that the javascript "client" uses, only it now becomes more important for that Web API to be stable, because a native mobile app or 3rd party app won't be updated at the same time as the Web site and API are updated.


• When or if you need to do any authentication with other sites or services, use OAuth.

OAuth comes into play for two situations: to authenticate users to another site such as Twitter, but also to authenticate apps or other sites that get special permissions on your service.  Rather than cook up your own special key logic, just use OAuth, plugging in one of the libraries that already exist.


• While you move along, let your Web framework help you as much as possible.

Django and Rails both have a ton of useful stuff.  The biggest one in my view is their database migrations assistance.  Bundler, in Rails, is key to looking after all the gems you'll have after the above steps. Try to also be familiar with the scripts and scaffolds that allow one to build new functionality fast in the completely cookie-cutter format. 

• While you move along, start to add automated tests

I add these right at the beginning.  Selenium, and capybara let me automatically load Web pages from my site, fill in forms, press buttons, and do simple tests on the results. RSpec and its mocking features are critical both for unit tests and for mocking out dependencies.  I run these myself and also, as soon as there are more than 1 developer, on a continuous integration server.  I'm currently trying out Semaphore as my first hosted integration server but would still go to the trouble of running Jenkins on EC2 if Semaphore doesn't work out.

Qualms

Node.js has great promise to unify Web programming under one language, bridging the divide between Web frontend javascript programmers and Web backend programmers using Ruby, Python or other.  A lot of people I respect  for being simultaneously practical, realistic and visionary, do like Node.js.  My qualm is that there is no standard framework to use with node.js -- the level of services provided by Node.js is more primitive than that of Django or Rails, and thus additional components are needed.

NoSQL is very powerful and often appropriate in large-scale Web sites and services.  I've used it before.  However, I have qualms suggesting a NoSQL solution at the outset, because the tools to hook frameworks and NoSQL together, as well as the tools to manage NoSQL systems, are not the default tools most familiar to Web developers, and not the most tested and evolved.  That said, a startup with experience using NoSQL might, without my labeling them crazy, use hosted Mongo, Redis or CouchDB, provided they still abstract away the store from the Web framework's data model.  In the long run this is something a hugely successful system probably needs to scale.   A combination of SQL and NoSQL is especially powerful for large or complex services but if you're at the "pick a data store" step you're not there yet.

Summary

The subject matter or domain used to matter a lot when a startup began to build a Web site or service.  These days, it doesn't seem to matter as much -- most startups I've seen can benefit from following most of these steps.  It's a real benefit to keep the startup's special thing limited and cordoned off (think of it as making your specialness more concentrated).  It's easier to hire and onboard people, as well as to benefit from community advances, when the overall Web architecture is cookie cutter.

Girls in games

The cultural gestalt these days seems to include more talk about women and video-games than ever before.  Suddenly, women are 47% of video-gamers and game designers (and game advertisers) are still figuring out how to deal with that.  If you haven't seen any of Anita Sarkeesian's videos before, they're really good.  Her concepts help me understand a bunch of common patterns and gave me names to more easily identify them (the "smurfette effect" for example).

Anita's concepts have not ruined my enjoyment of video games, but I may be getting more selective.  We have a PS2 and have been playing some old games.  One of them is Okami which is an outstanding game in many ways: playable, artistic, incredibly rich and creative.  In the game, the player's character is a wolf who begins a quest to regain divine powers the character once had, and rid the world of demons and darkness so it can bloom and the sun can shine again. When the world blooms it's simply gorgeous; all the art is painterly and beautiful.

Although the wolf fights, and is the best fighter in the world, its gender is ambiguous. My son thought the wolf was a 'he', missing the repeated interactions when the wolf meets other minor gods and they always call the wolf "Amaterasu, mother of us all".  Throughout the world there are reasonable ratios of male and female characters, adult and child and often the female is more powerful than the corresponding male (in the main city, the empress is clearly more important than the emperor).  Both male and female characters are killed off, not just the females.  Females need rescuing more than males but not exclusively.  So there's more balance than normal.

A few things grate only a tiny bit; the main non-player character who fights with a sword is alcoholic, lazy, scared and male, and goes off to save his brave, supportive lady love.  The character who seems good but is secretly taken over by a demon is called "busty babe" over and over by the pixie (and she is pictured as having unreal boobs), and there are scenes about this 2-inch tall male pixie trying to literally get into her shirt.  All in all, did not make me love the game any less -- the overall balance is so pleasing that an annoyingly-stereotyped interaction could be enjoyed as mild low humour.

And then there's this kind of thing.  *sigh*  I thought the Internets were supposed to know all about me by now and target ads right at me.  Clearly the Internets know I'm a gamer, but can't they tell I'm female?

Commentroversy, plus owls are the trendiest fowl.

Commentroversy is a five-year-old word according to Urban Dictionary, but it was new to me, when I saw it in the comment section of Yarn Harlot's latest blog post.  And the storm in a teacup that has so many commentators making puns about stitches and needles? The USOC asked Ravelry not to use the term "Ravelympics" to describe the event in which knitters set ambitious knitting goals during the real Olympics. Read the Yarn Harlot or Gawker, I think the shitstorm is funnier than the issue itself and don't have much else to say.

In unrelated news, I bought this t-shirt because it made me laugh.  But then I had to try to explain it.  Does it have something to do with YOLO being an acronym for "You Only Live Once"? I do not know.  But I did have a suspicion that owls are an arty crafty Thing.  And I confirmed they are a Thing, without even counting macrame owls.  You thought that ducks and chickens were overdone on home decorations?   Owls are the trendiest fowl by far.  In my totally scientific study based on Etsy search totals, here is the order of popularity of animals in kitsch:

Bird: 294,000
Butterfly: 144,000
Cat*: 118,000
Owl: 112,000
Dog*: 110,000
Fish: 67,000
Bear: 63,000
Horse: 56,000
Rabbit: 49,000
Mouse**: 47,000
Monkey: 43,000
Elephant: 42,000
Bee: 32,000
Deer: 25,000
Frog: 25,000
Ladybug: 24,000
Turtle: 23,000
Duck: 22,000
Lion: 20,000
Turkey: 19,000
Pig: 18,000
Fox: 17,000
Chicken: 17,000
Cow: 17,000
Sheep: 16,000
Sparrow: 15,000
Wolf: 13,000
Eagle: 10,000
Crab: 8,800
Panda***: 8,200
Parrot: 6,900
Beetle: 5,800
Lizard: 5,700

Rat: 4,400
Gorilla****: 1,200

* I subtracted the hits for pet accessories, because there's handmade dog and cat collars, beds, and toys, which don't count as Cute Handmade [Animal] Themed Tchotchkes.  If you just count housewares, for example, owls beat cats.


** Turns out, half the mouses on Etsy are Minnie.


*** You think you're cute, panda?  You're small-time, niche, pissant cute.  You can't even reach the tail of a stuffed owl.  I mean, really, you got beat out by crabs.  Go home and eat shoots and leaves.


**** Disturbing.


***** Tiger and Zebra were omitted due to the hard-to-control-for number of results with tiger eye gems, tiger stripe and zebra stripe fabrics. ***** <-- self-referential footnote

Better software development estimates

Is it possible to estimate software tasks better?  In certain circumstances, my experience tells me yes!  It's not just a wish (/me waves to commenter on last post).

First, to establish what kind of estimates I'm talking about.  It's not possible to estimate anything past two months, in my experience.  When I was working at Microsoft and heard lots of estimates from other teams, I realized that "This software product will take us two years" really meant "We have no idea and we will never finish a product that recognizably matches our plan".  A one year estimate turn into a somewhat recognizable outcome in two or three years.  Even a six month estimate, while it might turn into a year-long project fairly reliably, ends up finishing quite different tasks than the planned tasks which originally led to the six month estimate.

Other things that make a difference in estimation accuracy:
 * What language is being used?  Snags can be a bigger time dilator with a compiled or lower-level language like C, whereas hitting a snag in Python might not throw out the estimate that much.
 * Is something new being integrated? Any time a task involves compiling and linking a new library, or adding a new Ruby gem, I know the estimate is weaker.
 * How routine is it? Adding a new page to a Ruby on Rails project can be pretty damn predictable.
 * How many people are involved?  Estimating a one-person task is way more accurate than estimating a team project.  Even a one-person task that has an issue that requires another person's answer is less accurate than the task that can be done by one person independently.
 * How many other estimated tasks are dependencies?  If you have a chain of tasks to estimate, a change in one task can throw out the others.

And then there's bias.  Have you ever noticed that some developers always estimate high and some always estimate low?  It's intriguing, because even though they're consistently wrong, they are consistently wrong in the same direction.  That means they are actually giving management good information, if management knows their biases.  I once managed one optimist and one pessimist in the same team for over two years.  The optimist always estimated about 1/3 of his actual time required. The pessimist always estimated about three times the actual time required.  I would assign what looked like three months of work to the pessimist, and what looked like a week and a half of work to the optimist, and they would finish around the same time.

One of the things I really do love about Agile (and Pivotal Tracker reifies this) is how it understands the above points.
 * Task length? Agile encourages developers to break work down into smaller tasks (no three-week tasks in any agile process I've ever seen)
 * How many people are involved?  Agile encourages issues to be resolved before the estimate is even made, and it is designed for single-person tasks.
 *  How many unfinished dependencies are there?  Agile encourages planning with a shorter horizon, so the chain of dependencies is usually reduced.
 * Consistent bias?  Agile tracks velocity, not accuracy, so a consistent bias is simply a constant part of a consistent velocity.

With all that, one of the main things that intrigues me, and this is what I'm unpacking from my previous post, is whether better feedback would help developer estimates get even better than Agile already makes them.  Agile does not measure time spent so it doesn't give developers feedback that would allow them to either fix a consistent overall bias, or to start to recognize tasks that need to be estimated a little higher.

Scrum thinks people are stupid

One thing I both love and hate about the hardcore agile process, particularly the Scrum process, is that it tends to work around participants' biases, weaknesses, and lack of trust, rather than try to fix them. Scrum may not even leave room for improvements in trust or reliability. Some examples:

Developers' estimation biases are worked around, not fixed
Because developers have biases in estimating tasks (biases that are consistent over time and on average), scrum "velocity" measures estimates against results.  It doesn't attempt to fix estimates, e.g. by showing a developer the difference between their estimates and time spent.  In fact, velocity doesn't measure time spent at all, and lumps all developers in a team, over-estimators and under-estimators together, in one velocity measurement.

This is probably for the best -- it's one simple measure that's remarkably consistent.  Still, I wonder if it wouldn't be more useful in the long run to learn to estimate better.  I've never seen a really good estimation feedback loop in the software development context, but wouldn't it be neat to try?

Product owners' changes are either completely allowed or disallowed
Traditionally engineering teams have to train product owners not to change the product plans all the time.  This involves frequent team arguments about product plan changes.  Instead, scrum tries to carve out one small space where product changes are forbidden, and allow all other changes without argument.  In the current interval, the product owner cannot make any changes or the current plans are all tossed out and the estimation process is restarted, a consequence severe enough to effectively forbid any small changes.

This rule does keep product owners off the developers' backs.  If the product owner is thinking of a change to this week's plans, the consequences of an "abnormal sprint termination" probably stop them.  If the product owner is thinking of a change to next week's plans, the upcoming sprint planning meeting is where they'll discuss it.  Either way, the product owner does not walk up to a developer and say "Hey!  I've got a great idea!"

If keeping the product owner off the developer's back sounds like a really good idea to you, well you probably haven't worked with trusted, experienced product owners.  And if the team has processes that reify the distrust, then there's less chance to build trust.

Demos help people who can't analyse abstract plans
Weekly demos are a course correction mechanism. In order for anything to be marked done, it must be demoed.  And when a feature is demoed, the product owner can see the practical consequences better than they could when the feature was designed.  Now the product owner is able to immediately add things to the backlog, which might get done in the next iteration, and this is good.  Iterative design for the win.

The practical consequence of this appears to be less specification and planning work, which is good (avoid overplanning), but taken to the point where the product owners do not feel any particular pressure to understand and analyze the design.  Instead of sitting in front of the wireframes and thinking it through -- "What happens if I press this if it were implemented?  If this were a real system and the item had been deleted, what would the user need to see? " It's a difficult what-if skill and takes practice.   It doesn't make design any less iterative!  Instead, it moves the iterations into the design phase rather than the costly implementation phase.

Frequent demos, and the scrum rules which allow for any changes in the backlog, seem to remove some of the need to develop abstract design skills.  That makes me a little sad.  Still, frequent demos and iterations are a tool I'd use in any software development process.

Optimizing considered harmful
In hardcore scrum, developers are practically forbidden from doing any changes now that would make future work easier.  It's discouraged, and the way the system tracks tasks makes it unrewarding to do.

Let's say that today my task is to create a feature for users to delete items. There's also a story in the backlog or icebox for undeleting items (e.g. finding them in a trash folder and returning them).  Well, the way success is structured in scrum, I estimate the time to delete items at the beginning of the period where I do that work, and it makes everybody happier if I figure out how to delete items without much work, and we make more features fit in this period.  It doesn't help me now to estimate high in order to prepare for the 'undelete' feature.  It doesn't help me in the future either -- when we get to the scrum meeting where we estimate the  'undelete' feature, it might not even be me doing the feature.  (In theory, developers are supposed to be interchangeable in agile/scrum).  Even if it is me,  it's no big deal to have to do the undelete by rewriting the way delete worked -- I just build that into my estimates for this period.

There's no overall project schedule view that would have showed the value for doing these two features together and doing delete right the first time.  There are other ways where optimizing is actively discouraged:

• Literally, optimizing speed or resource use is discouraged.  Functional stories only say that the feature has to work.  Other stories in the future might or might not say that the feature has to work in 2 seconds.  Security might also be an afterthought!
• Optimizing by expertise is discouraged.  Everybody in the team is supposed to be interchangeable.  If all of this week's features are GUI features, then everybody does GUI work. If all of the next week's features are security features, then everybody does security work. 
• Optimizing the sequence of work by engineering constraints is discouraged.  If it would be faster to do feature A after the more general feature B is completed, too bad.  If there's a pileup of work with dependencies, and it's slipping to the end of the backlog where the dependencies will start to slow down each task, too bad.  Only the priority sorting of the product owner is valid.

I've seen passionate arguments for "You Aren't Gonna Need It" (YAGNI), and they're right.  Engineers often optimize prematurely.  Engineers often predict what they think the user will need and can be wrong.   But none of the passionate arguments for YAGNI think that it can be used without a sense of balance, right?  So when scrum methodology encourages developers to always put off optional work, scrum puts its thumb on the balance.  It short-circuits the discussions of complex tradeoffs and simply says do it later. 

Summary

What's common among all these traits is an attitude in scrum not only that people are fallible, but they're routinely wrong, and that they can't be trusted to work together on complex issues for the greater good. That means that it works best in environments where that attitude is more true -- e.g. a contract development team, comprised of decent engineers but not particularly specialized, who don't know much about the end use and maybe don't care too much, working with product owners who aren't experienced in software design, are way too busy to write specs and discuss complex tradeoffs, and are always tempted to change things they shouldn't.  

Agile should be done differently when the engineers are specialized, care about what they're doing and what it will be used for, work closely with the product owners in the same business unit, and the product owners are smart and can be trusted to learn good software development practices.  Just how it should be done differently is an open question to me. There are agile process variants that take different attitudes (e.g. those that encourage planning a whole release) and for the kind of team I am currently working in and enjoy working in, I'm interested in those variants.

One more thing about using Sorcery in a Rails project

One more thing about using Sorcery in a Rails project. I did mention that to make unit tests work, I had to stub out "current_user" in every single controller:

  Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  end

Even in Cucumber scenarios that do not use one of these steps that stub out current_user, I found that I had to explicitly unstub the method. I did not investigate why this is the case; I'm guessing that Cucumber would have to reload the whole environment between running each scenario, which would take too long. Perhaps there's a better way for Sorcery to provide "current user" functionality without patching each controller, or perhaps there's a better way for me to test mocking logged in users. In any case, this works, explicitly adding a step to make sure that the mocked-up logged-in user is no longer stubbed in:

  Given /^I am not logged in$/ do
    begin
      VisitsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    begin
      DoctorsController.any_instance.unstub(:current_user)
    rescue RSpec::Mocks::MockExpectationError
    end
    // etc...
  end

Pennants Afghan pattern

I finished a major knitting project and wrote up the pattern for it as a PDF, licensed under Creative Commons.
We've agreed to share the blanket at home, where it will live on the couch:

I'll be posting the pattern to Ravelry too, several commenters there have asked for it based on seeing my project photos.

Given I'm a wizard

I'm working on a Rails Web site (ShareTheVisit) and using a gem called Sorcery. I like the idea behind Sorcery, as a library that offloads the common features around picking passwords, activating accounts and so on. The Railscast on Sorcery really helped too. However, I've run into a few snags.

The major snag is the automation that Sorcery does on the object you designate for the user object. For example, if you tell Sorcery that a "User" instance is what it logs in, then the User account needs a password according to Sorcery. If you also tell Sorcery that you wish to use the email activation feature, then it does its magic every time a User object is created.

What's the problem with this? Our use case required me to create some information around a user who hadn't joined the site yet. E.g. I'm inviting my doctor to the site, and when she joins, I need her account associated with mine. For better or for worse, I decided to do this by creating a model for the invited doctor, but it couldn't be a User instance because that needs a password which we're not going to go choosing for the doctor, and even if we let the doctor choose the password later, creating the User account too early makes the email activation happy too early. So instead of creating a User instance when the doctor is invited, I created an Account model for the doctor. When the doctor comes to the site and chooses a password, then I create the User instance, associated it with the Account, and the activation email is sent by Sorcery.

So far so good, but the email activation is still a problem around invitations. If somebody comes to the site and registers ad-hoc, we want Sorcery to send an email before activating the account, so that we are certain that email will work for resetting passwords and such. But if somebody clicks on a link in an invitation they received in email, doesn't that validate the email address even more smoothly? So I'd like to skip the activation-required email depending on whether the registration form came with a token indicating the User was invited via a working email address. No luck doing this directly, Sorcery does its magic whether I'd like it to or not, when the User account is created. I hacked around it like this, setting the email initially to something bogus that will fail to be delivered, before resetting the email after the save event.

  def register(account, params)
    user = User.new(params[:user])
    user.account = account
    if ValidEmailToken.exists?(:email => account.email, :token => params[:token])
      user.email = "#{params[:token]}+temporary@example.com"  #Temporary -- make the "activation required" email fail to be sent
      if user.save
        user.activate!
        auto_login(user)
        user.email = account.email
        user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    else
      user.email = account.email
      if user.save
        AdminMailer.user_registered_notify_email(user).deliver
      end
    end
    return user
  end

Ugh. Even uglier, I had to put this in ApplicationHelper rather than in either the User or Account model as I would have liked. Sorcery only makes auto_login available to controllers (and ApplicationHelper is a mixin to controllers). I'm going to have to totally refactor this when I figure out how. I'd like to reunite the User and Account objects or at least reduce the redundancy -- because of the path by which I got this working, an Account has an email address when a new person is invited by an existing user, and a User has an email field too because that's what Sorcery needs.

Another snag came in automated functional testing. I'm using Cucumber and Capybara. I write tests like this:

 Scenario: Creating a visit with a new patient but no contact info
    Given I'm a doctor
    When I am attempting to create a new visit
    And I can add a patient
    And I click Create Visit
    Then I should see an error with "Need contact information"

To make this work I need to write the code behind each step, including "Given I'm a doctor".

 Given /^I'm a doctor$/ do
    @user = User.create!(:email => "testuser@example.com", :password => "password")
    DoctorsController.any_instance.stub(:current_user).and_return(@user)
    VisitsController.any_instance.stub(:current_user).and_return(@user)
    PatientsController.any_instance.stub(:current_user).and_return(@user)
  
    @doctor = Doctor.create!(:first_name => "Gregory", :last_name => "House", :degree =>"MD", :user => @user)
  end

This shows how I had to stub out the magic Sorcery method "current_user" in each controller, because Sorcery does something like (I'm guessing, and I'm too lazy to read the Sorcery code to confirm) add methods to each actual controller. I definitely tried stubbing out "current_user" in the ApplicationController, which each other controller extends, but that just did not work.

I hope these tips help somebody else make Sorcery work for them, or to choose something else; or that somebody can let me know if I should be doing something different. I'm coding by myself these days so when I ask myself, "Self, is this a good idea?" I don't usually get a very good answer. It's quite probable I'm overlooking some decently obvious improvements to my situation.

To fully unpack the title of this blog post, "Given I'm a wizard"? The keyword "Given" is how Cucumber, in its domain-specific language, expresses pre-requisites, as in the example above. Obviously I'm a wizard if I'm using Sorcery, plus "In the basement rolling dice... I'm a wizard!" And yes, in the D&D campaign I'm currently playing in, I'm a sorcerer.