Dear applicant

I wrote up a series of overly honest responses to job applicants a few years ago.  Didn't send them, of course, it was purely therapeutic writing.  Now that the job posting itself is stale and I don't even remember which company I was at when I wrote this, it's safe to post...

Dear applicant with broken links,

Thanks for sending your resume in regards to our open position.  Unfortunately, we're looking for the experience we asked for in the job description.  I appreciated the personalized cover letter but it would have been even nicer if it weren't formatted so oddly in HTML.

Dear applicant with 20 jobs in the last four years,

Thank-you for your interest in our position posted on craigslist.  At least I think you're interested, but since you just cut and pasted your resume into email and didn't reformat it, nor did you add any personalized cover letter, I'm not sure I really feel the interest.  You might like to know that your portfolio site has nothing on it but three static pages with a purple background.  I'm sure that was an oversight?

Dear new-grad applicant,

Thank-you for your interest in our position.  I feel at this time we're looking for somebody who has the experience they claim -- your "ten years experience" summarized in your cover letter looks more like four small short-term, part-time contract jobs in the last four years when I look at your resume.   Also, anybody who feels "proficiency with Microsoft Office Suite" is a skill worth listing for a technical position has too little of substance to say about their skills.

I was intrigued to see that your portfolio site looks like it came from the nineties.  Is Web 2.0 passé now, and retro static sites are the new white canvas?  I find it fascinating that you recreated that limited, static feel with CSS and Javascript!

Streets are safer than ever

I recently looked up some data in response to a sad person thinking that "our streets" (the writer was writing from New York, but the perception is widespread) are less safe for kids than they were in her time. As long as I was doing the research I thought I should post it here too.  TL;DR: The streets are safer than ever.

There’s more relative deaths due to accident and homicide than due to illness, but that’s because childhood deaths due to illnesses have plummeted. So that means a parent was right to be much more worried about kids dying from measles than guns in the early last century. But the risk is down for all causes, so today a parent ought to be much less worried overall, AND less worried about each individual cause.

“For children older than 1 year of age, the overall decline in mortality during the 20th century has been spectacular. In 1900, more than 3 in 100 children died between their first and 20th birthday; today, less than 2 in 1000 die. At the beginning of the 20th century, the leading causes of child mortality were infectious diseases, including diarrheal diseases, diphtheria, measles, pneumonia and influenza, scarlet fever, tuberculosis, typhoid and paratyphoid fevers, and whooping cough. Between 1900 and 1998, the percentage of child deaths attributable to infectious diseases declined from 61.6% to 2%. Accidents accounted for 6.3% of child deaths in 1900, but 43.9% in 1998. Between 1900 and 1998, the death rate from accidents, now usually called unintentional injuries, declined two-thirds, from 47.5 to 15.9 deaths per 100 000.” from Pediatrics journal

The CDC data shows that death by accidental injury is several times higher than death by homicide, for ALL age groups, even for 15-24 year olds.  Accidental injury is mostly vehicular, so I tend to ask people if they drive on the freeway if they are worried about their kids playing outside or going trick-or-treating.

And in case one is worried about city streets, the drop in risk can't be attributed only to the suburbs and the country. Living in the city is now less dangerous. This article is about town vs country but also talks about overall safety in cities going way up.  I looked up data for New York county, the densest county in NY State, compared to other NY counties using CDC Wonder, and found that New York county had 22 deaths per year per 100,000 between the ages of 1 and 19. That's much closer to the lowest county, Westchester with 17 per 100,000, than to the highest, Sullivan county with 38.6 per 100,000. 

How I hire engineers for startups

I went looking for articles on how to interview programmers/engineers for startups.  I didn't like much of what I found (I did like Elad Blog's take), and none of them addressed engineering skills and new-technology choices the way I wanted, so wrote my own guide (and as usual, wrote too much).  My thanks to Mark Ferlatte and Rachel Gollub for reviewing a first draft of this. Mistakes remain mine.

My thesis is that a startup needs coders who are good engineers too.  A startup that hires coders who are bad team players and engineers will quickly dig itself into a hole of technical debt.  A startup is in a worse position than most other employers to teach engineering skills like technical teamwork, using specifications and unit testing; also it is in no position to spend 5 times as much time and money fixing bad code.

1.  How do you use specifications in your programming?  Please expand...

Look for mid-range answers here.  “Never” or "Specs are a waste of time" is bad — using specifications, wireframes or user stories is an important skill.  An engineer who doesn’t know how to review and use a specification will at best put their own assumptions and that specification's errors into code, which will turn up as bugs later. 

“I always need specs” may also be bad for a startup, unless the engineer takes some responsibility e.g. “If a spec doesn’t exist, I write down what I’m going to do and send it to the team."  

A good engineer should also be able to talk about reviewing specs and improving them, and possibly even writing them or teaching product people how to write them.  Look for information behind the quick answer, because companies and experiences vary hugely. A developer who can talk intelligently about how rapid iteration reduces need for elaborate specs sounds decent to me.

2.  How much do you rely on unit testing?   When and how do you write unit tests? 

Here the right point on the scale is at least 80% of the way towards “always”.  Unit testing is an important skill and art and another one you don't have time to teach.

Excuses are not that impressive here.  If a candidate said their past companies didn’t do unit testing I would wonder why the candidate didn’t learn and even promote unit testing on their own.

Good answers: “I rely on unit testing to help me think through a problem and I write unit tests sometimes before I write the function, but sometimes after, depending."

3.  When do you decide to use a new technology? Give an example of learning a new technology, promoting it in your team, and executing it. 

Look for mid-range answers here too.  Candidates should not be on the bitter bleeding edge of technology.  There needs to be a drastic advantage to taking on the risk of brand-new technology.  Ask how they justified picking up something untested, and how they assessed and minimized the risk.  See my addendum below for examples of bleeding edge vs behind the curve.

On the other hand, candidates should be able to point to some new-ish technology (not just new to the candidate), year after year, that they learned and adopted.   Try to find out if a candidate eagerly learned the technology or had to.  Did the candidate choose and promote, or follow teammates’ beaten paths?

4.  What’s important to you in your work/employment?  

Most answers are probably not critical hire/no hire factors here.  However, you need to know what the candidate needs, in case 1 week later you’re saying “How do we convince Lisa to join us now” or in 6 months asking “Lisa is really awesome, how do we make sure we keep her?”.  You must treat tech candidates' needs respectfully.  You need them more than they need you. 

That said, there are some bad answers for startups: “Working with rock stars” (or ninjas).  “Stability”.   “A mentor” (or a hands-on education in engineering).  

Good answers: “Learning” (if self-directed).  “taking on a variety of roles”.  “Being very responsible for something”.  “Moving fast” (if under control via unit tests, etc).   “Being involved at the level of the business model.”   “Having some non-trivial ownership in a venture. “  “Gain the experience to do my own startup.”  “Be in a position to develop a great engineering culture." "Working with a good team." 

5.  Can we see some samples of your code?  (Alternatives: solve problem code on the whiteboard or join for a trial day coding on the project)

For this part, you need two or more really good startup engineers that are so good you CAN’T hire them to advise you.  Have them do in-person or phone interviews or read source code to evaluate candidates.  

If the candidate can’t point to some open source contributions, it may be possible to see some previous code under NDA; or the developer may have done some tutorial coding to learn some new system.   Looking at source code is an effective use of time both for the candidate and for your startup.  If this is not possible for the candidate, then a whiteboard or “homework” coding session is needed.  

Another option is to have them work on a project for a day or two.  Sometimes this can work and sometimes it can't.  A programmer who can't or won't dedicate full days of programming to prove themselves to a startup (see above about you need her more than she needs you) may still be the right person to hire.

What else to take notes on

While asking these questions, find out if the candidate is nice and easy to get along with. Startups are stressful enough already. Working with arrogant people makes startups worse.  You need good communicators.  Techies who assert and tell what to do may sound knowledgeable initially so dig deeper -- they need to be able to explain in detail how and why, too.

Try to find out if the candidate learns fast, both by asking about learning new technology, and ideally by asking making suggestions in the technical part of the interview (or after reviewing their code) that require the candidate to listen to another way of doing something and figure out how and why to do it that way. 

The "new technologies" question should help you also answer: is a candidate narrow or broad, a generalist or a specialist. Rather than just hiring C++ engineers, startups need people who may write C++ and can deploy EC2 servers, do metrics in Python and debug some Javascript.

Try to find out if the candidate can handle uncertainty and change-- the spec question is a good time to address that as a side concern. 

Addendum: Examples of bleeding edge, cutting edge and behind the curve

Bleeding edge: Almost no startup in 2009-2010 needed to or could afford to build their entire codebase on Node.js that year.  Don't hire the person who advocates this or something equivalent, especially if they haven't done it themselves.  (Not their team.)

Cutting edge: If the candidate learned Node.js in 2010, however, and decided that 2012 was the year to make it the basis of a product, and had a plan for hiring other engineers that know or want to know Node.js, that is a good sign. 

Behind the curve: They should not still be promoting Java servlets for brand-new code-bases (at least not without Hibernate and Spring, and only if Java is already needed for the startup).  Somebody whose only new technology learned in the last 5 years was Java servlets, because their manager assigned it to them, is not adopting new software technology at an appropriate pace or time.

You notice that my examples are older or heavily hedged.  This is a really tricky area.  I disagree with people I deeply respect about whether certain examples are a bad sign, a good sign, or "it depends" -- though  I suspect if we talked about a specific candidate and what that candidate had said about this question we'd be able to come to a conclusion about the candidate, even if not about Java Servlets.  Take really great notes so you can talk about the nuances with at least 2 of your tech advisors.

You must be this tall to ride the Elastic Beanstalk

Elastic Beanstalk seems like it’s meant to allow a startup to easily deploy Web sites using the most common Web frameworks, and scale those sites gracefully with an integrated AWS scaling group.  I’m judging it by that supposed purpose, and comparing it to Heroku.

Here's what we discovered using it in a startup. 

1.  Elastic Beanstalk deploys are inherently risky - delete working copy then cross your fingers

EB documentation instructs users to push new code to an existing environment in order to deploy a new version.  However, EB takes the site down in order to do this.  If something goes wrong with the new version, the old version is gone.   It should be possible to return to the previous version on the same beanstalk but it can get in a state where the environment is corrupt. Experienced users advise always creating a new environment, test it then redirect URLs, then throw the old environment out. 

Compare to Heroku, where if the deploy fails, Heroku stays with the old working site, which is not replaced until the new site works.  I also never experienced a failure to revert to the older version. 

2.  EB requires more manual or automation labour 

Because of EB’s risky deploys, best practice is commonly to create a new environment for each deploy, and test that environment before swapping the URLs with the running environment.  This is fine, but as soon as one has a handful of environment variables and custom settings like VPC membership, the creation of that new environment needs to be automated in scripts and config files.

There are still risks.  If one environment is not working, AWS refuses to swap the URLs!  Creating a new environment reduces the chance that production would be pointing to a broken environment, but doesn’t completely eliminate it.  If that happens, one has to use DNS to redirect the public URL to the working environment — because even if one deletes the broken environment, AWS doesn’t permit changing the URL of the working environment!  

Even if a startup does all this automation, compare to Heroku, where neither manual environment swapping nor custom automation are required. It's a good engineering practice to still automate source control checkout, tagging and deploy, but the deploy itself would be one line of the script.

3.  AWS support is not what it should be

About half the time, AWS support was useless or worse.  

• A common example of when it's useless is in asynchronous support, when the support representative asks repeatedly for information that was already given earlier in the conversation, or to try things which one has already tried (and mentioned).  This happened to me several times.
• An example of when it's worse than useless is when the support engineer suggests disruptive measures which do not help.  Once when our production site was unreachable because we'd misconfigured VPC membership, the support engineer didn't help figure this out.  Instead he had me repeatedly restart the environment in single instance mode then back to cluster mode; then restart the environment with a different machine image (AMI). It took hours and hours. He seemed knowledgeable but the things he knew were not the problem and he didn't seem to know how to confirm or rule out his hypotheses. 

My experience with Heroku is that I need support less than 10% as often as I did with AWS EB. 

4.  AWS documentation is not what it should be

If the service is complicated and doesn't always work the way one expects, then the documentation needs to be excellent.  In the AWS documentation is that there's a big gap where the mid-level documentation should be.  There is high-level "Here are the awesome features" documentation and low-level stuff like field names and config file formats. The missing parts are "Here's a mental model so you can understand what's happening" as well as "Here's an approach that we know works well for other users in this situation".  

Without any documentation to supply good mental models of what's going on, or good practices for various common situations, it's really easy for inexperienced people to construct a setup that has unexpected drawbacks.  

Summary: Disappointed.  

While EB is a definite improvement over running machines manually or on a hosted service at the level of EC2, I am disappointed that it is not as easy to use or as well-supported as Heroku.  I have heard that with experienced, dedicated operations engineers, it's great -- but I also know of companies with experienced, dedicated operations engineers who run their own EC2 instances and manage scaling with auto-scaling groups, rather than let Elastic Beanstalk handle it for them.

I've heard the argument that AWS is cheaper than Heroku, but at a small scale, employing experienced AWS operations engineers is much more expensive.  So is disappointing users and gaining AWS experience the hard way.  If you must be on AWS eventually, do so when you're "tall enough" to really need the scale and hire the resources -- migrating between Heroku and EB itself is not that hard.