You must be this tall to ride the Elastic Beanstalk

Elastic Beanstalk seems like it’s meant to allow a startup to easily deploy Web sites using the most common Web frameworks, and scale those sites gracefully with an integrated AWS scaling group.  I’m judging it by that supposed purpose, and comparing it to Heroku.

Here's what we discovered using it in a startup. 

1.  Elastic Beanstalk deploys are inherently risky - delete working copy then cross your fingers

EB documentation instructs users to push new code to an existing environment in order to deploy a new version.  However, EB takes the site down in order to do this.  If something goes wrong with the new version, the old version is gone.   It should be possible to return to the previous version on the same beanstalk but it can get in a state where the environment is corrupt. Experienced users advise always creating a new environment, test it then redirect URLs, then throw the old environment out. 

Compare to Heroku, where if the deploy fails, Heroku stays with the old working site, which is not replaced until the new site works.  I also never experienced a failure to revert to the older version. 

2.  EB requires more manual or automation labour 

Because of EB’s risky deploys, best practice is commonly to create a new environment for each deploy, and test that environment before swapping the URLs with the running environment.  This is fine, but as soon as one has a handful of environment variables and custom settings like VPC membership, the creation of that new environment needs to be automated in scripts and config files.

There are still risks.  If one environment is not working, AWS refuses to swap the URLs!  Creating a new environment reduces the chance that production would be pointing to a broken environment, but doesn’t completely eliminate it.  If that happens, one has to use DNS to redirect the public URL to the working environment — because even if one deletes the broken environment, AWS doesn’t permit changing the URL of the working environment!  

Even if a startup does all this automation, compare to Heroku, where neither manual environment swapping nor custom automation are required. It's a good engineering practice to still automate source control checkout, tagging and deploy, but the deploy itself would be one line of the script.

3.  AWS support is not what it should be

About half the time, AWS support was useless or worse.  

• A common example of when it's useless is in asynchronous support, when the support representative asks repeatedly for information that was already given earlier in the conversation, or to try things which one has already tried (and mentioned).  This happened to me several times.
• An example of when it's worse than useless is when the support engineer suggests disruptive measures which do not help.  Once when our production site was unreachable because we'd misconfigured VPC membership, the support engineer didn't help figure this out.  Instead he had me repeatedly restart the environment in single instance mode then back to cluster mode; then restart the environment with a different machine image (AMI). It took hours and hours. He seemed knowledgeable but the things he knew were not the problem and he didn't seem to know how to confirm or rule out his hypotheses. 

My experience with Heroku is that I need support less than 10% as often as I did with AWS EB. 

4.  AWS documentation is not what it should be

If the service is complicated and doesn't always work the way one expects, then the documentation needs to be excellent.  In the AWS documentation is that there's a big gap where the mid-level documentation should be.  There is high-level "Here are the awesome features" documentation and low-level stuff like field names and config file formats. The missing parts are "Here's a mental model so you can understand what's happening" as well as "Here's an approach that we know works well for other users in this situation".  

Without any documentation to supply good mental models of what's going on, or good practices for various common situations, it's really easy for inexperienced people to construct a setup that has unexpected drawbacks.  

Summary: Disappointed.  

While EB is a definite improvement over running machines manually or on a hosted service at the level of EC2, I am disappointed that it is not as easy to use or as well-supported as Heroku.  I have heard that with experienced, dedicated operations engineers, it's great -- but I also know of companies with experienced, dedicated operations engineers who run their own EC2 instances and manage scaling with auto-scaling groups, rather than let Elastic Beanstalk handle it for them.

I've heard the argument that AWS is cheaper than Heroku, but at a small scale, employing experienced AWS operations engineers is much more expensive.  So is disappointing users and gaining AWS experience the hard way.  If you must be on AWS eventually, do so when you're "tall enough" to really need the scale and hire the resources -- migrating between Heroku and EB itself is not that hard.