Modern Web architecture, cookie-cutter, view from the trenches

Web architecture has come together quite nicely of late.  The components work together better than ever before, and it's easier to put together a great Web site or service -- putting more emphasis on the design and user experience, of course, which remain important and hard.

This list of steps is an overview of the cookie cutter approach.  It works for most new Web sites and services.  Many people I've worked with believe they have something different, a special wonderful unique thing that is not right for the cookie cutter approach, and to they extent that they're right, their differences should be minimized (focus on making your value differentiated, not your architecture).  So, if you're getting started on a new site or even a new service within a service, try to put off being special and different as long and as far as you can.  To minimize early development costs while remaining flexible and scalable (new hires as well as increased usage), try to make the default choice for every step below until it's proven you need to be different.

I came up with this list by using these components myself in situations both early and at scale, by talking with lots of other developers about what actually works and is easy, and by keeping abreast of developments by following a few blogs and twitterers I admire.

1. Pick a modern Web framework

The popular favourites are Python and Django, and Ruby on Rails.  If you are or have a technical person already, personal preference is a fine way to make this choice.  These both have data abstractions, views, templates, lots of inherent flexibility, and a rich set of free extensions (e.g. Ruby 'gems').  If you're thinking about node.js, see "Qualms" below.


2. Pick a data store

Hosted MySql or PostgreSQL both work great for a long time and a lot of scaling.  Use hosted databases for online test, staging and production and either the same database or SQLite for development and local testing.  It would be fine to just go with whatever is cheapest and easiest, which might mean making this decision together with "pick a hosting service" (E.g. Heroku uses PostgreSQL by default).  NoSQL solutions go under "Qualms" below.


3. Pick a hosting service

Look.  Just don't run servers yourself, OK?  Not until you're the size where your hosting costs annually are approaching the annual cost (roughly twice salary) of a dev/ops employee.  Don't use EC2 either, that's still a lot of manual work.  Startup efforts don't have the manpower or processes to handle an early new customer and a vulnerability disclosure somewhere in the Web stack, in the same week.  So leave the operation and upkeep of the Web stack up to the hosting service. I like Heroku.


4. Design and implement landing page

This is where you'll need to pick a CSS layout library like Skeleton.  Use this library's columns and rows to lay out your pages.


5. Plug in a user login module if you need login

Either pick one that supports passwords, email verification, forgetting passwords etc, or pick one that supports using Facebook, Twitter, Google, etc identities.  Plugging this in should be very quick if you're still on the cookie-cutter path.  Bootstrappers can configure their service to send those validation and forgotten-password emails via a free GMail account initially.  Django comes with authentication, whereas for Rails I've used both AuthLogic and Sorcery.


6. Add JQuery to your Web site if you didn't already do it for the login feature

A combination of other Javascript libraries could be used to replace JQuery, but JQuery does both common widgets and functions (like handsome buttons) and AJAX -- the AJAX part is necessary for accessing data from your service in the next two steps.


7. Start with some kind of site template 


You probably need Web page headers, footers, and common colours and fonts.  A combination of Web framework page templates, including partial pages (e.g. the Web page footer is commonly a partial Web page that is included on every full page in your site) and CSS templates solve this.


8. Work out your core data model, using your Web framework and migrations

Your core data model is that special wonderful thing you're building.  It could be real estate listings, restaurant reviews, task lists or BBQ recipes.  Frame it as objects and collections.


9. Expose your data as a Web API with JSON, and show in the Web page with AJAX. 

Read up on how your chosen Web framework supports REST and apply it to your data model.  Read up on how your chosen Web framework supports AJAX.  Often the way to expose the data model is a direct extension of your data model. However, sometimes it's driven more by the view.  If the UI always shows collections of comments, you don't need a way to get just a single comment; you jump straight to "GET all the comments for this item."  Use JQuery or another Javascript library to make those calls and load data dynamically into pages.

This step and the previous step will happen over and over again as you add functionality.

Additional steps

• While in development, use free email; soon move to a hosted email provider.

To begin with use your Web framework's default email manager (ActionMailer in Rails), and just use any IMAP account to send mail, such as a free GMail account.  This is one of the things you'll need to upgrade sooner rather than later, by moving to a paid, hosted email provider.  Sendgrid is working OK for me and took less than half a day to sign up, set up and switch all test/demo/production servers to.  Sending SMS is similar -- in addition to choosing a SMS service you'll need to plug in a module.


• When or if you need native mobile or 3rd party app support, firm up your Web API

It's pretty cheap to get started with mobile functionality via HTTP/HTML/AJAX, at least for prototype, demo or even MVP.  There are probably cases where a native app is needed right away, but that's not too common.   The mobile client can use the same Web API that the javascript "client" uses, only it now becomes more important for that Web API to be stable, because a native mobile app or 3rd party app won't be updated at the same time as the Web site and API are updated.


• When or if you need to do any authentication with other sites or services, use OAuth.

OAuth comes into play for two situations: to authenticate users to another site such as Twitter, but also to authenticate apps or other sites that get special permissions on your service.  Rather than cook up your own special key logic, just use OAuth, plugging in one of the libraries that already exist.


• While you move along, let your Web framework help you as much as possible.

Django and Rails both have a ton of useful stuff.  The biggest one in my view is their database migrations assistance.  Bundler, in Rails, is key to looking after all the gems you'll have after the above steps. Try to also be familiar with the scripts and scaffolds that allow one to build new functionality fast in the completely cookie-cutter format. 

• While you move along, start to add automated tests

I add these right at the beginning.  Selenium, and capybara let me automatically load Web pages from my site, fill in forms, press buttons, and do simple tests on the results. RSpec and its mocking features are critical both for unit tests and for mocking out dependencies.  I run these myself and also, as soon as there are more than 1 developer, on a continuous integration server.  I'm currently trying out Semaphore as my first hosted integration server but would still go to the trouble of running Jenkins on EC2 if Semaphore doesn't work out.

Qualms

Node.js has great promise to unify Web programming under one language, bridging the divide between Web frontend javascript programmers and Web backend programmers using Ruby, Python or other.  A lot of people I respect  for being simultaneously practical, realistic and visionary, do like Node.js.  My qualm is that there is no standard framework to use with node.js -- the level of services provided by Node.js is more primitive than that of Django or Rails, and thus additional components are needed.

NoSQL is very powerful and often appropriate in large-scale Web sites and services.  I've used it before.  However, I have qualms suggesting a NoSQL solution at the outset, because the tools to hook frameworks and NoSQL together, as well as the tools to manage NoSQL systems, are not the default tools most familiar to Web developers, and not the most tested and evolved.  That said, a startup with experience using NoSQL might, without my labeling them crazy, use hosted Mongo, Redis or CouchDB, provided they still abstract away the store from the Web framework's data model.  In the long run this is something a hugely successful system probably needs to scale.   A combination of SQL and NoSQL is especially powerful for large or complex services but if you're at the "pick a data store" step you're not there yet.

Summary

The subject matter or domain used to matter a lot when a startup began to build a Web site or service.  These days, it doesn't seem to matter as much -- most startups I've seen can benefit from following most of these steps.  It's a real benefit to keep the startup's special thing limited and cordoned off (think of it as making your specialness more concentrated).  It's easier to hire and onboard people, as well as to benefit from community advances, when the overall Web architecture is cookie cutter.

Girls in games

The cultural gestalt these days seems to include more talk about women and video-games than ever before.  Suddenly, women are 47% of video-gamers and game designers (and game advertisers) are still figuring out how to deal with that.  If you haven't seen any of Anita Sarkeesian's videos before, they're really good.  Her concepts help me understand a bunch of common patterns and gave me names to more easily identify them (the "smurfette effect" for example).

Anita's concepts have not ruined my enjoyment of video games, but I may be getting more selective.  We have a PS2 and have been playing some old games.  One of them is Okami which is an outstanding game in many ways: playable, artistic, incredibly rich and creative.  In the game, the player's character is a wolf who begins a quest to regain divine powers the character once had, and rid the world of demons and darkness so it can bloom and the sun can shine again. When the world blooms it's simply gorgeous; all the art is painterly and beautiful.

Although the wolf fights, and is the best fighter in the world, its gender is ambiguous. My son thought the wolf was a 'he', missing the repeated interactions when the wolf meets other minor gods and they always call the wolf "Amaterasu, mother of us all".  Throughout the world there are reasonable ratios of male and female characters, adult and child and often the female is more powerful than the corresponding male (in the main city, the empress is clearly more important than the emperor).  Both male and female characters are killed off, not just the females.  Females need rescuing more than males but not exclusively.  So there's more balance than normal.

A few things grate only a tiny bit; the main non-player character who fights with a sword is alcoholic, lazy, scared and male, and goes off to save his brave, supportive lady love.  The character who seems good but is secretly taken over by a demon is called "busty babe" over and over by the pixie (and she is pictured as having unreal boobs), and there are scenes about this 2-inch tall male pixie trying to literally get into her shirt.  All in all, did not make me love the game any less -- the overall balance is so pleasing that an annoyingly-stereotyped interaction could be enjoyed as mild low humour.

And then there's this kind of thing.  *sigh*  I thought the Internets were supposed to know all about me by now and target ads right at me.  Clearly the Internets know I'm a gamer, but can't they tell I'm female?