Sunday, March 25, 2012

Scrum thinks people are stupid

One thing I both love and hate about the hardcore agile process, particularly the Scrum process, is that it tends to work around participants' biases, weaknesses, and lack of trust, rather than try to fix them. Scrum may not even leave room for improvements in trust or reliability. Some examples:

Developers' estimation biases are worked around, not fixed
Because developers have biases in estimating tasks (biases that are consistent over time and on average), scrum "velocity" measures estimates against results.  It doesn't attempt to fix estimates, e.g. by showing a developer the difference between their estimates and time spent.  In fact, velocity doesn't measure time spent at all, and lumps all developers in a team, over-estimators and under-estimators together, in one velocity measurement.

This is probably for the best -- it's one simple measure that's remarkably consistent.  Still, I wonder if it wouldn't be more useful in the long run to learn to estimate better.  I've never seen a really good estimation feedback loop in the software development context, but wouldn't it be neat to try?

Product owners' changes are either completely allowed or disallowed
Traditionally engineering teams have to train product owners not to change the product plans all the time.  This involves frequent team arguments about product plan changes.  Instead, scrum tries to carve out one small space where product changes are forbidden, and allow all other changes without argument.  In the current interval, the product owner cannot make any changes or the current plans are all tossed out and the estimation process is restarted, a consequence severe enough to effectively forbid any small changes.

This rule does keep product owners off the developers' backs.  If the product owner is thinking of a change to this week's plans, the consequences of an "abnormal sprint termination" probably stop them.  If the product owner is thinking of a change to next week's plans, the upcoming sprint planning meeting is where they'll discuss it.  Either way, the product owner does not walk up to a developer and say "Hey!  I've got a great idea!"

If keeping the product owner off the developer's back sounds like a really good idea to you, well you probably haven't worked with trusted, experienced product owners.  And if the team has processes that reify the distrust, then there's less chance to build trust.

Demos help people who can't analyse abstract plans
Weekly demos are a course correction mechanism. In order for anything to be marked done, it must be demoed.  And when a feature is demoed, the product owner can see the practical consequences better than they could when the feature was designed.  Now the product owner is able to immediately add things to the backlog, which might get done in the next iteration, and this is good.  Iterative design for the win.

The practical consequence of this appears to be less specification and planning work, which is good (avoid overplanning), but taken to the point where the product owners do not feel any particular pressure to understand and analyze the design.  Instead of sitting in front of the wireframes and thinking it through -- "What happens if I press this if it were implemented?  If this were a real system and the item had been deleted, what would the user need to see? " It's a difficult what-if skill and takes practice.   It doesn't make design any less iterative!  Instead, it moves the iterations into the design phase rather than the costly implementation phase.

Frequent demos, and the scrum rules which allow for any changes in the backlog, seem to remove some of the need to develop abstract design skills.  That makes me a little sad.  Still, frequent demos and iterations are a tool I'd use in any software development process.

Optimizing considered harmful
In hardcore scrum, developers are practically forbidden from doing any changes now that would make future work easier.  It's discouraged, and the way the system tracks tasks makes it unrewarding to do.

Let's say that today my task is to create a feature for users to delete items. There's also a story in the backlog or icebox for undeleting items (e.g. finding them in a trash folder and returning them).  Well, the way success is structured in scrum, I estimate the time to delete items at the beginning of the period where I do that work, and it makes everybody happier if I figure out how to delete items without much work, and we make more features fit in this period.  It doesn't help me now to estimate high in order to prepare for the 'undelete' feature.  It doesn't help me in the future either -- when we get to the scrum meeting where we estimate the  'undelete' feature, it might not even be me doing the feature.  (In theory, developers are supposed to be interchangeable in agile/scrum).  Even if it is me,  it's no big deal to have to do the undelete by rewriting the way delete worked -- I just build that into my estimates for this period.

There's no overall project schedule view that would have showed the value for doing these two features together and doing delete right the first time.  There are other ways where optimizing is actively discouraged:

  • Literally, optimizing speed or resource use is discouraged.  Functional stories only say that the feature has to work.  Other stories in the future might or might not say that the feature has to work in 2 seconds.  Security might also be an afterthought!
  • Optimizing by expertise is discouraged.  Everybody in the team is supposed to be interchangeable.  If all of this week's features are GUI features, then everybody does GUI work. If all of the next week's features are security features, then everybody does security work. 
  • Optimizing the sequence of work by engineering constraints is discouraged.  If it would be faster to do feature A after the more general feature B is completed, too bad.  If there's a pileup of work with dependencies, and it's slipping to the end of the backlog where the dependencies will start to slow down each task, too bad.  Only the priority sorting of the product owner is valid.
I've seen passionate arguments for "You Aren't Gonna Need It" (YAGNI), and they're right.  Engineers often optimize prematurely.  Engineers often predict what they think the user will need and can be wrong.   But none of the passionate arguments for YAGNI think that it can be used without a sense of balance, right?  So when scrum methodology encourages developers to always put off optional work, scrum puts its thumb on the balance.  It short-circuits the discussions of complex tradeoffs and simply says do it later. 

What's common among all these traits is an attitude in scrum not only that people are fallible, but they're routinely wrong, and that they can't be trusted to work together on complex issues for the greater good. That means that it works best in environments where that attitude is more true -- e.g. a contract development team, comprised of decent engineers but not particularly specialized, who don't know much about the end use and maybe don't care too much, working with product owners who aren't experienced in software design, are way too busy to write specs and discuss complex tradeoffs, and are always tempted to change things they shouldn't.  

Agile should be done differently when the engineers are specialized, care about what they're doing and what it will be used for, work closely with the product owners in the same business unit, and the product owners are smart and can be trusted to learn good software development practices.  Just how it should be done differently is an open question to me. There are agile process variants that take different attitudes (e.g. those that encourage planning a whole release) and for the kind of team I am currently working in and enjoy working in, I'm interested in those variants.


arcwhite said...

I think many of the symptoms you describe come down to a lack of good communication between developers and the product owner/scrum master type.

In my current environment, we have a very good product manager, and it's the role of the developers to communicate situations like your 'delete/undelete' task pairing to him. He takes that information on board and schedules tasks appropriately - and if we can't get it all done in one sprint, we at least adjust the estimate (or add a new task) to pave the way for the next step.

Scrum works when the process is strict, but nobody says you have to be a mindless automaton about the format of the tasks or where they fit together - remember, Agile favours interpersonal communication over process and procedure!

Johnno said...
This comment has been removed by the author.
Johnno said...

I think its more a question of "Scrum thinks people are people" ;-)

The role of the product manager/owner is very important and scrum won't mitigate having an idiot in that position or having stupid devs.

I don't believe that scrum tries to dumb things down but rather encourage separation of work/concerns into small enough chunks where both activity and expectations are manageable and adaptable.

There's no rule that says developers have to forget about what may be coming up in the future and take a very blinkered view over just the current iteration and tasks within them. Often one iteration acts as thinking space for work in the backlog and if it may be related to work in the current iteration or could change how that would be implemented then that can be discussed when tasking out a story.

If everybody wants to do a good job then they'll talk to each other and come up with well written stories for each iteration.

It would be certainly possible to follow the scrum process to the letter and produce a pile of trash. For example, a developer could follow story cards with flaws in them blindly, and not discuss engineering concerns. I'd see that as somewhat passive aggressive and not a flaw of scrum itself.

Scrum is really a skeleton to attach the real meat of whats going on each iteration onto and its up to people from that point onwards.

Anonymous said...

Ah ... an idealist ... how touching!

Edwin Martin said...

People *are* stupid. Software development is an profession since the seventies and books are written since then how to do this right. And still, after fourthy years, people don't seem to have learned anything from the mistakes from the past and still make the same mistakes.

Scrum is just a tool to limit these mistakes.

Anonymous said...

You guys will love this Agile parody song!

- John Quimbly

Jordan said...

I have a number of posts discussing the shortcomings of agile/scrum and some alternatives on my blog that you and your readers might find interesting.

There are a lot of things wrong with Scrum and it's becoming more and more obvious that the solution to too much long term thinking is not merely too much short term thinking (scrum).

Scrum combines the worst of short term thinking with an essentially autocratic PO -- certainly a fuse and a match -- whether it blows up or not is blamed on the users tho.


Thorbjorn Sigberg said...

Interesting post. I think you are pushing the disadvantages of scrum to the extreme here, but you have some interesting points.

Personally I find scrum to strict as a process. One usually say that you either do scrum or you don't, there's no such thing as a "scrumish" or "scrum-like" process. If that is I case, I prefer to not do scrum.

I much prefer lean / kanban that have a few, select rules to optimize the processes you have, but doesn't dictate the process itself.

Olly Oechsle said...

Thanks for an interesting and provocative article!

I guess it just reinforces the need to be flexible and do the people over processes thing.

In our scrum team we are measured by the features we deliver, but each one does have to meet a number of non-functional requirements, such as performance, memory use, caching, testability etc. These are reflected in the points we give for each task.

There is definitely something to be said about the batched nature of sprints, where stories can only be taken in and planned at one point. For this reason some teams in our organisation have switched to Kanban.

Unknown said...

I could only manage to read your first item and I think you're showing a lack of experience on software development.

May I suggest you read the article by Donald Knuth "Computer programming as an art" and also "Large limits to Software Estimation" by J. Lewis? It might bring you some better understanding why your wish for better estimation is just that, a wish.

Lisa said...

I would love to try out Kanban. I've heard good things. Someday I may read up on it, or if I end up working with somebody who knows and loves it I will jump on the chance to try it.

saneman said...

I think one of the comments that "people are stupid" indicates very clearly who Agile is good for. Companies with average or stupid developers. Or developers with bad organizational skills.

jeniffer said...

Nice post! and well explained

Sam said...

I agree with what has been disscussed here.Keep updating us!

Ella Mapple said...

I really doubt whether its scrum falling short or adapting scrum the wrong way. A suggestion would be to invest on professional certifications. We use Scrum org wide for project management, and its working well compared to waterfall method. As part of company sponsored training, I also got my agile scrum certification; Scrum Master Certification recently.

Lisa said...

Oh Scrum totally works well compared to the waterfall method. That's why I only use agile methods for the last six or more years!

I just wonder if certification would help, or if it would reinforce the almost-religious "There is one right way to do SCRUM" attitude that does tend to oversimplify and underadapt.

Unknown said...

I reached this page when I searched "software estimation is idiotic"

How do you estimate spikes? How difficult it is for managers to understand.

Agile will always fail when run by the same old people who ran waterfall.

Blog Archive

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.