Not happy with Agile, but why?

We were used to getting shit done … then they told us about Agile.

- Hadi Hariri, NDC 2012

 

I stated that something went wrong with Agile. This probably boils down to how Agile is applied in the Enterprise. And I am not the only one stating this.

But what went wrong and why? Criticising and pointing out problems is easy. But finding the root cause for being unhappy requires more reflection.

So what does Agile really mean?

We all know the Manifesto for Agile Software Development. But that are some high-level principles that were formulated more than a decade ago.

Software development looked radically different at that time. In the meantime we have gathered about a decade of experience in Agile Software Development and the software development landscape has changed quite a lot. If you believe some studies then Agile is now mainstream and the most widely applied methodology in software development.

Of course the values from the Agile Manifesto are still valid today. They still are the base for an agile mindset. I guess they are commonly accepted today in the industry and hardly anybody will directly disagree with them.

But it became clear that the Agile Manifesto is not enough. Even if we agree with it in principle, do we reflect it in our every-day actions?

Some people tried to go further with more manifestos:

• The Manifesto for Software Craftmanship
• The Software Team Leader Manifesto

However it became obvious that the manifestos and the reality in the trenches of the enterprise were not congruent:

• The Manifesto of "Programming, Motherfucker"
• Manifesto for Half-Arsed Agile Software Development

Today we have at least 12 Agile Methods. But are they any relevant for us as developers?

A lot of times when I speak with other developers, then they say that "Agile" is actually just common sense. A lot of developers claim to have been "doing stuff the agile way" before it was even called "Agile".

So, what does it mean to "do stuff the agile way"?

My favourite interpretation is what Jason Yip calls "Agile as a Doctrine" in his post: What do you mean when you say "Agile"?:

1. Reduce the distance between problems and problem-solvers
2. Validate every step
3. Take smaller steps
4. Clean up as you go

That is to the point what we developers do when we do Test-Driven-Development. It even does not have to be "pure" TDD: It's the Mindset not the Tool! 
It's also reflected in the basic methods engineers or scientists learn to practice (build up knowledge, take small steps, validate expectations).

Further more it's what most of us practice in some form to manage our personal live. Implicitly with "common sense" or more explicitly with GTD or Personal Kanban: we know that we have a limited amount of time and we (more or less successfully) optimise to get the most value out of it. We do not optimise for productivity, we optimise for value! We can do that, because its our lives.

If we as developers already do "stuff the agile way", where is then the problem in the current state of software development?

Here it comes: The software development industry today is not about the craft of writing software, it is mostly about management! At least in the environments where most of the software developers work.

And that is where Agile is currently failing. Despite the fact that the principle of Plan-Do-Check-Act is taught in most basic management courses, in the enterprise environments I have been involved, things are not done the "agile way":

• Instead of reducing the distance between problems and problem-solvers we introduce more layers of indirection with several layers of "architects", "product owners" and "customer proxies". We add several layers of processes for roadmaps, plannings and priorizations. We introduce portfolio-, product- and release-management and build up several layers of backlogs. (example)
• Instead of validating small steps we focus on long-term roadmaps. We increase our effort in crafting complete backlogs. We believe that tracking activities on a micro level will somehow lead us to success.
• We have a hard time focusing on value, therefore we keep ourselves busy with optimising for productivity.

For developers this environment is especially frustrating, since "doing stuff the agile way" works on the technical level and also on the personal level. It just doesn't work on the management level in typical enterprise environments.

Lately there are fresh movements that try to address the chasm between Agile development and Agile management:

• Impact Mapping
• The Lean Startup Movement
• The NoEstimates Movement

Those movements try to address shortcomings in the current adoption of Agile in the Enterprise. 

But I remain sceptical. Of course we can still go a lot further in adopting Agile on all levels. But I have the feeling that Agile just does not scale well on big projects or on big organisations. The bigger a project/organisation the bigger the need for structure and formalism. And structure and formalism prevents "doing stuff the agile way".

So how can we developers deal with that?

I think as developers we have almost no chance to influence the Agile adoption in the typical enterprise environment: Since it has to happen on management level, once you would be in a position to really have an influence, you are not a developer any more.
But as developers we can choose who we work for. This is especially true with the current shortage of developers in the industry. I am confident that there are environments out there where software development can happen in a more agile way on all levels. I even think that because of current trends in the industry these opportunities will increase in the future. More and more small organisations prove that there is less and less need for big corporations in the IT industry. 

Of course big corporations have their advantages. Security and stability are often among them. So every developer has to decide for himself if "doing stuff the agile way" is really worth it in comparison.

Finally, how do we developers find those jobs? I think it's by really looking at the management of a company. What does management really do? What are the management processes?
Github Inc. is the current poster-child of an Agile company. In a talk about their model of distributed management they summarise my above rantings with this venn diagram:

Paradise Lost: SAFe banned from the Agile Heaven?

The Scaled Agile Framework (SAFe) recently seems to have fallen from grace in the agile community.

Most of us noticed that something went wrong with Agile. The Scaled Agile Framework now gets stigmatized into the role of the traitor of the True Agile Mindset™, it serves as the negative showcase how the original Agile ideas are corrupted by The Enterprise.

I almost get the impression that nowadays every Agile guru has to pick on SAFe, to prove that he stays truly Agile:

 

• Ken Schwaber (one of the two founders of Scrum): unSAFe at any speed

The boys from RUP (Rational Unified Process) are back […] They would be at a waterfall conference, but they are no longer.

A core premise of agile is that the people doing the work are the people who can best figure out how to do it. The job of management is to do anything to help them do so, not suffocate them with SAFe.

 

• David Anderson (Pioneer of Kanban in IT): Kanban - the anti-SAFe for almost a decade already 

This seems awfully similar to Rational Unified Process (RUP) from the 1990s but updated to include Agile practices from the 2000s

 [The SAFe] approach is the antithesis of the Kanban Method!

 

• The IT risk manager: Two Legs Good

I did not encounter a single person who had successfully implemented SAFE.

Someone just shot the Agile brand in the back of the head.

 

• Scrumorakel.de: A critical View on SAFe

This means that we are back to the good old days where there was a chasm between product management and development rather than working hand-in-hand as it is intended in Scrum.

Teams are not allowed to self-organize their alignment and predictability is the goal rather than maximization of value

SAFe accepts the status quo in many enterprises and distinguishes between Product Owner and Product Management. [] This does not help collaboration but is a tayloristic division-of-labor approach.

People don’t have to change with SAFe – they can just go on as they always did and call themselves Agile now.

SAFe believes that the world is good as it is [...] so the status quo has to be accommodated. Agile believes that the world is not a good place (at least for people working in software development) and should be improved. The status quo – especially the tayloristic thinking - has to be changed. This doesn’t fit together. SAFe isn’t Agile.

 

Update 2014-03-03:

• Ron Jeffries (one of the founders of Extreme Programming): SAFe – Good But Not Good Enough

SAFe wraps those ideas [of Lean and Agile] in a package that is designed — intentionally in my opinion — to appeal to today’s managers and executives who do not understand Agile, but who know they have a problem to which Agile may be the solution.
If everyone in the organization were to read the fine print in SAFe, then the organization might very slowly evolve to the level of effectiveness that real Agile provides. That’s not going to happen. Managers and executives are too busy to read the fine print. They are too busy doing their job to study how to do their job. They will too easily fall into old patterns of management behavior, and when they do, SAFe will be installed in a fashion that won’t just fail to support Agile, but that will suppress it.

 SAFe assumes that you need a big “solution” and then provides it. More than likely you don’t need a big solution. 

Something went wrong with Scrum: Analogies with the Underpant Gnomes and Sovjet Era Deathmarches

Story card hell is when you have 300 story cards and you have to keep track of them all. [...]
Every customer I've ever worked with wanted to put story cards in a database. That misses the point of story cards. If you're worried about losing story cards, you're doing too much with them. 

- Jim Shore, Beyond Story Cards (2005)

 

When I was getting to know Scrum, I was taught that it was very much about reacting to change. Product owners define and prioritise the product backlog. They can react to change by re-prioritising the product backlog. Sprints are short focused iterations where the team is protected from change and the associated task switching costs. The effort is streamlined during this period. But because Sprints are short, reacting to change is still possible on the level of the project.

Scrum poses several challenges for the product owner:

• He is challenged to break down backlog items so that they fit into sprints and are still valuable and independently deliverable.
• He is responsible for the detail-level of the items in the product backlog. This is a balance-act since the items on the top must be detailed enough for the team to implement them. However items lower in the backlog should not be too detailed, since they can change completely or even become obsolete during the course of the project. Spending too much time upfront on those items is wasteful. I was taught  that items that are not on the top of the product-backlog should just be placeholders for a "just-in-time specification".

It seems that the current state how Scrum is practiced does not reflect those ideas any more. The following burn up-diagram from a real project I have been working on is a good illustration:

The large gap between "Scope" and "Work Done" hints at a chain of problems:

• The product-backlog is used to plan the whole project and to monitor its progress.
• In order to plan and monitor the product-backlog should be as exact as possible. Therefore all items on the product backlog must be estimated as exactly as possible.
• When items on the backlog are estimated, they are no longer "placeholders for future refinements". They become specifications.
• An exactly estimated product-backlog suggest a certainty we simply don't have.
• Following from the above, the backlog strives to become a complete specification for the whole project.
• Reacting to change is no longer a main concern, when this complete specification is in place.
• … where is now the difference to a Waterfall Project?

 

It seems that my experience is not unique:

 

In his talk "Co-making Great Products" Jeff Patton draws a hilarious analogy between collecting a huge Scrum backlogs and the business plan of the Underpants Gnomes in South Park:

Gojko Adzic is even a bit more provocative (as always). In his talk "Make Impacts, Not Software" he compares the "Complete-Backlog-Antipattern" in Scrum to the Sovjet Era deathmarches like the construction of the White Sea Canal which was a showcase of the success of the First Five-Year Plan:

 

 

According to a recent Forrester Study the above way to practice Scrum is the mainstream today. Forrester calls this "Water-Scrum-Fall":

The reality of agile adoption has diverged from the original ideas described in the Agile Manifesto, with many adoptions resembling what Forrester labels water-Scrum-fall.

 

According to that study Scrum is not adopted by the whole company. Scrum is seen as the "Development Process". The input for that process is a long term roadmap or a complete feature list defined by the business. Then the development team does all their esoteric ceremonies like "Standups", "Sprints" and "Demos" but when you wait long enough, the output will finally be the complete delivery that goes to production.

 

I personally don't think that we gain many benefits from "Water-Scrum-Fall". In some cases it can even be harmful, and the project would be better off with a honest waterfall process. Gojko makes the analogy of putting a strong engine into a crappy car.

 

However I think that this is an inherent conflict between Agile methodologies and plan driven approaches. If you really need a plan driven approach (i.e. because you have a defined long term roadmap) then probably all attempts at using Agile methodologies are just pretenses and will eventually fall back to a disguised waterfall approach.

  

Postagile: A large Scrum backlog is like a big pile of underpants

We were used to getting shit done … then they told us about Agile.

- Hadi Hariri, NDC 2012

 

If your product backlog contains more than 50 Stories, there probably are inconsistencies due to lack of comprehensiveness.

- Urs Enzler on twitter

 

In his talk "Co-making Great Products" Jeff Patton draws a hilarious analogy between an anti pattern in Scrum and the Underpants Gnomes in South Park:

The business plan of the Underpants Gnomes looks like this:

If you think about it, then the business plan of the gnomes is not all too different to an excessively collected and groomed backlog in a Scrum project:

Let's remember what Agile is about:

• Individuals and interactions over processes and tools
• Responding to change over following a plan

Lately it seems that many incarnations of Scrum (typically flavours of "Scrum in the Enterprise") have lost their Agile roots.

In my opinion excessive grooming and estimating of a product backlog beyond the next two sprints is waste. Of course this depends on the definition of "excessive". A backlog should represent an idea where the product is heading, but demanding an exact and fully estimated backlog to plan and forecast efforts and deadlines of a whole project is just not what Agile was all about. In fact, with an idolized  backlog in Scrum there is not much difference any more to a phased waterfall approach … or to an ingenious gnome scheme of fervently collecting with the belief in subsequent profit...

In all the cases where I have seen this antipattern of an idolized backlog the pressure came from management which took the idea too far by striving for the perfect backlog. In most cases the urgent need for the perfect backlog is a result of overusing planning and monitoring tools: it doesn't matter if it is the handcrafted excel-sheet of the project leader or a sophisticated report-dashboard in Jira or TFS. All those tools imply that there is value in a more complete, more detailed and more accurately estimated backlog … but that was already the fallacy of the waterfall, wasn't it?

However it might be that all of this is just the return of an ancient and evil conspiracy:

 

Update 2013-06-01: I just watched "You Are Not a Software Developer! - Simplicity in Practice" which has a quote that fits totally in the context of this post:

For planning we need a roadmap, not a road! A roadmap is about option and choices, a backlog is not a roadmap.

On TechTalks Mind: New Podcast Episode

Check out our new podcast episode. This time Claudia talks with Willy about requirements gathering in Scrum projects. This is a german episode.

Visit "On TechTalks Mind" or subscribe in iTunes.

follow me on twitter, I need some friends :-)

 

Self enabling teams: a demonstration.

One of the most impressive memories from my Certified Scrum Master course last year, is the experience of growing a self enabling team.

The exercise was simple:

• Given was a bucket full of tennis balls.
• The team was about 20 people.
• Only one team member could take balls out of the bucket. He could take out as many as he liked at any time throughout the sprint.
• The team would score a point for each tennis ball that was touched by every team member and then touched again by the first team member, he who took the ball out of the bucket.
• The team would do 10 sprints. Each sprint is 2 min. Between two sprints is a break of 2 min for retrospective and optimization.
• Before each sprint the team announces the score it expects to achieve.

Before the exercise I thought that the main challenge would be figuring out the optimal formation of the team members. I expected that after that there will not be much improvement possible.

I was wrong! This was the result (second column: expected, third column: achieved):

I am still amazed how the team was able to optimize and improve in each of the ten sprints. This improvement was rooted entirely in the team itself and is a perfect demonstration for a self enabling team, that was given the chance to inspect and adapt.

Some of the optimization steps included:

• Finding out the optimal formation for the task: a circle.
• Optimizing the formation into two circles of people. The outer facing inward and the inner facing outward, this way we could pass the balls to a person in front of us and not besides us.
• Ignore dropped balls, continue with the flow.
• Not throwing balls to your peer, but dropping them into his cupped hands.
• Passing two balls at a time to your peer.
• Sitting down on the floor reduces the danger/consequences of dropped balls.

When is Scrum successful?

@EricMinio asked an interesting question last week in his Scrum in Depth course. It really made me rethink my understanding of Scrum.

 

A company is not able to deliver its software projects, but does not know why. You introduce Scrum. After 3 sprints you have shown up the problems. Does this mean Scrum works?

 

I think this very much summarizes what Scrum is about, however I think management will have a hard time to see this as a success...

By the way this question was @EricMinio's answer to a students question "Does Scrum work?". Again a great demonstration of a ScrumMaster's function not to provide answers but to ask the right questions.

 

follow me on twitter, I need some friends :-)

Podcast: Scrum in Fixed Price Projects

In the first episode of TechTalk's podcast Christian Hassa interviews Mitch Lacey about the applicability of Scrum in fixed price projects.

They talk about common misconceptions, pitfalls and dangers and how to avoid them. Mitch also presents some patterns that worked in his experience.
The podcast is in english.

You can download the podcast directly or subscribe to the rss-feed or subscribe to the podcast in itunes.


BTW: For people living in Switzerland or Austria: This February Mitch is holding his Certified Scrum Master courses in Zürich (8.-9.03.2010) and Vienna (11.-12.03.2010).

follow me on twitter, I need some friends :-)

Scrum Master Certification in Zürich with Mitch Lacey

February 8. and 9. Mitch Lacey is running his Scrum Master Certification course in Zürich Switzerland.

If you are thinking about getting the Scrum Master Certification, I can fully recommend Mitch's course.
I did attend his course in Vienna last autumn. Those two days were very interesting, informative and never boring. Mitch really is a great teacher and story teller with a lot of practical experience.

If you are interested you can find more details and registration here.

Update: On TechTalk's Mind hosts a podcast with Mitch discussing Scrum in fix-price projects.

follow me on twitter, I need some friends :-)