Agile software development, steps to work with Requirements, Estimation and Planning

Often managers determine a team member’s workload capacity in terms of time. That is, managers estimate how long they anticipate certain tasks will take and then assign work based on that team member’s total available time. This is problematic because it does not distinguish between a story that is very hard to complete and one that is undemanding; it only considers how long the work will take!

How to Estimate?

In the Sprint Planning Meeting, the team sits down to estimate its effort for the stories in the backlog. The Product Owner needs these estimates, so that he or she is empowered to effectively prioritize items in the backlog and, as a result, forecast releases based on velocity. This means the Product Owner needs an honest appraisal of how difficult work will be. Even when the team estimates amongst itself, actions should be taken to reduce influencing how a team estimates. As such, it is recommended that all team members disclose their estimates simultaneously. Because individuals “show their hands” at once, this process is not unlike a game of poker. Unsurprisingly, teams often call estimation “planning poker”. Some teams have even developed their own decks of playing cards expressly for this process.

I like to remind people that bad habits are not overcome by the tool alone. You can still put silly estimates (ranges in this case) into the tool. The same pressures that got you into the bad habits in the first place can do the same to the newest tool, no matter how good the underlying methodology is.

Software Estimation: Demystifying the Black Art by McConnell's. It's the definitive work on the topic. Anyone running a software project should own a copy.

So following are some good practices for estimating software:

1. User Stories
2. Planning Poker
3. Story Points
4. Product Backlog
5. Velocity
6. Re-estimation

What is a User Story?

A user story describes the desired functionality from the customer (user) perspective. User story descriptions are traditionally handwritten on paper note cards, Ron Jeffries has named these three aspects with the wonderful alliteration of card, conversation, and confirmation. The card may be the most visible manifestation of a user story, but it’s not the most important. While the card may contain the text of a story, the details are worked out in the conversation and then recorded and verified through the confirmation. A good user story describes the desired functionality, who wants it, and how and why the functionality will be used. - Wiki Definition.

Bill Wake provides us with the INVEST acronym, which says that items should be Independent, Negotiable, Valuable, Estimable, Small, and Testable.

Why User Stories?

User stories emphasize verbal communication. Written language is often very imprecise, and there’s no guarantee that a customer and developer will interpret a statement in the same way. For example, at lunch recently I read this on my menu: “Entrée comes with choice of soup or salad and bread.” That should not have been a difficult sentence to understand, but it was. Which of these did it mean I could choose?

1. Soup or (salad and bread)
2. (Soup or salad) and bread

Useful for Planning

A second advantage of user stories is that they can be used readily in project planning. User stories are written so that each can be given an estimate of how difficult or time–consuming it will be to develop; use cases, on the other hand, are generally too large to be given useful estimates. Also, a story is implemented all in a single iteration of an agile project, while it’s common to split a use case across multiple iterations (even though those iterations are usually longer than on a story–driven project).

Who Writes User Stories?

Anyone can write user stories. It's the product owner's responsibility to make sure a product backlog of agile user stories exists, but that doesn’t mean that the product owner is the one who writes them. Over the course of a good agile project, you should expect to have user story examples written by each team member. Also, note that who writes a user story is far less important than who is involved in the discussions of it.

Planning Poker

Software estimation that really works!!

What if you could answer the question “How long will my software take to build and when will it get done?” by playing a game of cards?

No I am not ... Let's see how we can do it

1. Planning poker is attributed to Grenning and is a fairly recent development (2002)
2. Planning poker combines expert opinion, analogy, and disaggregation into an enjoyable approach to estimating that results in quick but reliable estimates
3. Participants in planning poker include all of the developers on the team
4. The product owner / business owner / business analyst participates in planning poker but does not estimate
5. Someone (typically the PM) plays the moderator
6. For each user story or theme to be estimated, a moderator reads the description
7. The product owner answers any questions that the estimators have
8. After all questions are answered, each estimator privately selects a card representing his or her estimate
9. Cards are not shown until each estimator has made a selection
10. At that time, all cards are simultaneously turned over and shown so that all participants can see each estimate.
11. It is very likely at this point that the estimates will differ significantly.
12. If estimates differ, the high and low estimators explain their estimates
13. Repeat until the estimators have reached a lose consensus
14. The consensus number is size estimate
15. The team will decide how much work they can achieve in the given iteration based on estimates, for ex, from the estimates above the team decides they can finish the story “X” in the one week iteration
16. After the iteration finishes if the team has successfully delivered the story “X”
17. we can assume that the team can deliver 5 size equivalent of work every week, this is called the “velocity” of the team
18. In other words, “velocity” is the amount of work the team has been delivering over a period of time
19. Once a velocity is known, one can play Oracle and answer the question “How long will the software take to get done and when will it get done?”
20. In the case above if the team’s velocity is 5 size per week, one can predict that the team should be able to finish the work in 3 weeks
21. The time estimate of 3 weeks was arrived at by dividing the total size of the project with the velocity i.e (5+8+2) = 15 / 5 = 3 weeks

Why it Works?

• Those who do the work estimate it.
• Emphasizes relative estimation
• Estimates are within one order of magnitude.
• Reduces anchoring - Everyone's opinion is heard.
• Modeled for open discussion – forces thinking.
• It brings together multiple expert opinions to do the estimating.
• Kind of like “wisdom of crowds”
• Very web 2.0 ish
• Improves the quality of estimation
• estimators are called upon by their peers to justify their estimates
• averaging individual estimates leads to better results
• Fibonacci numbers – “golden ratio”
• The unexplained, kind of like the “X – Files”
• It works because it’s fun
• Don’t things always work out well when you are having fun?

Side Effects

• Greater understanding of work to be completed
• Expectation setting
• Implementation hints
• High Level architecture and design discussion
• Ownership of estimate

Story Points

• Very common way to estimate work
• Used to denote a name of the estimating unit
• Derived from calling a unit of work a user story
• Produces Unitless but numerically relevant numbers
• A 10 point story would take twice as long as a 5 point story  

• Have three key benefits 
• Forces the use of relative estimating
• Focuses us on estimating the size, not the duration
• Puts estimates in units we can add together  
• Based on historical reality
• Uses Fibonacci sequence – 1, 2, 3, 5, 8, 13, 21, …  
• Easy to use and understand

 

Important Notes (by Martin Fowler)

• "Story Points" is the most common name that I hear these days, but various terms have been used over the years, often with whimsical names that emphasized their arbitrary nature. I particularly like Joseph Pelrine's gummy bears and Josh Kerievsky's: NUTs (Nebulous Units of Time).
• This is a Fibonacci sequence
• Using the top number as too big is saying that a story sized at '8' really means '8 or more'. If you do this beware of using this top number when making forecasts of things like completion time, since '8' can turn into all sorts of numbers when it finally gets broken down. It's usually better to explicitly say its too big to be estimated rather than use a false marker number. 

Side Effects  

• If you're using Fibonacci numbers (or something similar), it limits the number of options when estimating a story.
• I know a group that used low numbers only: 1, 2, 3, 5 and 8. They had a reference story that was a 5. This enabled them to easily make snap decisions on a story's complexity while doing Planning Poker.
• The other side effect was that anything rated a 8 probably had insufficient information and needed to be broken down further. I seriously doubt it would have been that easy and straightforward if they were using raw hours.

Plan

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.--Abraham Lincoln  

A good plan is:

1. Clear 
2. Reliable 
3. Used 
4. Available

Why traditional planning fails?

• Plan is not adaptive.
• Focus on activities – Not on features
• Never finish early.
• Lateness is passed down.
• Dependencies.
• Plan is not priority driven.
• Higher risk.
• Lower ROI.
• Belief that customers will not change
• Estimates become commitments.

What makes planning agile ?

• Planning Is not considered as a phase in the project
• Focus on business priorities
• Deliver something each iteration.
• Changes are expected, embrace change.
• Plans are easily correctable.
• Plan by feature not by activity.
• Clear definition of progress
• 50% = 50% of features complete.

First, We Need a Backlog

Add Value to That List

Estimate and Asses Risk

Prioritization

Now it is Prioritized – We are Ready!

Sprint

Before the sprint:

• The team committed to the following:
• As a user I want to see my balance
• As a user I want to see last transactions.
• Creates the sprint backlog:
• They break it down into tasks.
• They estimate the time for each task.

Tracking Progress

Initial Stage

Next Stage

End

The Sprint has Finished

Velocity

• How many points can the team(s) complete in one iteration.
• Easy to measure.
• Fixes estimation errors.
• Easily reflects the project status.
• Primary parameter in planning.  

Warnings about velocity

• Don’t create incentives for velocity goals
• Don’t make stretch goals
• Don’t measure individual velocity
• Don’t compare the velocity from one team to another 

Sins of Software Estimation (ref. Fred Brooks and Steve McConnell)

Common Mistakes

#1: Estimating how long “it” will take to build before anyone knows what “it” is.

 

#2: Assuming that the most reliable estimates come from the people with the most powerful vocal cords…
Assuming that the sales department is better at estimating software projects than the programmers are…

#3: Creating an estimate from a new project by comparing it to a past project… which overran its estimates (worked massive overtime..) and ultimately realizing that you based the new project’s plans on the past projects estimated result instead of its actual results.

#4: Creating estimates that assume that no one needs initial training..
Or attend meetings…
Or be called to work on another project…
Or need to support a key customer…
Or take a vacation…
Or get sick…

#5: Presenting estimates with a high degree of precision (e.g. 67.4 days) but that are supported by only a low degree of accuracy (±2 months) 

Confusing Estimates with Targets

• Target setting is a key part of the art of estimation.
• When you are asked to provide an estimate, determine whether you’re really supposed to be estimating or figuring out how to meet a target.
• This is best treated as an iterative process that brings estimates and targets into alignment

Saying “YES” When you Really Mean “NO”

Why Developers say YES ?

Fred Brooks says it best: “It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.”

• Schedule “Negotiations”
• Software developers tend to be introverts and relatively young
• Marketing and sales personnel tend to be more extroverted and organizationally senior to the developers they negotiate with.

Making Commitments Too Early in the Cone of Uncertainty

Assuming Underestimation has a Neutral Impact on Project Results

Parkinson's law

“Work expands so as to fill the time available for its completion.” by Cyril Northcote Parkinson

• Overestimation will let Parkinson’s Law to kick in which is: Parkinson’s Law will kick in—the idea that work will expand to fill available time.
• Underestimation will create numerous problems like:
• Reduced effectiveness of project plans.
• Statistically reduced chance of on-time completion.
• Poor technical foundation leads to worse-than-nominal results.
• Destructive late-project dynamics make the project worse than nominal.

Effect of Estimation Accuracy

Fixating on the Science of Estimation When You Really Need the Art of Estimation

• Science is well-developed and well supported by software tools.
• The art of estimation relies more on rules of thumb and still needs some work.

Providing Off-The-Cuff Estimates

• Project teams are sometimes trapped by off-the-cuff estimates.
• While this is a simple point, off-the-cuff estimates is one of the most common errors that project teams make. So in order to become a better software engineer, we need to stop delivering off-the-cuff estimates!

Surveys

This survey was performed the last two weeks of January 2012 and there were 149 respondents. The survey was announced on the Agile and Lean Discussion group on LinkedIn, the Agile Project Management group on Yahoo groups, and by me via Twitter. The goal was to find out from agilists how they were addressing planning activities on their projects.

1. What is the value of common agile planning practices?

2. Adoption rate of look-ahead planning.

3. The frequency of coordination meetings.

Conclusion 

 

I want to repeat the same lines from my previous article:  

A good agile team picks and chooses the management and technical practices that best work for them. When trying to adopt Agile practices, there will be a ton of excuses as why it won’t work. Those who understand the real benefits of the approach – and genuinely want to make the transition – will likely have success. Those who are searching for reasons why it will fail – well, they will likely find them and either abandon the effort entirely, or end up practicing what Elisabeth Hendrickson calls “fake agile”.

It is better not to fixate on any given methodology, because the needs/conditions of the company and project are likely to change regularly, and you need to be flexible in how you approach managing projects if you want them to be successful. No single methodology is a silver bullet, so the trick is to determine which methods work for you and tune your methodology to suit your individual needs. This is what being "Agile" is fundamentally about.

• Get Part 1 of Agile software development methodologies.

References

• Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))
• http://agile.dzone.com/users/drianmitchell (Agile Estimation in Practice)
• http://agile.dzone.com/articles/agile-estimation-practice
• http://www.mountaingoatsoftware.com
• http://www.construx.com/
• http://en.wikipedia.org/wiki/Brooks%27s_law
• http://www.stevemcconnell.com/
• http://martinfowler.com/
• http://www.wikipedia.org/