In my last post on April 12 I announced that I was planning a re-launch of my consulting services to help software teams and organizations take their game from zero-to-hero with a new range of services to help them transform toward iterative/incremental delivery practices.

I’m pleased to announce here that as of April 19, 2011 this new venture has been federally incorporated as Derailleur Consulting, Inc. and now open for business!  Props to my legal team at Cognition LLP for making this a straightforward process with exceptional customer service – it’s well worth taking the time to have folks like them help any new business venture to ensure getting the job done right from the beginning.

My reason for creating this new business is simple and straightforward:  I want to help software teams and organizations, wherever they may be, restore meaning to the concept of R.O.I. or Return On Investment through the application of iterative/incremental software delivery practices and attendant techniques.  As my byline states:  “Agile Team Transformations for World-Class Software Delivery.”

Derailleur Explained

Why the name Derailleur?  Good question:  I’ve had a long-time interest in bicycles and cycling, and it naturally occurred to me to draw an analogy between working with teams to help them be more productive and the simple device that helps multi-speed bicycles slip their chains to various cogs to enable the cyclist to be more productive with each pedal stroke.

Transforming teams toward iterative/incremental processes like Scrum are akin to when a cyclist shifts gears according to the conditions he’s facing:  Low gear for hills, high gear for flats and downhills to maintain momentum.  So it is with agile software delivery, where teams continually shift gears according to the conditions of their project and the market.

Shifting Gears

Since I started blogging in 2004, my focus has run the gamut of technologies and practices - things that have interested me as I’ve wound through several career changes.  So it shouldn’t be much a surprise that I will be winding this blog down as I focus on my new business.  It won’t be shut down right away – I do intend to keep it up for historical purposes – but it will eventually be retired.  Thanks to everyone who have followed my posts and offered public and private feedback!

For now, I’ll be signing off here and moving over to my new site, http://www.derailleurconsulting.com. It’s quite simple right now as I am still evaluating some Content Management Systems (Orchard is winning right now, if I can figure out how to brand it properly).  I’ll also have new Twitter and Facebook feeds for connecting with customers and readers – all of which are accessible on the new site page – while maintaining my current Twitter feed (@crchapman) for my usual off-beat musings.

Do drop by, add me to your Friends list, follow me, etc. to hear about how the business is developing and my exciting, new service offerings that can help teams and businesses become productivity powerhouses.

Cheers!

A brief note today as I celebrate a small anniversary and victory marking my first year back as an independent consultant.  At this time last year I embarked on my first project after leaving the world of Microsoft Consulting Services (MCS) Canada with a little ditty helping a local customer get started with SharePoint 2010 Forms Based Authentication customizations.  One thing led to another, and before I knew it I had a profitable first year.

Over this time, I began to help my customer (and eventually their customers) introduce discipline and rigor to their software delivery practices by applying a process that I’d been studying and applying for almost ten years:  Scrum.  As I had seen many times before, the benefits were tangible and almost immediate with improved team morale and productivity:  Where once confusion reigned, there was now a regular rhythm that helped focus and align the business and development practices.

As a result of these experiences, I began to see my priorities changing – I thought I’d be doing a lot more SharePoint work, but I definitely was getting a higher calling.

Opening Soon:  A Practice Dedicated to Better Practices

In September, I fulfilled a long-time ambition to attend a Scrum Master training course delivered by its most famous (infamous?) co-creator, Ken Schwaber at his new training offices in Burlington, Mass.  I came away stuffed with experiences, ideas and clarity about where I wanted to next focus my energies and talents:  Helping software teams and their organizations become great software teams and organizations by applying better practices like Scrum.

It’s a good time to do this as now, finally, after so long “agile” software development is now ascending into the mainstream (even if a little late here in Canada).  There are a lot of good teams who are languishing under bad practices that make it nearly impossible for them to achieve success:  The deck is stacked against them with prevailing practices and failure always looms large, requiring all kinds of unsustainable effort to stave off.  The business climate is demanding that they do more with less and produce real results that justify the investment.

Exactly the job that Scrum and its kin were designed to do.  By the end of the year, I was beginning to inspect and adapt my own processes and it was suggesting that a change in direction was required. 

In the next few weeks I will be re-launching my consulting practice with service offerings directed at helping software teams and organizations take their game from zero to hero:  This includes an array of on-site training programs, coaching and development best practices guidance directed not only at preparing the “boots on the ground”, but also managers and the larger business for the cultural shifts in thinking that just-in-time processes require. 

More information will be forthcoming, so stay tuned:  Season 2 is promising to be a blockbuster…!

Last week, Ken Schwaber opined in a blog post that Scrum is akin to learning the rules of chess:

Scrum is like chess. You either play it as its rules state, or you don’t. Scrum and chess do not fail or succeed. They are either played, or not. Those who play both games and keep practicing may become very good at playing the games. In the case of chess, they may become Grand Masters. In the case of Scrum, they may become outstanding development organizations, cherished by their customers, loved by their users, and feared by their competitors.

I’ve heard Ken use this metaphor before, most recently when I was at his PSM I course in Boston last Septmber.  He argues that intrinsically, Scrum and chess are just frameworks for achieving a desired outcome: One is an intellectual diversion with strict rules of engagement that govern “play”, the other is a process to get complex work done with strict rules of engagement that govern “who does what, when”.

It seems simple, right?  Well, clever by half for some of the detractors who had some bitter complaints in the comments (emphasis mine):

(Michael Dubakov) Chess is a complex game, still it has defined rules and there is NO DEPENDENCY ON ENVIRONMENT. In software development, environment is VERY diverse. It means it is hard to create a defined set of rules that should be followed and lead to success.

Are you one of those who believe that defined set of rules can be applied to any situation?

(William Pietri) As far as practical things go, the only things that can’t succeed or fail are religious activities. If a particular surgery doesn’t fix your medical problem, then it has failed. If your favorite prayer or voodoo ritual doesn’t deliver results, well maybe you did it wrong, or maybe the gods just have other plans…

Scrum originates and is almost always applied in a business context, something inextricably bound with material purpose…

As long as Scrum is being sold and bought to accomplish particular ends, I think it should be evaluated like any other practical intervention.

…

If Scrum can be thought of as something that can’t succeed or fail, then that should be equally true of any other software process. That makes questions of which approach one should pursue nonsensical, because they can’t be evaluated on practical grounds. I think that’s a mistake that would destroy a lot of productive dialog… Whatever you… claim, a lot of money is being made by people who raise expectations of what Scrum can do. There are a lot of employees suffering right now when Scrum fails to do what it says on the label.

These comments reveal a problem that the certification industry and associated snake oil salesmen have created for agile software delivery in its ascension to the mainstream.  In IT, we’re used to having a lot of IF..THEN structures embedded into our thinking and culture, and this leads some to see things in a very binary way.

Success or Failure in Scrum

As Ken explained way back in Scrum’s early days, the rules of engagement it sets out are deliberately built as a non-prescriptive framework that is checked and balanced to promote the optimization of team performance when working on complex, collaborative projects like software development.  They thus increase the probability of success.  In and of themselves, the rules of Scrum do not succeed or fail - they just are.  Success or failure (which must be objectively defined by the business) is a byproduct of the collaborative work of the entire Scrum Team, including the Scrum Master/Coach, Product Owner and Scrum Team.

What Scrum Says “On the Label”:  Continuous Improvement

When we look at the Scrum Guide (freely available as a PDF from Scrum.org here), we read the following:

Scrum is not a process or a technique for building products; rather, it is a framework within which you can employ various processes and techniques. The role of Scrum is to surface the relative efficacy of your development practices so that you can improve upon them while providing a framework within which complex products can be developed.

This has been the “promise” of Scrum since Schwaber and Sutherland first promoted it over fifteen years ago: A framework for continually improving and refining a product and a team.  It takes what you have on-hand (people, skills, technology, requirements) and gives you the best possible shot at Return On Investment.

However, if we take what William Pietri says in his comments at face value, teams are suffering because of Scrum.  Scrum is failing them.  But what does this mean?

Failure?

In my experience, “failing” at Scrum usually comes from several places all at once with new teams as they transition away from the chaotic governance of BDUF/waterfall or other high-ceremony/low-trust methodologies:

1. Scrum is open.  VERY open.  There’s nowhere to slack-off or hide;  the Scrum Dev Team moves as fast as its slowest members and the Daily Scrum makes this evident.  However, it also provides an opportunity to help team members remove obstacles every 24h – this is either welcome or it isn’t.
   
2. Scrum requires some intense participation by the Product Owner to establish what they want to build and to keep a prioritized queue of features.  They can throw an entire project into chaos or success depending on their approach (see this Agile Journal article for a real-world example).
   
3. Scrum doesn’t prescribe how to gather feature requirements or attain development best practices – eXtremeProgramming does.  It leaves it up to the team how they can achieve the goal of a functional, potentially-shippable product increment every 2–4 weeks.  This said, the obvious solutions to the problems a team will face meeting this goal includes writing user stories with good acceptance critieria, estimating with a point scale, using burndown charts to track progress and automated builds and tests to keep the developed system stable over successive iterations.  Ultimately, the team should coalesce around what works for them to meet the goals of each Sprint and Release.
   
4. Scrum depends on having a good Scrum Master to provide servant leadership to the Product Owner and Scrum Team.  For mission-critical projects, this shouldn’t be a member of the team who wears both hats or someone who just read the Scrum Guide – this must be someone who understands and has applied the fundamentals of Scrum (and indeed agile philosophy, values and principles) and is vested with management authority to ably help the team remove impediments.  This can and does include helping the business align their processes to support the Scrum Team.  The role requires rigidity to adhere to the basic rules but flexibility to allow the team to find their own solutions to big problems.

Success and Failure Defined by Context

A lot of folks like William and Michael want some objective measure of success, which should be easy in Scrum:  It’s the output of each Sprint – working, tested software.  If this isn’t the case, Scrum doesn’t prescribe what to do, but it should be readily apparent based on the “success” of the team achieving each Sprint’s goal.  This is something I have seen with every team I coach into adoption of Scrum.

Often, the Product Owner will seem discomforted with the reality of what their team is able to achieve, especially in the early iterations of a new team. They will fault everything and everyone else for the failings – it’s a common reaction, because previously they’ve never been so aware of what their team was capable of doing. 

However, because the team is given first-order preference in Scrum (vested with the authority to self-organize around their work and determine how to resolve problems themselves) and because they are given the opportunity to continually improve through Scrum’s baked-in two-week delivery windows, they can and do improve over time.  They become more and more able to achieve world-class delivery.

In my most recent engagement, I saw this over the four iterations I coached a completely green team:  Objectively, they were “failing” to deliver all features as completely tested.  They were taking shortcuts and impacting quality.  But they knew it.  Every Sprint Review.

Part of the problem was organizational, with a lot of delays getting automated testing tools that could have made the QA members of the team more productive and able to support the developers.  Part of the problem was with the team learning a new skill and technology.  Every Sprint Retrospective we considered what could be done to improve the situation.

By the last Sprint, the team was becoming a more cohesive whole;  they attained more fully-tested and delivered work than ever before.  They planned, estimated, worked and delivered like professionals.  This is success in Scrum.

In the previous installment of this series (Part I), I introduced the first seven observations and notes that I had compiled over the course of a recent project where I was engaged to help a new team and product owner shepherd a small SharePoint 2010 portal into existence using Scrum.

Here are the next seven observations that in contrast to those in Part I, go a little deeper into the meat of organizational and implementation challenges that new Scrum teams encounter.

1.  Build a Release Plan. Revisit and revise it often.

• It’s difficult to know if you are reaching the “promised land” without knowing what it looks like.  A release plan sets out goals for each iteration along with expected features.
  
• Go through the activity of building the plan to first determine if the project is feasible within stated parameters like cost and budget.  Return to the release plan as a preface to each Sprint Planning session and Sprint Review.

2.  You can only go as fast as your team.

• Your project can only progress as fast as your team can deliver features according to their abilities and your definiton of “done”.  Don’t goad them into taking more on to satisfy your ambitions:  It will only lead to an increased probability that they won’t fully deliver a functional, potentially-shippable product increment.
  
• Do not fall into the trap of believing that overtime solves all problems: In reality, quality often suffers as a result.

3.  Set an explicit definition of “done” and stick to it.

• When you set a goal for a Sprint or Release, be explicit with a shared understanding of what it means when a feature is considered “done”.  Often, this defaults to the old trinity of coded/tested/documented, however even this needs clarity: 
• Coded: Usually means writing the code, ensuring it compiles without breaking and is checked-in;
• Tested: Is wide-open to interpretation: Developer unit tests? Acceptance tests?  Automated?
• Documented: This can mean comments in the code, along with capturing pertinent design details in a central repository like a wiki.  On this project, it was deemed satisfactory if a feature could be traced from requirement to implementation and back again.  Fortunately, TFS with the Microsoft Visual Studio 1.0 Scrum template along with associating checked in code to a work item made this a snap.
  
• Without a definition of done that is set and agreed-to by the Product Owner, the team is left to their own devices to determine this, which can and does lead to unpredictable results.

• On the flipside of each user story card, you should capture some simple criteria to prove that feature has been implemented according to spec.  If you run out of room, you may need to break the feature up as you’re likely discovering new usability scenarios.
  
• This tripped up my team on several occasions, and often left QA wondering what they were testing.  This takes practice to be diligent about capturing and remedying.

5.  Involve QA as part of the team from the start.  Give them automated test tools.

• When QA is integrated into the Scrum Team, they are able to understand and discern features and create effective test plans by actively engaging the developers.  This has to happen on a daily basis.  It’s too late to leave it even for a few days.
  
• QA will fail if they are expected to run tests manually; they will fall behind and never catch-up.  Get them an automated UAT suite so that they can compile their test cases and run them automatically at the push of a button.  I recommend Telerik’s Web UI Test Studio.

6.  Don’t design in absence of the team.  Iterate.

• Wireframes and PhotoShop mock-ups are great for “seeing” a design and gettings a feel for it.  However, when they are developed in exclusion of physical implementation by the team, you really have a simulated sense of reality.
  
• This simulated reality often comes at a high cost the longer you wait to implement it;  it can and does create gaps where the mock-ups “hide” how certain features are designed to work, which leads to significant re-work later on, and always takes a lot of work to implement.
  
• Integrate design into planning for each iteration so that the team can translate them into working software.  The goal of every dollar spent in a project should be directed toward creating a functional, potentially-shippable product increment every two weeks.  Period.  Full-stop.
  
• Give your design time to breathe; leverage the opportunities that Sprint Review provides to check your assumptions against reality.  Adjust and improve every iteration.

7.  Trust the team.

• Don’t obsess over whether the team knows what tasks to take on and that you or a Project Manager knows better.  They can and will organize themselves around the work to be done in the most efficient manner because it is always less painful to do so.
  
• Have a question about the product under development?  Ask the team.
  
• Have a question about a new feature you’d like to see in the next Sprint?  Ask the team.
  
• The team will always have a deeper understanding of your product and the challenges faced to make it real:  Always be transparent.  Always take any issue or question to the team.

Recently, I concluded an engagement where I was coaching a new Scrum Team and Product Owner who wanted to build a SharePoint 2010 web-facing content management system with extranet access.  This proved a challenging endeavour on multiple fronts, from lack of proper training and preparation (no time) and experience with the technologies and problem domain to an inconsistent level of engagement for planning each Sprint and defining “done”. 

Over the course of the project, I compiled a diary of observations and notes about what worked and did not work for the Scrum Team and Product Owner as they moved through their project.  Below are the first seven lessons learned I noted: While certainly not new or unknown in agile circles, I present them to edify others who may be taking on a Scrum project for the first time and need some support in making the right choices to avoid problems they might encounter in their implementation.

1. Get training before you start the project.  For everyone.
• While some teams can learn-as-they-go, to be really effective you need to do some dry runs before beginning.  Learn the roles and their responsibilities.  Write some user stories, build and prioritize a Product Backlog.  Decompose it into a Sprint Backlog.  Have a Sprint Review.

• Effective training will give you 40% theory and 60% practical application - you need to do this so that you can begin your Scrum project with less learning curve friction.  You should be able to plan a two week Sprint from start to finish in four hours or less.

2. Scrum is not magic.  It's easy to begin, difficult to master.
• When I instruct teams on Scrum, I make a point of repeating the phrase: "This isn't magic;  it's a framework with rules of engagement that help you get things done."  Too often I find that organizations understand the brochure but not the manual when it comes to Scrum, and this disconnect leads to some shock and awe when the full scope of responsibilities is understood.
   
• Scrum isn't a means to get more done faster - in fact, for new teams they will feel a little slowed down - this is normal because they're building the discipline it takes to be a world-class delivery team.

3. The rules of Scrum are there for a reason:  Don't work around them.
• The first questions I'm often asked by a Product Owner and team is how they can abandon or work around the handful of rules that Scrum requires.  They are there for a reason:  To keep you and your team productive while promoting ROI. 
   
• They have been "play tested" by thousands of teams over two decades - they work because they are simple and effective, but it is hard work to stick to them.  This is why you need a strong Scrum Master / Coach - to keep you motivated as you build your competency.

4. Choose your Scrum Team according to your project.
• It's common sense:  To work effectively, your team needs to have, as an aggregate, the skills required to take on your project.  If they do not, they will move more slowly as they acquire these skills, which can be OK if you've got the budget to accommodate it.

• Having access to a Subject Matter Expert can help a team remove obstacles more quickly, but they are not necessarily an accelerator:  At the end of the day, the team still needs to do the research, whether with the SME or Google, and implement a solution.

5. It takes at least three iterations to get a team together.
• Learning Scrum is easy;  applying it consistently is the challenge.   I often relate the analogy of training to run your first 10k or half-marathon when you have no experience running at all:  You need to understand that newbies will take time to adapt to the pace and practices, and will sometimes falter in their training.
   
• Your new team will likely follow a curve where they are "failing" for the first three iterations as they learn and internalize how to work together as a unit.  Be patient;  expect them to hit their stride in the 4th - 6th iterations. 

6. If you're going to be the Product Owner, be sure you can meet the demands of the role.
• This is a tough and challenging role that has no real analog or precedent:  You are the active guardian of ROI:  You set the goals for each release and iteration and you steer the project based on the inspect/adapt planning and feedback points that bracket every Sprint timebox.  This means keeping your Product Backlog groomed and prioritized with good user stories and working closely with the Scrum Master and Team.

• If you feel that you've got too much on your plate to do this role well, you have two choices:  Delegate your plate to someone else so that you can devote enough attention to the project or delegate the role to someone who can.

7. Celebrate the achievement of every Sprint.
• I make it a point to celebrate the end of every Sprint with each team I coach by taking them out to a pub for a round on me - including the Product Owner.  By the end of a Sprint, a team is often exhausted and exhilarated:  They've hit highs and lows on the road to Sprint Review and they may or may not have made their goal.

• By taking the time to recognize their achievement through a small reward like getting together for a beverage, you validate each team member as a human being and not just a resource.  It gives them a relaxed forum to swap war stories, talk shop or what's up for the weekend while building team morale and loyalty.