Measuring and Comparing


The question or “normalizing” story points or comparing story points and team velocities has reached me once again. And once again I struggle. My canonical answer “you cannot compare story points of two different teams” was met with “I don’t believe it” and “then it is worth for nothing”.

Was the person I talked to being unreasonable? Well, with regard to the matter at hand, definitely yes – there are basic underlying assumptions in the method, which in itself make story points team specific. Yet, on the other hand, the person wanted a problem to be solved. And contrary to agile “feel-good” literature, measurement and control do have their place in real world work environments.

Why measure

When we start small, with one or very few teams, and everyone is collocated and aligned with goals and the general “this is how we do stuff” we are actually in a very good place. The people interact, and given a collaborative culture, overall throughput will be good throughout every team. In my experience the rather typical yet friendly competition between teams will also take care for improvements. In these settings I have rarely seen the call for measurements and comparisons. If a team has throughput problems or becomes a bottleneck in cooperation, the whole bunch will typically come to help.

But in other settings the picture does not look that good. Assume that the teams have to work together, yet on different components, using different technologies and in a climate of distrust and non-collaboration. To some extent they cannot help each other. To some extent, they wouldn’t help each other, even if they can. And if the environment is one, where a wrong action taken on your own might come with strong repercussions (a so-called CYA environment), everybody is busy just looking better than his neighbor anyway.

These are to some extent dysfunctional environments, where we are confronted with the urge to measure “to incentivize the performers and get rid of the rest”. Admittedly that is not what we want.

But there are good reasons for measuring in a similar scenario, especially if the culture is not as bad, as I have described it right now. Think about the multi-national company, think about different locations, near- and offshoring. Think about different technologies applied and a heterogeneous IT environment. As a manager I want to know, where I get best value for money. It is my duty to look into that, even if we tech people (yes, I still think about myself as a technician) don’t like that too much.

And yes, as you can see, I might be interested in understanding, if team A can implement a feature faster and/or better than team B. Not because I want to get rid of a team. Implementing something faster or better can have many different reasons. And as a manager I want to know these reasons and address the root cause.

Example: Think of a team which continuously delivers rather low quality increments, as compared to the other teams in your company (and admittedly, error rates are easier to check). You might look into their development methods. You might look into the quality of their stories. You might look into their testing, their test environment, and their build pipeline or version management. And what you might find is state of the art. Test driven development, superb build and deploy pipeline, highly educated people knowing their stuff, well written stories. So what is it? In our example it is actually a piece of technology the solutions are built upon, which is very hard to manage and configure and withstands reasonable automation approaches due to the lack of open APIs. And yes, there are these pieces of software out there, setting up even the best teams to fail.

What do we really want to know

An agile team tries to improve its performance and to become better. As a company I have to become “better” continuously as well (admittedly, depending on the market, the pressure to improve might be vastly different). As a manager I am interested in improving (beware of the buzzword) “value stream”. That is what I am really interested in. Getting out things that matter at optimum cost. Beware: this is not always (or better: nearly never) the cheapest implementation. As we all know, most of the costs of a piece of software or of a system are actually accrued during operations. Quality and good design matter a lot.

So I might want to know, whether it is better to do a piece of software in house or work with a trusted and known outsourcing partner. Measurement comes into play here: If I know the one side delivers better value for money – that I get the same system in the same quality cheaper – then I want to know that.

What we really should compare

Based on that reasoning the only thing we should be interested in comparing is business value delivered for a given amount of money. The given amount of money should include all costs that the deliverable will incur in the future. Since that is hard to estimate, I go for the second best solution I can think of. A set of thoroughly checked acceptance criteria, from code quality to acceptance test, from code style and architecture down to the operational documentation and user manual.

So when it comes to comparing teams and the value they provide, the only thing we should look at is their results and their quality. Because in the long run we are not paying for story points. We often pay for bugs, unnecessary or badly designed features and customer dissatisfaction. Yet we want to pay for the next big thing!

A short PS on corporate culture

One valid criticism to this approach is, that the whole thing would not be necessary, if the corporate culture was “good enough” to allow agile principles to work and to be widely understood and supported in the first place. I agree. But in my experience there is nothing more complicated than changing a culture, especially when we think about multi-national, multi site companies. And while you should always work on improving culture and organisational trust (this is in the interest of all companies), the aforementioned approach is something which might help faster.

Leave a Reply

Your email address will not be published. Required fields are marked *


Time limit is exhausted. Please reload CAPTCHA.