Is “Quality” a Trade-Off?

One tool taught in Agile Methodology is the use of “Project Success Sliders”, initially created and devised by Rob Thomsett in his book, Radical Project Management. Agile training takes this approach and there is now a nice tool to use when considering project success sliders (see Mike Cohn’s Blog). The theory is that Sliders are a way to convey expectations to the team. By default there are six sliders, each of which reflects some dimension by which project success can be determined—for example, delivery of all planned features, quality, meeting an agreed upon schedule. Each slider goes from 1 to 5 with the midpoint being 3.

My first reaction to this is “What? Trading off Quality?” What does it mean to have a Quality “slider”?

In my years of software management, I’ve always believed there’s “3 fuzzy peas”, not four (the 3 P’s that drive a software project: Plan, People and Product. See my blog on Fuzzy Peas). Related to the sliders above, Plan=Time, People=Budget, and Product =Scope.

The first slider above, Stakeholder Satisfaction, I believe comes from delivering a quality product on time with the features they deem valuable. Team Satisfaction comes from working in an environment where your work is valued by the customers. I don’t think those really are project “sliders” although I imagine someone could argue for some usefulness in having them discussed by the project team.

On the other hand, “Quality” isn’t an item to be traded off.

I was happy that in Mike Cohn’s Blog, in every example the quality slider is always in the middle, at “3”. That makes sense – there is a middle ground on ‘reasonable’ quality. You typically don’t need to run a battery of tests that takes every possible path through the system regardless of how obscure it is. I’ve agreed to let a release go out when there was a potential bug in a place a user could only get to if the moon was full, they stood on their head and typed ‘MeMeMeMeMe’ 5000 times in the input box. Well, maybe a little easier to get to than that but some boundary test that goes far beyond any reasonable business case. Or a low-level issue (typos or an issue only an admin user would see and could easily work around). I suppose some could argue those are still ‘bugs’ in the code. Those bugs we list as 3-Low/3-Non-Critical and move them to the “Future” bucket. If we ever run out of work, a someday clean-up pile. On the other hand, if no one will ever encounter them, probably not worth messing with the code to fix them. Every code change is a potential new bug, they say.

But that isn’t what I think when I see a “Quality Slider” on a screen. I think that people may think they can actually move it off of the centerline “3” position into a lower-quality setting as an up-front project “trade-off”. This isn’t a tool used at the end to help figure out how to get an overdue release out-the-door, it’s an up-front tool. “Let’s add these 3 additional features and we’ll deliver it with a few more bugs.” Yikes.

And I fear (I know) it is common in the software industry to do just that. At Azerity and other places I’ve worked we’ve had a zero-bug policy for releasing a release. Every bug that was identified and classified as a potential user issue had to be resolved and tested in order to release the product. We of course also had an ‘exceptions’ clause (in business, one has to be sensible) and if there were one or a handful of issues found at the very end of the test cycle that were not deemed to be likely to cause any user real problems, or if it may only affect, say, one admin user who could be counseled how to avoid it until a fix were released, we had a sign-off process for those. But, believe it or not, that sign-off wasn’t needed in every release or even most releases. With an appropriate “Plan”, schedule (sufficient “People” for the Plan) and reasonable new features or “Product” size (the 3 P’s), you shouldn’t ever consider quality as something to be traded off.

What does it mean to “slide” the quality slider to the the left (lower quality), from a ‘3’ to a ‘2’ or even a ‘1’? If you ship software with known bugs that clients will find, it impacts tech support, causes escalation issues, client dissatisfaction, and costs more to fix later.

What’s the cost of bugs in delivered code? It’s a proven software fact that if fixing a bug when found in the current release takes the developer an hour (say $100), if instead it’s found by QA it’s $1000 (QA test time, goes back to development, potentially other code is impacted, code re-checked in, re-tested – at least a day total). But if found by the client it could easily end up $10,000 or more. (Tech Support time, trouble-shooting, reproducing it on in-house systems, getting developers on it – diverting them from their work, pulling the old branch to their dev systems to fix, merging code if other changes have occurred since). Ten-fold increase in cost for each development phase the ‘bug’ goes through without being caught. (That is also the argument for good code reviews, unit tests, and skilled developers).

I can see an argument for saying the Quality slider moves right for FDA or satellite software (you often can’t fix ‘bugs’ in space). But that is more of a company-wide process issue (how is the program run, what types of tests are required for delivery), not a Agile project-by-project team “trade-off”. So I still don’t see a reason/purpose for a “Quality” slider.

What bothers me with having “Quality” as a slider is I’ve been in companies where quality actually “is” considered a slider. I’ve been in meetings where engineering VP’s actually say “well, you can’t deliver code without bugs”. People I’ve worked with retort “of course you can – we’ve always done it.” But too often it seems to be a common software industry belief that software inherently has to have bugs. In my experience, that isn’t the case. There needs to be a customer outcry against shoddy software deliveries.

I think having a Quality slider is wrong. While it’s true that in real life quality does get compromised. Requirements keep morphing until there’s no time to finish the job, engineering over-designs software beyond the requirements and then can’t deliver, or management doesn’t come up with the people needed to meet the schedule but the schedule doesn’t move. In those instances, often, as the deadline for releasing the product is fast approaching the decision is to “Ship as is – we’ll fix the bugs in the first maintenance release.” That’s always the wrong decision.

The right approach is to continue, throughout the project, to manage the P’s – Plan, People and Product. If the schedule (Plan) is the most important part and at some point the people you have are all you will have (at some point bringing on new bodies doesn’t help due to ramp-up time), smart managers will start cutting out functionality – quickly. I was always quite draconian with my red pen if releases looked like a release was in trouble. Reduce the load and save the schedule. Or negotiate for more schedule. But that’s at the END of the project cycle, not during the planning phase.

Up front, where the project sliders are used, the test plan needs to be appropriate for the amount of code being implemented/modified.

Quality isn’t a “Trade-Off”.

Advertisements

Where there’s Smoke, there’s Fire

Last month, Northern California was ablaze! 1200 wildfires burning – most due to dry lightning, some unfortunately from arson. We went out anyway, spending weekends dawdling on the delta, in an anchorage with our powerboat tied up next to our friends’ big new sailboat, the sky smoky, the sun reddish through the haze. We should probably have been inside a house with the windows closed for our health! It was apparent to everyone in Northern California that we have both smoke and fire here

But sometimes the smoke is hard to see even when there’s fire a brewing. Maybe the smoke is just a wisp or the fire is smoldering under wet ashes and everyone just thinks it has been extinguished.

It’s similar with software management. How can you be sure you aren’t missing the signs of your project going astray? Problems brewing under the surface.

The best way is to focus on your customers and how they perceive your company and your product. The old adage “The customer is always right.” is a good barometer for gauging how your company and your product are doing. Is there smoke spewing and you are not paying attention or is it smoldering and hidden but underneath the surface?

Almost every company “says” they are focusing on the customer and usually decisions are being made in what is perceived to be the best interest of the customer. But often we think we are working for the customer’s benefit but we’re missing some key points. Are we focusing on only one aspect of what they want yet not delivering what they really need?

For example, how often have you heard a project team say “The customer’s schedule for delivery is on July 7th so we had to freeze the design last week to meet their schedule.” But is the design that was frozen going to meet their needs? Is the code that is being delivered going to best solve their problem? What if the design team was stymied with how to meet the requirements. Should the team just go ahead freeze the design, code and then deliver just to meet the schedule? Where is the trade-off between quality and schedule? Maybe delivering whatever we can in the required timeframe avoids the big explosion, the blow-up that would occur if the project manager had to tell the client that they can’t meet their schedules. But it doesn’t change the fact that there’s smoldering embers underneath the ashes and eventually when the wind blows (when the customer starts doing their final testing) those smoldering embers will erupt in flames. When the software is delivered but it doesn’t meet the requirements, there will be fire. The best managers will be willing to take the heat and tell the customer up-front if the team can’t meet the schedule. Of course, the underlying problem – why the team thought they could meet the schedule but then missed their target – needs to be examined and rectified so the problem doesn’t happen again. But if the team typically estimates well and is able to perform, but for one project there is a snag, then the customer isn’t served by focusing only on one element of the delivery, the schedule. Quality always has to come first. In this case, “quality” means delivering the software which meets the agreed-upon requirements, requirements that truly meet the need from the customer’s perspective.

Project Managers need to continually scan the horizon for smoke that indicates a fire about to erupt. A project manager that declares milestones complete without actually completing the work is always a sure danger sign. Schedule versus quality is just one example but one that seems to occur far too often in real life.

Fuzzy Peas

We lived in North Carolina many years ago (our youngest daughter was born there).  On weekends we liked to take rides in the car, my husband and I and our oldest daughter, then two.  We’d go into the mountains, visit the furniture stores, or drive off to the sea side.  One of our favorite places was Ashville, NC – in the foothills of the Blue Ridge Mountains.  It was lovely there in June – just the right temperature.  Not too humid.  Enough elevation to get away from the early summer heat. 

There was a great big old house in Ashville the looked like it could have been a plantation or stately manor.  It had a huge porch all the way around the house .  The owners had turned it into a restaurant and the porch now had picnic tables set for visitors.  We went there the first time with our 2-year old daughter and were seated at one of the picnic table overlooking pine trees.  There were no menus but the table was already set with plates and dinnerware.  Shortly a woman came to our table and said “Today we’re having chicken and pork.  Would you care to stay for dinner?”  Not sure what that meant, having expected a menu so everyone could order their choice, we wanted to find out what this Southern option was so agreed and soon large bowls of roasted chicken, pork, boiled potatoes, and black-eyed peas were brought to our table – Southern cooking, family style.  I’d never had black-eyed peas before.

Years later I attended a software management lecture by a man from Tennessee who, with his very Southern accent, talked about the “fuzzy P’s”.  I initially thought he was referring to those black-eyed peas from the South.  But no.  He was referring to the 3 P’s that drive a software project:  People, Plan, and Product.  “People” are the number of heads you can put on the project.  And while you can’t gather 9 women and produce a baby in a month, there are some impacts that can be made if the right resources are allocated to the right schedules.  “Plan” is the schedule – moving the schedule in or out is an obvious choice and one of the ways a manager can effect the end result.  And “Product” refers to how much product (how many changes, bug fixes, enhancements) is included in that release or that service delivery to the customer.  Remove some features, save some time. 

The three “P”s can be adjusted to affect the end result.   But that’s it.  Those are the only viable axes in the three-dimensional world of software that can be controlled and still produce a good, product.  If axis one doesn’t get shortened, the others will not be impacted.  If a software schedule can’t be met, then either more people are needed or less changes / enhancements/ fixes can get into the delivery. 

Usually CEOs want it all – they want the product with all the specified features in the timeframe they want it using only the resources that fit their budget.  But if the three axes don’t align, something’s got to give.  And it’s the software manager’s job to juggle the axes – more people here, less product there.  But CEOs push back and too often software managers try to appease them and agree to accept the dictated schedule with the resources allocated and all the specified product features. 

And there’s only one result – the hit is on quality.   When there aren’t enough resources to do the job right, quality suffers.  It isn’t always apparent to the CEO.  Perhaps the team even thinks they are doing a good job by delivering the product and making the milestones.  There’s a big party to celebrate the release, and everyone is congratulated.  But it’s the customers that will be impacted when they encounter the bugs that ultimately will result. 

And ultimately this approach will affect the bottom line.  It’s another well-known software rule that bugs found by a customer are 1000 times more costly to fix than bugs found during the design phase.  If a bug found during design (or at that point, an issue or problem) take a few minutes – say it would cost $10 to fix, if it isn’t found in design but rather during coding it will cost a couple of hours or $100 to fix.  If found during the QA cycle it costs $1,000 to fix (develop, re-test).  And the same issue found by the client costs $10,000 to fix.  Measure it.  It’s a fact.  Issues found by clients need to go back to the design, impact code, changes are likely to cause other issues, QA needs to be re-done.  Manuals updated.  Other clients notified.  It’s a very expensive proposition.  Not only that, it affects the customers’ perception of the company and it’s software.

So why isn’t quality the primary focus since it’s the most expensive error to make?  It’s the fourth fuzzy P.   “Perception.”  As long as the CEO “perceives” that the product is going out regularly, that everything is on track, managers are rewarded and all’s well.  Or seems to be.   But letting quality slide is a slippery slope.  If no one is tracking the overall quality metrics, quality can slide without anyone noticing until the product has degraded to the point the customers rebel.  Take the Microsoft operating system years ago where the blue screen of death was the well-know scenario. 

Bottom line – the trade-off should never be quality.  Good software managers need to watch their P’s and their Q.

Carrying the Torch

2008_Olympic_Torch_Relay   As the Olympic Torch heads towards San Francisco and what should be an event that joins nations together instead has been mired in controversy, I think of how often one person carrying a torch can bring to light issues and needs that otherwise would remain in the dark.

In a company, torch bearers are needed in every organization.  It’s easy to get in a rut of complatency, doing your job day after day.  Often no one notices the slow deteriorization of quality or effectiveness.

Another way of saying it may simply be that no one is “watching the ship.”  Some might argue “Isn’t that just a normal part of good management?”  But to that I’d respond “Yes.  But…”

Yes.  It’s true that good managers continually monitor and measure performance.  But after 30 years in the software business I can also say it’s common for even good managers to get focused on the wrong metrics.  Or focused on fighting fires and the performance and process metrics go by the way-side.  Or focused on following the company mantra and miss the signs that indicate real, underlying trouble.

March’s blog talked about the use of metrics to identify and quantify changes in effectiveness over time.  But often the message that the metrics are elucidating go unnoticed unless people are ready and willing to carry the torch to help the decision makers and top-level management see the light.

As Time Marches On – Use Metrics

It’s March already.  As days, months, and years pass by, often we just move ahead, one step after another, and don’t lift our heads up to see if we’re going in the right direction or what progress we’ve made.  Periodically we need to stop, step back, and assess our progress and how we’re doing.  True in life, true in software companies.

Sometimes in a software company, all organizations are hard at work but something is amiss.  In one software company, the technical support team was feeling that the customer’s needs weren’t getting addressed yet all of the product organizations were working hard, producing new releases with client-requested enhancements, and regularly issuing standard bug fix maintenance releases.  All of the orgs felt they were busy and overworked but that the product and quality were on track.  But by using metrics, they were able to assess the real status.

Metrics were evaluated about the number of customer calls currently being reported that were product bugs or other product issues versus the number one year prior and two years prior.  The metrics included turn-around time to get the issue resolved. 

What was clear from the metrics was that the number of bug reports had been steadily increasing as new clients buying and installing the software and existing clients were steadily upgrading to the newer releases.  In parallel, several new projects were underway, stretching the bandwidth of the product marketing, development and QA orgs.  So instead of trying to quickly fix all newly reported issues as they came in, which had been the process in prior years, in order to reduce workload on the developers and QA, fixes were being pushed out to maintenance releases two, three, or more months in the future instead of the next planned release.  As a result, more clients were finding related product issues and more issues were being escalated.  So to appease the clients who complained the loudest and wouldn’t wait for the future releases, the clients were sent one-off class files, tested only by the support organization instead of QA.   If multiple clients needed the change in different releases, the developers zipped up sets of fixes.  Then confusion ensued about which client had what file and instead of easing the load, this new degraded process was actually increasing the amount of work due to more call and more one-off fixes.  And as a results, the overall product quality was impacted, causing more client frustration.   When compared with prior years where bugs were immediately categorized and important issues quickly fixed, now there were too many fire drills and much confusion.

Metrics in this case uncovered both the negative quality trend and the underlying cause.  But there is a right way and a wrong way to use metrics.  A company can recognize metrics used in the wrong way when employee behavior is effected in non-useful ways.  For example, one company used metrics to measure their Technical Support response time and rewarded the techs for maintaining 90 percent first-customer-contact turn-around time in less than four hours.  The TS metrics looked great but in reality what the techs were doing was that when they received an automated call from a client, they would place their return call during the lunch hour or just after the company closed, raising the probability that they would be able to simply leave a voice message thereby responding to the call within 4 hours but without having to spend time discussing the call or resolving the problem which could tie them up and make them miss another client’s 4-hour call window.  As a result, clients were not talking to a human for one, two days or up to a week and were playing “telephone tag” and getting frustrated.

In another company, a percentage of each developers merit plan was based on low bug count.  But often issues reported by users as “bugs” were in reality items that were never spec’d or were spec’d incorrectly. So a lot of conflict resulted, arguments between the development org and support arose (“It is a bug.”  “No, it isn’t a bug.”)  Team members became opponents which created organizational silos and mistrust.  Once the underlying issue was realized, the process was changed and a new Tracker category was created separate from “bug” or “enhancement” to denote a design flaw or spec bug.  This allowed the Technical Support team to push that the issue was perceived to be a bug in the client’s eyes and thus get the problem resolved in a maintenance release rather than wait for the yearly enhancement releases. 

But correctly removed the “blame” from the development organization since the issue wasn’t caused by a coding or process issue like a real bug would be and the correct metric was then being used to measure developer performance.  The finger-pointing and arguments ceased, silo walls came down, and the product organizations coalesced into a supportive, cohesive team.

It’s easy to maintain status quo – to march along without noticing the slow and gradual deterioration of quality and effective processes.  But by stepping back periodically and reviewing key metrics, teams can make sure they are working effectively and efficiently.

PS:  Make sure you have measurable metrics – use Tracker to track Calls, Bugs, Enhancement requests and more.  For at-your-fingertips metrics for future use.