Select Page

The ReleaseTEAM Blog:

Here's what you need to know...

DevOps Metrics: Measure Your DevOps Results

DevOps can help improve business culture, speed up time to market, reduce errors, and more. This year make sure you’re tracking key performance indicators (KPIs) to demonstrate how DevOps is improving business outcomes and where additional opportunities to eliminate bottlenecks exist.

Determine Your Baseline

Before implementing a new tool or process, first understand how well your development and deployment processes are going today. You can identify bottlenecks with a value stream map. Take this time to document how long development and deployment processes are taking, how many defects are detected before and after deployment, and whether the app’s uptime and performance are meeting expectations. Collecting this baseline now allows you to compare with KPIs later to see if the change achieved the desired outcome.

If you have existing KPIs that your organization is measuring, you might continue to collect those before and after making changes to your development and deployment processes.

"Value" Metrics vs "Counting" Metrics

In a 2019 Forrester Report on DevOps Quality Metrics That Matter, Forrester interviewed over 600 enterprise leaders to understand what metrics mature DevOps practices measured compared to lower performing practices. Although the report reviews 75 different DevOps metrics, not all KPIs provide value.

As an example, the report shows the difference between measuring the total number of tests executed versus measuring the test case coverage. The first metric is a "counting" metric and can be easily manipulated by adding redundant passing tests focused on a narrow feature set, leaving large portion of the application untested. The second metric measures how well tests cover the application’s functional requirements. This is a far more valuable KPI to measure if your goal is to reduce defects in releases and improve user experience.

DevOps KPIs to Consider

Over the next few months we will take a closer look at specific KPIs across each phase of the software development lifecycle (SDLC). We work with customers who adopt DevOps and DevOps tools to reduce software release cycles, improve application quality through automation, and save both time and effort.

 
DevOps Metrics
Metrics to Measure Software Release Cycles
Feature Lead TimeHow long does it take to incorporate new features, from idea to release?
Deployment FrequencyHow often do you deploy new versions?
Deployment TimeHow long does it take to roll out a new release to users?
Change Failure RateCalculated by dividing the number of failed deployments by the total number of deployments
Metrics to Measure Application Quality
Automated Test CoverageHow well do tests cover the application requirements, and what percentage of those can be automated?
Defect VolumeHow "buggy" are builds/releases? Are you releasing more quickly but spending more time fixing errors after release?
Mean Time to DetectionWhen are defects identified?
Mean Time to RecoverHow long does it take to fix defective code or patch a released app?
Failed DeploymentsHow successful are deployments?
Security BreachesHow many attempted and successful security breaches are there against your application?
Metrics to Measure Customer Experience
Number of Customer TicketsIf customers are reporting a lot of issues, you may not have good test coverage or quality control
App/System AvailabilityIs the application up and available for users to use, or do deployments and defects disrupt availability?
SLAsAre you meeting all committed customer SLAs, or are you paying penalties?
Customer UsageThe number of customers using your product and how their usage of features and time spent in the app is increasing, decreasing, or remaining stable.
 

A Word of Caution

Choosing the right number and type of metrics to measure is a balancing act between actionable data and overload. Steer clear of vanity metrics used to make teams appear successful but do not provide meaningful, actionable data to continue improving. At best these numbers can be distracting, at worst they can mask real issues and cause teams to miss opportunities.

Meanwhile, reporting on and sending alerts for too many metrics causes "alert fatigue." If your team receives a constant barrage of alerts, they will start ignoring even the more critical ones. Reporting on too few events means a longer time (and more lost work effort) before course-correcting an issue.

Finally, do not use metrics to either reward or punish employees. You’ve recruited intelligent people and they will learn to game the system in their favor and to avoid blame.

At ReleaseTEAM, we know people, processes, and tools work together in a successful DevOps team. Contact us to find out how we can work with your people to establish processes, implement the right tools, and identify which KPIs to measure for your objectives.

Corporate HQ

1400 W 122nd Ave.
Suite 202
Denver, CO 80234
720-887-0489

GSA Schedule:
GS-35F-186BA

Massachusetts

1257 Worcester Rd.
Suite 108
Framingham, MA 01701
866-887-0489

Canada

PMB# 604
1-110 Cumberland St.
Toronto, ON M5R 3V5
866-887-0489