The ReleaseTEAM Blog: Here's what you need to know...
DevOps Metrics: Measure Your DevOps Results
DevOps can help improve business culture, speed up time to market, reduce errors, and more. This year make sure you’re tracking key performance indicators (KPIs) to demonstrate how DevOps is improving business outcomes and where additional opportunities to eliminate bottlenecks exist.
Determine Your Baseline
Before implementing a new tool or process, first understand how well your development and deployment processes are going today. You can identify bottlenecks with a value stream map. Take this time to document how long development and deployment processes are taking, how many defects are detected before and after deployment, and whether the app’s uptime and performance are meeting expectations. Collecting this baseline now allows you to compare with KPIs later to see if the change achieved the desired outcome.
If you have existing KPIs that your organization is measuring, you might continue to collect those before and after making changes to your development and deployment processes.
"Value" Metrics vs "Counting" Metrics
In a 2019 Forrester Report on DevOps Quality Metrics That Matter, Forrester interviewed over 600 enterprise leaders to understand what metrics mature DevOps practices measured compared to lower performing practices. Although the report reviews 75 different DevOps metrics, not all KPIs provide value.
As an example, the report shows the difference between measuring the total number of tests executed versus measuring the test case coverage. The first metric is a "counting" metric and can be easily manipulated by adding redundant passing tests focused on a narrow feature set, leaving large portion of the application untested. The second metric measures how well tests cover the application’s functional requirements. This is a far more valuable KPI to measure if your goal is to reduce defects in releases and improve user experience.
DevOps KPIs to Consider
Over the next few months we will take a closer look at specific KPIs across each phase of the software development lifecycle (SDLC). We work with customers who adopt DevOps and DevOps tools to reduce software release cycles, improve application quality through automation, and save both time and effort.
|Metrics to Measure Software Release Cycles|
|Feature Lead Time||How long does it take to incorporate new features, from idea to release?|
|Deployment Frequency||How often do you deploy new versions?|
|Deployment Time||How long does it take to roll out a new release to users?|
|Change Failure Rate||Calculated by dividing the number of failed deployments by the total number of deployments|
|Metrics to Measure Application Quality|
|Automated Test Coverage||How well do tests cover the application requirements, and what percentage of those can be automated?|
|Defect Volume||How "buggy" are builds/releases? Are you releasing more quickly but spending more time fixing errors after release?|
|Mean Time to Detection||When are defects identified?|
|Mean Time to Recover||How long does it take to fix defective code or patch a released app?|
|Failed Deployments||How successful are deployments?|
|Security Breaches||How many attempted and successful security breaches are there against your application?|
|Metrics to Measure Customer Experience|
|Number of Customer Tickets||If customers are reporting a lot of issues, you may not have good test coverage or quality control|
|App/System Availability||Is the application up and available for users to use, or do deployments and defects disrupt availability?|
|SLAs||Are you meeting all committed customer SLAs, or are you paying penalties?|
|Customer Usage||The number of customers using your product and how their usage of features and time spent in the app is increasing, decreasing, or remaining stable.|
A Word of Caution
Choosing the right number and type of metrics to measure is a balancing act between actionable data and overload. Steer clear of vanity metrics used to make teams appear successful but do not provide meaningful, actionable data to continue improving. At best these numbers can be distracting, at worst they can mask real issues and cause teams to miss opportunities.
Meanwhile, reporting on and sending alerts for too many metrics causes "alert fatigue." If your team receives a constant barrage of alerts, they will start ignoring even the more critical ones. Reporting on too few events means a longer time (and more lost work effort) before course-correcting an issue.
Finally, do not use metrics to either reward or punish employees. You’ve recruited intelligent people and they will learn to game the system in their favor and to avoid blame.
At ReleaseTEAM, we know people, processes, and tools work together in a successful DevOps team. Contact us to find out how we can work with your people to establish processes, implement the right tools, and identify which KPIs to measure for your objectives.