One interviewee said: “80 points from 100 is the minimum score to achieve a sufficient quality in the project. We’ve gone from at least 20% of projects surpassing this threshold to now 60% to 80% . . . . With the use of UFT we are able to decrease the number of defects during the test process.”

Another interviewee noted: “Now absolutely every defect that’s being found is being reported. I think another thing too is, we have lots of defects in our test environments, but what really matters is if we’re preventing the defects from getting to production. [When a defect gets to production] a worst-case scenario would be something so critical that it needs an immediate fix, meaning we have to pull together a SWAT team, we’ve got to work through the night, we may have to contact the third-party vendor that’s contracted with that application. [When the defect is found earlier] I could be sitting in a bullpen with a developer and I find a bug and I communicate it to them, and they fix it before I can even finish the sentence of saying what the problem is.”

Explained one interviewee, with regard to performance issues: “Without increasing the number of QA engineers, we are able to run higher quality tests, more frequently than before. We have weekly sessions where basically any issues that have gone through into production are discussed. And I can’t remember the last time something performance-related made it onto that list. But it used to be a fairly regular occurrence that we were looking for a performance bottleneck of some sort, and that tends to not be the case now.”

Explained one interviewee: “If you can find the defect on the developer’s machine, that’s like the best case scenario. The next best case scenario is actually when you integrate your code, and then you run some tests in that environment or in a continuous integration environment, but if you can’t then the next best case is to find it in QA. Over the last three years, we have gone from 92% efficiency to 98 point something — 1% basically escapes. So what we are saying is there’s an almost 99% success rate, just to production.”

Forrester defines a “defect” in this instance to be a bug that causes moderate to significant functionality issues or system failures that can lead to impacted user experience or downtime.

The composite organization averages four defects per month for each of its 25 applications tested using Micro Focus. As applications are onboarded, this increases to 100 total defects per month by Year 3.

Prior to Micro Focus, 8% of defects made it into production. With Micro Focus, this decreases by 7 percentage points down to only 1% of defects making it into production.

The cost to fix a defect in production is assumed to be $10,000, while fixing the defect in development or testing is on average 100 times cheaper.

One interviewee said: “Having seen a reduction in the frequency of those incidents [production issues] means that we have more stable applications out in production and a better end user experience.”

“So, for achieving that goal [of reducing downtime], we started to use Micro Focus products. It definitely has decreased, but it wasn’t only because of the Micro Focus products. Because we changed internally in terms of our organization and our standards and our people and culture. During the last five years, it’s at least a five times reduction in downtime.”

Prior to using Micro Focus, 96 defects made it to production each year. With Micro Focus, this number decreases to 12.

On average, 1% of defects result in application downtime.

On average, each outage lasts 2 hours and the cost per hour of downtime for the composite is $500,000. This is an all-inclusive cost that includes not only the cost to resolve the issue but also lost productivity and lost revenue. This cost can vary based on company size, industry, and region, and may be higher than average given the companies in our specific sample. For more information on how the cost of downtime can fluctuate, please reference the Ponemon report on data center outages, which concludes that the average cost per downtime is roughly $500,000 across its sample, verifying the figures from our interviewees.⁴

Micro Focus enables users to give the business more visibility into testing processes, allowing for better decision making.

Interviewed organizations talked about the dashboards and reporting capability in ALM that shows application testing progress which business users can view. One interviewee said, “We can give a clear definition to the business, saying, if you want this change to go in production, it might take a week of testing or three days of testing. So the business is happy because we are giving enough time that’s realistic, and we are putting quality product in production. That was not possible five or six years ago because the tooling was not in place.”

After the Micro Focus investment, interviewed organizations noted that customer satisfaction improved.

One organization’s customer satisfaction score improved by 20%, half of which the organization attributed to the use of Micro Focus specifically to deliver higher quality applications and more frequent releases. Another interviewee commented on more stable applications resulting in a better end user experience. Another interviewee went on: “We do performance testing on all of these applications before we put them to production, and if they don’t match the baselines, we actually talk to the developers, and find out why a particular area is slowed down. We go back. We do some root cause analysis and then fix it, and until it’s fixed, we don’t put it into production. Before, the performance was affected, nobody knew why, the customers were complaining — we all put so much importance on the dollar value, that’s really important, but just to see that the customers now have this trust in IT, that we can deliver a quality product, is crucial for us.”

Some interviewed organizations also noted compliance benefits.

One organization had an internal auditing department that needed details about testing processes as part of government compliance. It would have been a manual process to gather that data before, but with Micro Focus the organization can easily show detailed reporting in ALM. Another organization similarly commented that being able to start UFT tests from ALM and then store the results back into ALM gives the organization auditability that it did not have before.

Several interviewees have started or plan on moving toward parallel test execution to drive even more efficiency.

One interviewee explained: “What we’re doing is we’re using tools like [open source automation servers] to say, ‘I want to run a series of UFT tests on four or five machines all at the same time,’ which frees up the tester to be working on tomorrow’s test or analyze test results or do something else.” Another interviewee said: “In the last couple of years we looked at parallel test execution as well. We said, instead of having one user — one person equal to one robot — doing this work, how about we configure our framework so that we can run multiple threads, multiple tests at the same time. It’s amazing — once we were able to run things in parallel, now we were able to cover more areas but also we could go deeper. So, before, we were looking at only the top 10 or top 15 states, now we are looking at all states.”

Some interviewees are considering additional investments in Micro Focus tools, including cloud capabilities.

While most interviewed organizations are using on-premises versions of Micro Focus tools, some interviewees are moving forward with a more cloud- focused strategy that may lead them to invest in cloud options like StormRunner Load or StormRunner Functional. Moving to Micro Focus’ software-as-a-service (SaaS) solutions could reduce the burden on infrastructure and administration costs associated with on-premises solutions. Other interviewees have been upgrading their Micro Focus investment through the years, for example starting with LoadRunner and then moving to Performance Center, or starting with ALM QC and then planning on upgrading to ALM Octane for additional functionality.