I’d asked Araceli D’Apolito, one of our leading testing experts at Velocity Partners, to weigh-in on an article I had written about the Agile Test Automation Pyramid. I’ve included a snippet of her feedback below:

And this is to consider what will be the principal objective of automating the testing. As I saw in the diverse projects/clients that we work with, they focus automation in different directions. To mention some as examples:

• automate regression testing efforts as much as they can (one case with 95% of critical functionality covered);
• or sanity/health check tests;
• or UI and Business logic validations;
• backend code regression;
• or multi-browser/multiplatform testing, among others.

(I will not go deeply into the execution strategy itself, because it would be too extensive) So considering these, the purpose of investing in an automation effort, depends on what would be the expected benefit. Such as increased productivity, improvement in quality, doing more with less, etc.

Araceli brings up a very good point, that is—what is the “Business Case” for agile test automation. What are the overriding goals? In other words, what is the “business why” behind it? It’s often important to establish this because the Product Owner within most agile teams is chartered with making investment decisions within their Product Backlogs and therefor their products.

We often grouse or complain that they are not investing enough in test automation…in lieu of features (sometimes what I call feature-itis). But how can we complain if we haven’t established a strategy, communicated the need, and clearly agreed on the value proposition of test automation?

Unfortunately, I don’t think we can.

Where to Begin?

If you read any of my blogs, you know that I often go back to our traditional approaches to set the stage for each discussion. I’m not trying to imply that Waterfall approaches are “bad” and Agile approaches are “good”. In fact, I don’t think of it in those terms. Instead, I’m simply trying to leverage historical context and draw an evolutionary line towards the agile approaches.

Historically, the “Business Case” for test automation was typically three-fold:

1. Reduce Costs: probably the number one motivation is reducing the costs (people, time) associated with manual testing.
2. Speed up Testing: often we envision that the testing will occur “over night” or “over weekends” so that it’s faster and doesn’t interrupt the projects’ flow.
3. Reliability: repeated manual execution of test cases can often be boring and over time lead to mistakes and errors.

And the ROI of automation was mostly driven from the first two areas, and to be brutally honest, often led to a reduction in tester headcount over time. I personally found this to be shortsighted, but that was the reality.

Fast-forward, What is Agile’s Automation Business Case?

To start, I’d say that the traditional three business case drivers are still “in play” for agile test automation. However, I think we move them to second or third order considerations. Certainly they don’t drive the majority of the decision-making behind the automation.

Prime Directives

In agile test automation efforts, I’d like to recommend or substitute three prime directives behind the Business Case and your test automation strategies. They are:

1. Test automation is a team decision as to how much. They need to be empowered to invest in automation as a means of supporting continuous feedback, nimbleness in implementing new features and extending existing features, and supporting their go-forward velocity. Yes, the Product Owner (the “business”) plays a role in this investment, BUT they can’t ignore automation.
2. Test automation is NOT a cost savings play. The ROI cannot be based on team reduction or cycle time reduction or any time-based cost savings. This places the emphasis on Time over Quality. Any time the team realizes due to automation should be reinvested in delivering more content, improving their tools and processes, refactoring, learning, and perhaps defect repair. In other words, reinvest in the Team and the Product.
3. Automation is not a “choice” in Agile contexts. It’s an “imperative”. You may be in a context with no automation, so the going may be tough at first. But that you invest in test automation is a given. How you strategically attack it is the variable, as is how much you invest over time. Note: Salesforce.com has talked about an interesting guideline they have regarding automation investments, I’ll share it at the end.

So, What should Agile Test Automation focus on?

Back to Araceli’s points, and she’s certainly not wrong, but I’d argue that the agile teams should focus on everything she mentioned and possibly more. Not in a wasteful or fearful way, but as a means of supporting the prime directives above.

The teams need to decide, given the project and product goals, what is the RIGHT level of automation for them to do a great job. A job that includes providing outstanding feedback to their customer, generating a high-quality product, and keeping “team health” and engagement at a high level. Once they’ve rationalized this with the Product Owner, then they are trusted to implement their strategy without the weight or fear that the traditional business case induced.

Wrapping Up

I can hear what you’re saying. That’s no “Business Case”; where’s the clear ROI? My management and leadership would never go for that! It’s way too fuzzy and team focused. And what if the team over-focuses on automation? We need to get features out the door to be competitive.

Well, I understand all of that. What I’m saying is establish some goals with the team and then Trust Them to create a balanced strategy that includes an appropriate amount of Agile Test Automation. Surely you’ll get the chance to see how things are unfolding on a sprint-by-sprint basis. So you can ask questions, stay engaged, and understand how well they’re balancing their investment.

But keep in mind that the ROI is not time or speed or savings based. The ROI is faster insights & adjustments, investment in my teams learning, and improved product quality; which includes building what the customer needs. As a stakeholder, I’ll take that over trivial time or cost trade-offs any day.

Stay agile my friends,

Bob.

BTW: I’ve been talking about the shift in Automation ROI for quite awhile. I’ve got a 60 minute presentation that I’ve delivered for the SQE – STAR conference, as well as others. I would be happy to deliver this to any of our Velocity Partners clients. It’s a little over the top, but it makes the important point in the shift away from traditional automation goals and value measures. Here’s a as well.

Sidebar

As I promised above, I wanted to share a story from Salesforce.com. It seems they have historically had a constraint for teams that don’t minimally have 70% automated coverage of their individual codebase. If they don’t, then the Product Owner will take a 20% reduction from the Backlog and invest it in automation development sprint-over-sprint until they reach 70% coverage. That would be in addition to any automation/refactoring investment plans already in the Backlog.

This position is not coming from the team. Instead, it’s coming from Executive & Senior management. It seems they “get” the business case and ROI for the investment in automation. They don’t need to be convinced. I’ve used this as a critical example of how senior leadership can influence the overall quality of their teams.

Salesforce has been a fairly wide presenter at the Agile Conferences in 2007, 2010, and 2013 as well as other venues. I believe I first heard of this commitment to automation in 2009-2010.

Some follow-up references: