6 tips to avoid automation disasters

Senior software engineer Benjamin Willenbring was excited about automating software testing at Autodesk in 2015. But the excitement didn’t last long because the automation team didn’t communicate much with other departments, and when the tests made it to production, they didn’t deliver the results everyone had hoped for.

“We’re talking about non-deterministic failures, but we don’t have a lot of confidence in the test,” Willenbring said. “It was very difficult to start running the test because there were no reference cases, there was a lot of communication, and there was a lot of paperwork that I didn’t really know how to handle.”

[[316367]]

Automated software could make Willenbring's job easier. Over the next few years, these issues occupied much of his attention.

Willenbring's experience is not uncommon. As automation rapidly becomes more common through the application of IT technology, it provides some valuable experiences and lessons.

From automated workflows in DevOps to robotic process automation (RPA), automated processes are designed to reduce tedious work and free up skilled employees to perform higher-level tasks. However, a flawed workplace or an imperfectly deployed project can turn the dream of automation into a nightmare. Some IT experts have explained the automation failures they have heard of or encountered, and shared six tips to help companies avoid the fate of automation plans.

1. Automation is part of everyone’s job

Willenbring's automated testing faced a key problem: the only people who understood automated testing were the people who built automated testing software, but they didn't usually work at the corporate headquarters.

“One of the difficulties with a testing framework is that it doesn’t really provide good feedback when something fails,” Willenbring said. “If it fails, the first thing you need to do is contact the test owner through Slack, the collaboration app, and ask why did it fail? Then, by manually rerunning the test, you need to communicate what you saw or what would have happened for some particular version of the framework.”

Robert Haas, global DevOps product manager at DXC Technology, said its testing framework violates two basic rules that every automation regime should follow. The first is that automation code must be documented. "Whether you use modern methods like documentation or annotating Visio diagrams, it's easier to troubleshoot problems if there's some documentation that describes what was done in the first place," Haas said.

Without documentation, automated tests are incomprehensible to Willenbring's team. They can't understand the test results and lose confidence in the tests and the team that created them. "Sometimes developers may not care, thinking it can't be a real failure," Willenbring said. "Sometimes there are real failures in the test framework. However, the key issue is a lack of trust in the tests and the results."

Another piece of advice from Haas: "Prioritize the activities that need to be automated based on the business value that needs to be delivered." But because the testing team at Autodesk is so separate from the development team, Willenbring found that many of the areas in the code base he chose defied common sense. "You need to have subject matter experts or people who understand what's important choose what to test," he says. "If you're just looking through a bunch of bug logs, there's no guarantee that the number of tests you're testing is going to accurately reflect anything meaningful."

A new director of engineering helped Willenbring and his team break out of this vicious cycle. The director of engineering required that quality testing become everyone’s job, which included test automation and eliminated centralized testing, making individual departments responsible for writing test code for their work. Willenbring and his team can now tailor testing to their needs and integrate the process into their daily lives. In the end, he says, “there had to be a zero-tolerance policy for the ‘that’s not my job’ attitude.”

Willenbring was inspired by this and has written a detailed account of his automation journey, which includes in-depth material that people may be interested in, as well as details on his work with the Selenium and Cypress testing frameworks.

2. Be prepared for complexity

Security and compliance automation provider Tripwire recently noticed something strange about the automation solution at one of its large financial clients.

“This company deployed an automated solution, but it took so long that we kept wondering why we didn’t see a significant increase in license utilization, because they should have deployed it quickly,” said Irfahn Khimji, Tripwire’s Canada country manager.

It turns out that the ideal of a largely automated continuous integration (CI)/continuous deployment (CD) pipeline is being realized in the work of various business units in financial organizations, each of which relies on its own custom set of software components.

“They were trying to work with over 30 applications, and there were some challenges with the commoditization of those applications — modules that were needed in various libraries and things like that,” Khimji said. “Adapting the pipeline to handle all those different variations really slowed down the automation process because it wasn’t just a simple deployment, it was making sure all these different technologies worked together.”

There is no silver bullet to solve this problem, but it is important to understand that as the number of components in an automated process grows, the amount of plumbing required to connect those components will grow exponentially and become more complex. This complexity will require more time and resources when transitioning to an automated process.

Another factor that adds to this complexity is not only the number of components to be connected, but also where they come from. Most pipelines or RPA-driven environments contain a heterogeneous mix of internal and third-party components, which can be difficult to troubleshoot if problems arise.

“Make sure all components of your CI/CD pipeline or automation process have maintenance contracts with the software provider,” DXC Technology’s Haas said. “If there are open source components, do a risk assessment to determine if you should consider using a hosted version of the product rather than relying on the open source community’s web-based support.”

3. Beware of the “black box”

Financial institutions have been among those looking to deploy robotic process automation (RPA) and chatbots, but Muddu Sudhakar, CEO and co-founder of Aisera, warns that he often sees a situation in these environments where processes are treated as a single, monolithic functional unit, becoming a “black box” whose inner workings are difficult to troubleshoot.

“In the overall structure, the customer will check the status of their account, if they want to withdraw and transfer the money, it’s all in one step,” he said. “If something goes wrong, and there is no audited step monitoring, then the only way to get your money back in a catastrophic failure is to call customer service and maybe go to the bank in person to pick up the money.”

For Sudhakar, such designs are often a sign of early automation adoption. As long as everything goes as planned, such projects can produce good results. But if not, companies often have to come back and tear these black boxes apart. It’s better to avoid them in the first place.

“Break down each process into building blocks, where each building block is auditable and monitorable,” he said.

4. Establish checks and balances

Ari Meisel, founder of Less Doing and a productivity coach with a special interest in automation, often builds automation software to make life easier. But he learned a hard lesson about automation software when he was trying to avoid parking tickets.

Meisel owns a pickup truck, technically a commercial vehicle, and in New York City, where he lives, it’s easy for owners of such vehicles to dispute a ticket: “You can send a letter to the local finance department and say, ‘I was making a delivery.’ ”

He admitted that his practice was not entirely legal: He created an automated software system that randomly generated invoices and sent the invoices and letters to the local finance department to argue for the cancellation of the tickets. He said this method worked at first, but it became a problem because the system sent more than 100 identical letters and invoices. The finance department threatened to send me to jail, so I found a lawyer to defend me, and it ended up costing me $36,000.

What his automated processing software needed, he realized, was a poka-yoke. The term means "error-proofing," and in Toyota's automotive production system, a process step is split into two steps, with the second step dependent on the first. It seems paradoxical that adding steps increases efficiency, but errors can affect the line if they are not eliminated. In his plan to dodge the ticket, Meisel said, the automated program would compare each invoice with the previously sent invoice to see if the content is the same. That's a very easy thing to do."

This expands on Aisera’s Sudhakar’s advice: Each automated step should not only be auditable, but also constantly audited by other automated steps. In the AIOps world, automation platforms take the IT management burden off engineers. “I call it the NASA approach,” Sudhakar said. “NASA has to assume that failures are going to happen, and they have to have an AIOps solution with checks and balances in place.”

But unless you experience automation fail firsthand, it can be hard to see its value, Meisel says: “People fail 90% of the time. They say they’re not going to adopt this automation that they’ll never really benefit from, and then they discover that they do need it.”

5. Don’t neglect safety

There's a secret to automated CI/CD pipelines: Many of the pipelines that address security requirements were originally launched as shadow IT. "Developers want to get on with the next iteration (of developing code)," said Tripwire's Khimji. "So when the strict IT and security processes get in the way, they think, 'I can just build these images in the cloud and I can get around them,'" he said.

That doesn't mean these pipelines are inherently insecure, but it's worth investigating which security practices are being abandoned in favor of automation efficiency. Also, it's important to remember that any automated process represents another vector for attack. Processes that operate autonomously may have elevated privileges, making them tempting targets. Aisera's Sudhakar said he's aware of examples of what he calls "black swans": a competitor used a DevOps pipeline within a company to inject malware into the code base, allowing it to spread into production and rendering its systems inoperable.

6. Don’t automate on the cheap

Ajeet Dhaliwal, lead developer and founder of Tesults, found that misinformation can come from undue hype. Many organizations have a vision of “automated testing” that is far from best practices. This is especially true in smaller organizations without a technical background. Since they understand manual testing, it logically follows that automated testing should just be a version of manual testing that can be performed without human intervention.

As a result, Dhaliwal said, “traditional quality assurance (QA) testers who don’t have a software development background are encouraged to perform manual testing to automate testing. Sometimes this means using tools that only record manual user interface (UI) tests to be repeated later. The robustness and flexibility of these approaches don’t match what developers can achieve in terms of automation.”

“Automation test developers also need to be software engineers,” Dhaliwal added. “It’s OK to have some junior developers involved, but these companies need some experienced developers to lead this effort, and they don’t have that.”

And, as Autodesk's Willenbring said, software engineers also need to understand how to build automated processes. "That's just one of the skills that developers will have," he said. "When you can use those skills, hopefully those skills will help your automation projects be successful."