What Types of Bugs Aren’t You Catching?

Every time I meet a new potential client, I ask this question: “What types of bugs aren’t you catching?”

It’s never meant to be snarky. The point of the question is to make sure the client knows there’s a difference between the “what” of automated testing, and the “how”.

The “what” is the thing being tested: A webpage. Some webservices. A database. But the “how” is what types of tests are being run: Security. Stress tests. Fuzz tests. Model-based. Load and performance. Functional.

Functional testing is what many companies consider when they invest in a test automation solution:

I perform actions A -> B -> C and I’m expecting a result of X

But the problem is: it’s difficult to find all–or even most–of the bugs, when that’s the only type of testing being done.

What happens if there’s a really complex dataset that causes the bug? Do you write, run and maintain thousands of functional tests?

Or what if the bug is more to do with user experience? Does writing tests that make sure every element has the right location/color/size make financial sense for your company?

It’s like fishing. When you want to catch fish, you have to consider a few things:

• What type of bait do I use to catch that fish?
• What equipment should I bring to reel that fish in?
• Where are the types of fish that I want?
• Is it the right time of year for that kind of fish?

Notice that for each, the kind of fish is key. Fishing for bluegill, for example, has wildly different answers to these questions than, say, fishing for marlin.

Fishing for bugs is the same way. The right automation solution will catch the kinds of bugs that are impacting your company today, with a custom fit that grows with you as your needs change.

If you’re not catching the bugs you need to catch, let’s talk. Schedule your free 15-minute discovery call here.