Greater Saint Louis Area, Missouri, USA

Yes, Write Frameworks from Scratch

Strategically aligning with our customers needs!

Yes, Write Frameworks from Scratch

So I saw a LinkedIn post the other day saying “Please stop writing frameworks from scratch”. It ended up being a product plug for yet another tool claiming to do everything needed for test automation.

I get why this tool, and dozens of others, keep coming out. I’m sure a lot of it is because people want to make money. I’m also sure that some of it is altruistic. There’s this idea that automation is extremely difficult, and that the very people who need it most–the testers–aren’t tech-savvy enough to do it. They need help.

There are a lot of problems with this approach though:

It reinforces a dangerous idea about the craft

Not everybody in testing is non-technical. There are some, but they don’t represent everybody. Some are very technical. Everybody’s somewhere on the spectrum.

A lot of people don’t pick up on marketing tactics before making a judgment call. When one-stop-shop tools keep coming out, it reinforces a false idea that “testers are non-technical”. Otherwise why would they keep being produced?

It’s… kind of false advertising, a little bit

Let’s say for example, that someone actually did come up with a tool that addressed all the needs for tests that needed to be automated. What do you think would happen?

Would there still be dozens of new tools coming out? Or would that one tool propagate everywhere?

Would any new tools even need to be created anymore? Or would everybody double down and learn how to use that one tool?

What do you see happening right now?

It puts focus on the tool and not the skill

I’ve never seen a classified ad looking for someone skilled in using pliers, cutters, hacksaws, pipe wrenches, fittings, blowtorches and solder. I have seen ads for people looking for plumbers though.

When all-in-one solutions are the norm, then the focus becomes learning how to use the tool. Learning how to problem solve takes 2nd place.

It fosters dependence on single points of failure

Companies, teams and even single people come and go. Given enough time, it always happens. Teams can be disbanded, projects can be sunset and companies can be bought.

If your company is putting faith in one tool always being there for you, you may be in for a rude awakening.


It causes costly replatforming

Eventually, any tool may become the wrong solution for us. Our needs and goals change, and sometimes we need something different.

But if your entire solution is wrapped up in one big tool, instead of multiple smaller ones, then everything has to be replatformed. Very costly.

The Path to Success

All that being said, a way to solve this not only makes us immune/resistant to the above points, but helps to further the testing craft.

  • Demystifying coding: Computer programming isn’t magic. It’s a learnable thing. And there are so many options for languages that more people can pick it up.
  • Break down walls: There’s still this wall between developers and testers, and it seems to be centered on coding ability. Writing frameworks teaches us about coding, and augments the testing craft. Testing + coding = powerful.
  • For Testers: If you know how to test and automate, business are really going to want your skills.
  • For Managers: If word gets out that your company is a place where testers can come to learn how to automate, and not just use a tool, you will get access to some raw talent.
  • More independence: If you have one big tool, that’s a big risk. A group of smaller ones (particularly open source) increases your independence. If one small part isn’t the right solution anymore, change it, update it or remove it.
  • Just enough solution: So many tools oversolve the problem and do way too much for what we need. The more of a thing you have, the more it has to be maintained. Writing our own frameworks allows us to have just enough to solve what we need–no more, no less.

As a tester, I want to challenge you to take steps toward writing your own frameworks. As an automator, I want to let you know it’s not nearly as scary as it seems. Challenging, yes, but we’re testers–that’s what we thrive on. It’s a complex thing, but it can be broken down into simple, solvable parts.

12 Responses

  1. We seem to go through cycles where tools are produced for each item needing automation; then someone consolidates them or builds a framework followed by another wave of creation when new technologies need automation or someones sees a better way.

    However, as stated above, the fundamentals shift much more slowly and are always applicable.

  2. Tom says:

    Agree, but I have some other points to make, so let’s get in.

    1) On “technicalness” – claiming testers are non technical is gross misunderstanding, it’s an unfair opinion, and repeating that coding or programming is what separates techy from non-techy folk and is what needs or can be done to cross that magical demarcation line to the tech paradise sounds puzzling if said by someone pursuing career in QA (in fact, that’s a coder’s typical mantra – testers are non-technical, four legs good…). True story – some coders, oops, programmers, are in fact afraid of a tester’s growing skill set and ability to understand certain “technical” aspects (code, for example) and even discourage a tester from learning more hardcore technical stuff because they just might have to put in more effort in quality on their own part or might make them feel unnecessary and dumpable in some-distance run.

    2) On advertising and co. – well, that’s just selling dreams (has advertising ever been about telling the “truth”?) and is based on the algebra of need – as long as there are willing prospective customers there will be pushers of stuff – and one successful tool will not cause others not to appear, and some may fall for the advertised product, others not, the fact a VAG auto manufacturer makes popular cars does not stop, say, FIAT from making cars too.

    3) On tools and costs and skill – of all these, skill is the most important (remember the one about bad worker and tools?), using the tool still requires knowledge what to use it for and how to use it, the tool will not do the tester”s (or tool operator’s) work, and even the most bestest tool will be worthless in a fool’s hands. As for costs ($$$, €€€, ¥¥¥) – add to this the time needed to learn how to use the tool, and I guess in certain circumstances it is much better to do without a tool for this very reason, i.e. not enough time to learn it (&& I more guess that few people think about time as the related cost). Also, the tool, however “magical” and universal, may be just too big for a company’s needs, but I guess some guys or gals in the right places will not buy a tank to kill a mosquito, if you could put it like that, or may be difficult to configure, the technical support for the tool might be crappy, like you say the tool may be discontinued or its vendor might alter the licence so that it is totally unprofitable for you, or however big, your company would still have to look for or work on features to deal with some specific aspects of the developed software that the big-and-wonderful tool can’t handle. Moreover, sometimes it is just cheaper (money, time) and more convenient (features) for a company to develop their own solution (framework) and I guess such are often preferred due to the nature of the developed software product – as you said, just enough solution (almost like with some heavy and complicated construction machines which are buried at the site they were used or form a part of the finished construction).

    4) On breaking down walls – it’s easier for a non-coding tester to learn coding and automation than for a coder/programmer to try and wear a tester’s hat;

    5) LBNL (last but not least) – an automation tool, however fancy and with whatever promises to lure you into purchasing it with its siren singing, is still a software product that may be not without its own defects and unpredictable behaviours that might weirdly affect the state and functioning of the software you are developing, and this is very frequently on a tight budget that your customer has for the project, so investing in a new-and-fancy-and-shiny tool may be not worth the risk.

  3. @halperinko - Kobi Halperin says:

    While I agree with almost all points raised above – I still disagree with the conclusion…
    There are several points you neglected:
    1. Being able to leverage community experience – when using a tool that others use, you get also fix for what they thought should be improved.
    2. Normally a shared framework is written by more experienced people – who wrote and probably also rewrote 1-2 frameworks… based on their experience with it.

    What I see is people with no experience coming to one company, starting to write a framework, failing, fixing, wasting company time and money (hopefully automation is not abandoned – but it is surely delayed by months),
    moving to next company doing it all over again… (a bit better) –
    Its a major waste.
    @halperinko – Kobi Halperin

    • This might sound strange, but I think these actually reinforce my points. Here’s why:
      1) There’s still a big community of people online who do test automation, and it may even be larger than ones dedicated to specific tools.
      2) Everybody has to get a start somewhere. Experienced framework writers weren’t always experienced.

      While I see how having inexperienced people write frameworks may appear to be a waste, it’s not. They’re definitely learning, and applying what they learn. But if they never start, they’ll never get experience.

      Let me ask you this: What do you think would happen to the testing community if every tester had enough experience to automate just a little bit? And then a follow up question: What’s stopping testers from gaining that experience?

      • Anonymous says:

        Following this logic everyone should write their own tools as well. Why use Selenium when you can become a much better developer trying to write it yourself?

        • Because it’s too granular. It’s the same logic as why we have physical toolboxes with screwdrivers, hammers, pliers and wrenches. Each tool has a distinct purpose. If I offer you a Swiss Army knife, it wouldn’t be as effective. It likely has more solution than you need (nail file, toenail clippers, corkscrew), and probably not enough in the areas you do (hammer, wrench).

          A lot of off-the-shelf tools act like Swiss Army knives. They do a lot, but oftentimes don’t address how to handle [this weird thing that only our app does]. And then a lot of time is spent arguing with the tools and workarounds instead of being productive.

          Selenium has a single purpose and doesn’t try to do everything. Writing our own would be like hand-forging a screwdriver. You can do it, and you’ll learn a lot about metallurgy, but if your goal is to do electrical work it’s probably not a good use of time.

          What I’m suggesting with this post is: Every company has unique challenges, and no one tool or framework solves them all. Instead of relying on them, write just enough solution–get all the right tools in your toolbox–to get you efficient. And don’t worry that your toolbox looks different from another company’s–a carpenter’s and a neurosurgeon’s (hopefully!) do too. If one tool isn’t working well, or at all, swap it out for another. Focus less on the tools themselves, and more on the craft.

      • Anonymous says:

        I’m still not sure why would I want to write my own execution and reporting engine when there are so many available already? Junit, Pytest, PHPtest all have that written already and can integrate with CI too.

        • I wouldn’t want to either. Probably I was unclear: when some people say “framework” they mean “the test automation part”. Others mean “everything: test automation, execution, reporting, scheduling, etc.”. I’m in the 1st group. Problem is, many solutions offer the “everything” option, which I take issue with. I’d much rather have distinct chunks linked together, which sounds like what you prefer too am I right?

      • Anonymous says:

        Could this whole post be a misunderstanding then? Because here you use terms like tool and framework interchangeably while they are clearly different things. Tools just automate a certain thing in a certain way, frameworks collect these things into one base. Did you really mean ‘don’t lock yourself into a single tool’?

        • It seems like semantics to me, because I regularly hear it both ways. But I’m with you on not getting locked into anything–tool, framework, whichever, it should be avoided.

  4. Toine says:

    From the perspective of a tester who is working hard to improve his coding and automation skills, thank you for the excellent post!

  5. I have worked in UFT (aka QTP) for over a decade now. If you were to ask me which framework do I prefer to use there is not one I could name that you would know. Ever heard of SAFFRON? STAF? SIFL? You probably have heard of BPT, but that is less a framework and more
    an abstraction layer. As much as Jenkins could be called a framework.

    In my engagements most of my framework has been reused code designs that were built from scratch in earlier projects.

    I could build a framework from scratch, but in most cases today I just extend, enhance and optimize an existing framework. The exception being if the tests are all record and playback, which is rarely the case. Even in that case I build a framework to enhance the methods for stability.

    My designs are always required to be technically independent, hiding the code. The reality is most clients never assign someone to fill the Test Designer role and I get to be both that and an Automation Engineer. My latest project is the rare exception where two non-technical programmers are designing automation tests using BPT (Business Process Testing) scripted components that my team designs in BPT which has become a surprisingly robust product.

    A framework is an implementation of of several good processes. If your code says: “Browser.Page.OkButton.Click.”
    your results should say “PASS: The Ok button was clicked successfully” or “FAIL: The Ok button could not be clicked because it was disabled – checkout this nifty screen capture.”

    In addition every programmer has a different preferred style and approach to automation. While I prefer function calls and method override in UFT, the SIFL framework was based entirely on custom class designs. It was a bit over my head at the time, but I adapted it’s concepts to add early script termination detection and reporting to my later framework designs.

    End result, I like writing frameworks from scratch. If much of the heavy lifting was already done, I’d prefer to start there and move forward.

Comments are closed.