Alan Page is known as one of the authors of “How We Test Software at Microsoft”. But there is another good book and it’s called “The ‘A’ Word”. You can buy it on LeanPub.
The book is about automation in testing, but not how to do it — it’s about how to think about it. It’s short, just 58 pages, but very dense with ideas and Alan’s opinions.
As I am not qualified to give an opinion on automation topics (because I don’t have much experience with it), I’ve just gathered some notes for future referencing. Sections are divided by chapters.
Sometimes, testers use programming skills to help their testing. Sometimes, that code automates some application functionality. That’s it.
#Testing: failing to succeed
There is a very famous concept called “Orders of Ignorance” introduced by Phillip Glen Armour (more here). Chapter’s idea is that mostly tests are done on 0OI level, but we should never forget about 2OI test. 0OI is a lack of ignorance (I know) and 2OI is a lack of awareness (I don’t know what I don’t know).
0OI tests are knowledge-proving tests, while 2OI tests are knowledge-acquiring tests.
#The robots are taking over
Humans fail when they don’t use automation to solve problems impossible or impractical for manual efforts.
Automation fails when it tries to do or verify something that’s more suited for a human evaluation.
#To automate …?
Good testers test first — or at the very least they think of tests first.
Great testers first think about how they’re going to approach a testing problem, then figure out what’s suitable for automation, and what’s not suitable.
You should automate 100% of the tests that should be automated
Alan’s heuristic when to automate: “I’m Bored”
#The coding tester
Summary:
- the role of a coder-tester is not to automate everything
- testers do not need to have a computer science
- testers do not need to be able to program
- programming knowledge does not destroy “a proper tester angle”
- background similar to customer’s does not make you a customer
#GUI shmooey
For 95% of all software applications, directly automating the GUI is a waste of time.
For the record, I typed 99% above first, then chickened out.
#Design for GUI automation
Alan’s main points for disliking GUI automation:
- It’s (typically) fragile — tests tend to break / stop working / work unsuccessfully often
- It rarely lasts through multiple versions of a project (another aspect of fragility)
- It’s freakin’ hard to automate UI (and keep track of state, verify, etc.)
- Available tools are weak to moderate (this is arguable, depending on what you want to do with the tools — I’m particularly pleased, for example, with what good testers are able to do with selenium and web driver).
I love GUI automation that can automatically explore variations of a GUI based task flow.
I like GUI automation is in stress or performance issues.
#It’s (probably) a design problem
- Record & Playback automation is a non-starter
- Basic verification that would be hit by anyone walking through the basics of the application isn’t worth automation
- Tests that do exactly the same thing every time are not valuable
- Always think forward
- Plan for failure and ensure that all test failures tell you exactly what is wrong
- Tests should be reliable
- There is always a better alternative to Sleep statements
- UI is fragile, its testability should designed
#In the middle
Alan’s brainstorming technique: first spend a reasonable amount of time focusing on the extremes — because often, some great ideas for “the middle” comes out of that brainstorming.
#Test design for automation
The first step — and most important — is to think how you’re going to test.
From that initial test design effort, you can deduce what aspects of testing could be accomplished more efficiently with automation (and without).
#Orchestrating test automation
Designing good tests is one of the hardest tasks in software development.
#LOL — UR AUTOMASHUN SUCKZ!
Your tests don’t suck:
- when you treat their code like a production code
- core reviews
- static analysis
- running with the debugger to ensure they are doing what you think they are
- trust: if a test fails, it’s a product bug, not a test bug
- when they execute automatically
- when failures are handled automatically
- bugs are entered automatically — including logs, call stacks, screen shots, trace information, and other relevant info
- when bug is fixed, it’s checked automatically
- generation of “Test Result Report”
#Musings on test design
Some tests can only be run via some sort of test automation.
Some tests can only be done via human interaction.
You can’t effectively think about automated testing separately from human testing.
In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.
There is only testing
#Beyond regression tests & testing with code
Useful tests are tests that provide new information.
An automation strategy that only performs regression testing is short-sighted and incomplete.
How to make test useful:
- model-based testing
- introducing some randomness
- data driven testing
- scaled fault injection
- fuzzing
#More on test design
Test Design ideas are endless.
To be a good test designer (and tester), you need a lot of testing ideas, and you need to know how and when to apply them.