Since the last post was growing too long, I broke it into two parts. Here’s the remainder:
4) Engineering does not provide step-by-step test instructions
I mention this item because it is a required process at my current job that I have recommended be discontinued. It requires each JIRA bug and feature have step-by-step test instructions on how to verify the issue. Step-by-step instructions from engineering to QA defeats the whole purpose of QA. They should have the high level understanding of the feature and test it using their own method, hopefully with the mindset of trying to “break” the feature and find bugs. Step-by-step instructions is not only ineffective, it’s counter-productive. Talking with the QA engineers, they often simply follow the instructions, if it works, close the JIRA ticket and move on. To me, that means the issue was never truly tested, plus the time spent testing was more or less wasted as it is likely that the engineer ran those exact same steps for their testing. Thus, it goes back to relying on QA to have a good understanding of the feature and to be able to test it effectively with the axioms, not with step-by-step instructions.
5) Good balance between automated testing and ad hoc testing
I love automated testing. It’s great for catching regressions and can save time in the long run, but there seems to be a trend toward an over-reliance on automated testing. I recently manually tested a feature that is set up for automated tested. I found over 10 bugs, some of them major. QA had become so reliant on their automated tests that they simply ran the tests, if it passed, they assumed everything was working. QA must perform manual testing and when you have great QA engineers with attention to detail, they will develop some inventive testing methods and find some really subtle but important bugs, bugs that automated testing would never find.
6) Don’t test everything
Some things are not worth testing. Build breakages and minor or cosmetic changes to the source code are just some examples of things that may be logged and fixed in JIRA (or whatever bug tracking software you use) that need not be verified by QA. Basically, use common sense to determine when QA should spend their time testing. Don’t just mindlessly send every ticket to QA for testing. Use common sense so their time is well used.
Overall, it’s all about trusting QA and using common sense. Do not treat QA like monkey robots that simply follow prescribed testing steps. If you do, you’ll soon find your best engineers leaving the company. Trust that they are knowledgeable about the product and will use good testing practices to ensure the product is of good quality when it ships. You can trust them if they are logging lots of bugs and are actively asking questions. If they are silent, something is wrong. Basically, it comes back to the Agile Manifesto principal of People > Process. Process is important but enable and develop your People and that emphasis will be evident in the final product.