Automated Testing vs. Flight Recorder


In the past decay "Automated Testing" became a buzz word and a "must do" function in SQA departments.

Most of the SQA resources go to building scripts for "Automated Testing".

SQA schools spend most of there time teaching the students how to build and use "Automated Scripts".

This became the "SQA concept".


I dare to challenge this concept.


Once the scripts are built (and debugged), they are executed over and over again to find a "software defect".

 In some (rare) cases they find real problems, and in other cases, they fail to do the right check, or the "defect" they find is a legitimate change in the software that needs an update to the "automated Script".


Did anybody check the ROI on these types of tests?


In most cases, the scripts are built to check the main paths of the software, which are any how tested carefully by the developers and therefore do not have defects. The alternate paths and the less important areas are neglected.


When you send most of your resource on "Automated Testing" you miss the "Not obvious defects" that users will probably encounter in the field.


So how should we test software and new versions?


SQA team leaders should build "presentations" of many paths in the software using "Flight Recorder", and give them to the QA technicians. Each technician should learn his paths, and then try to do the same (without looking at it).

A flight-recorder should be installed on each of the technician's computer, recording everything he does.

When the technician encounters a defect, he should extract the relevant part from the recorder and attach it to the defect's report.

This should be done even in the case that there is no defect, but the technician doesn't understand what to do, because it means that the end user will have the same problem.


I think that this technique is much better, because it covers much more paths (it is much easier to create a presentation than "Automated Scripts", and because it finds heuristics defects that are due to user's errors (wrong way of doing things). You can view a presentation of a Flight-Recorder, and download free for a trial period