Redefining Exploratory Testing
Can an exploratory test become scripted? Is there really such a thing as a truly exploratory test? Or is it in fact just a scaled-down scripted test? With Exploratory tests becoming increasingly scripted in nature, I feel this is the right time to redefine the meaning of exploratory test and draw a line between exploratory and scripted test.
As any reference book or present-day tester would claim, the Methodology of modern software testing could then be divided into mainly two dominant categories: Exploratory and Scripted Testing. But what then is the difference between these two methodologies? After all, they share the same purpose: to find bugs and check functionality against requirements. Isn't it?
Some might suggest that the main difference is that Scripted testing is much more well-documented than Exploratory testing and thus weighs heavier as a formal proof of the code having been thoroughly tested. Others might say that it's really about automation. With Scripted tests, the code could easily be performance tested through automated tests that runs thousands of iterations and logs the program's every behaviour. Yet another group might claim that the Exploratory test approach applies solely to GUI-testing and should not be used for anything else. According to James Bach, the plainest definition of exploratory testing is test design and test execution at the same time.
Let's take a closer look
Obviously, there is a great deal of confusion about both the structure(1) and the purpose(2) of Exploratory testing as well as when it should be conducted(3). A further examination of these three issues is therefore warranted:
1. The Structure of Exploratory Testing.
Unless the program to be tested is very small, most Exploratory tests have a written template, a guide to follow when performing the test. Alternatively the Tester is extremely familiar with the product and can go through its different functions from the top of their head. This is not a recommendable situation though as there is no contingency plan for the company if the tester suddenly becomes unavailable. Also; we're only human. On a typical Monday morning, anyone could forget an essential part of the routine even if it has been performed a trillion times before.
So let's assume that the Exploratory test is in fact documented and that copies of that document are being used every time the product is tested. In that case it is also likely that the tester, after completion of the tests, signs off his task with a reference to that very same document. The document hence becomes formally connected with the performed test and its certificate of adequate test coverage and robustness. Doesn't that sound awfully familiar to scripted testing?
2. Purpose of Exploratory Testing.
The purpose of any testing (including Exploratory testing) is to find bugs and check functionality against requirements. But if we again try to scrutinize the said difference between the two testing paradigms mentioned above, we'll find that Exploratory testing mainly focuses of finding bugs that would appear when the program is used correctly, that is; according to the specifications. If, for example, a certain function requires the in-values of: "$100 to 1000" and a specific time-duration expressed in "full weeks" to calculate a ROI, then the Exploratory test would try the function with exactly these figures, whereas a Scripted test would also try to run the function with maybe "-£500" and "44.7 years" as typical in-values to see how the software behaved with anomalies.
However, even though the scope of the two methodologies could be different, a scripted test could easily be written to cover exactly the same functions, with the same depth, as any typical Exploratory test. All you really have to do is to use the same type of user-behaviour and in-data as stipulated in the requirements. Again, a blurring of the definition.
3. When Exploratory testing should be conducted.
After having just received the first Alpha version of a new product, a stressed Test Manager might say: "Give it a quick try with a Exploratory test now and then again just before we release", meaning that everything else in between should be Scripted testing. This strategy might be reasonable since there often is a need of quickly showing the developers what faulting functions needs to be addressed first after the initial release. There is of course also the necessity of once again going over the whole system just prior to the final release to make sure that all the latest bugfixes hasn't caused new ones. Exploratory testing, being rather cost-effective and rapid, might seem a good option in those two cases.
However, with modern software testing becoming introduced on all levels of the development cycle, the initial "quick look" should not be necessary. The test team should have had enough time to write proper test cases and suites while the program was being coded had they already been present at the planning stages. A final overhaul of the whole system could perhaps be more justifiable, especially if there is very little time left before the deadline. However, this would probably prove to be unsatisfactory in regards of quality, since if there is anything that the program should really be tested for at that stage, it should be its robustness and performance, not just basic functionality.
As any reference book or present-day tester would claim, the Methodology of modern software testing could then be divided into mainly two dominant categories: Exploratory and Scripted Testing. But what then is the difference between these two methodologies? After all, they share the same purpose: to find bugs and check functionality against requirements. Isn't it?
Some might suggest that the main difference is that Scripted testing is much more well-documented than Exploratory testing and thus weighs heavier as a formal proof of the code having been thoroughly tested. Others might say that it's really about automation. With Scripted tests, the code could easily be performance tested through automated tests that runs thousands of iterations and logs the program's every behaviour. Yet another group might claim that the Exploratory test approach applies solely to GUI-testing and should not be used for anything else. According to James Bach, the plainest definition of exploratory testing is test design and test execution at the same time.
Let's take a closer look
Obviously, there is a great deal of confusion about both the structure(1) and the purpose(2) of Exploratory testing as well as when it should be conducted(3). A further examination of these three issues is therefore warranted:
1. The Structure of Exploratory Testing.
Unless the program to be tested is very small, most Exploratory tests have a written template, a guide to follow when performing the test. Alternatively the Tester is extremely familiar with the product and can go through its different functions from the top of their head. This is not a recommendable situation though as there is no contingency plan for the company if the tester suddenly becomes unavailable. Also; we're only human. On a typical Monday morning, anyone could forget an essential part of the routine even if it has been performed a trillion times before.
So let's assume that the Exploratory test is in fact documented and that copies of that document are being used every time the product is tested. In that case it is also likely that the tester, after completion of the tests, signs off his task with a reference to that very same document. The document hence becomes formally connected with the performed test and its certificate of adequate test coverage and robustness. Doesn't that sound awfully familiar to scripted testing?
2. Purpose of Exploratory Testing.
The purpose of any testing (including Exploratory testing) is to find bugs and check functionality against requirements. But if we again try to scrutinize the said difference between the two testing paradigms mentioned above, we'll find that Exploratory testing mainly focuses of finding bugs that would appear when the program is used correctly, that is; according to the specifications. If, for example, a certain function requires the in-values of: "$100 to 1000" and a specific time-duration expressed in "full weeks" to calculate a ROI, then the Exploratory test would try the function with exactly these figures, whereas a Scripted test would also try to run the function with maybe "-£500" and "44.7 years" as typical in-values to see how the software behaved with anomalies.
However, even though the scope of the two methodologies could be different, a scripted test could easily be written to cover exactly the same functions, with the same depth, as any typical Exploratory test. All you really have to do is to use the same type of user-behaviour and in-data as stipulated in the requirements. Again, a blurring of the definition.
3. When Exploratory testing should be conducted.
After having just received the first Alpha version of a new product, a stressed Test Manager might say: "Give it a quick try with a Exploratory test now and then again just before we release", meaning that everything else in between should be Scripted testing. This strategy might be reasonable since there often is a need of quickly showing the developers what faulting functions needs to be addressed first after the initial release. There is of course also the necessity of once again going over the whole system just prior to the final release to make sure that all the latest bugfixes hasn't caused new ones. Exploratory testing, being rather cost-effective and rapid, might seem a good option in those two cases.
However, with modern software testing becoming introduced on all levels of the development cycle, the initial "quick look" should not be necessary. The test team should have had enough time to write proper test cases and suites while the program was being coded had they already been present at the planning stages. A final overhaul of the whole system could perhaps be more justifiable, especially if there is very little time left before the deadline. However, this would probably prove to be unsatisfactory in regards of quality, since if there is anything that the program should really be tested for at that stage, it should be its robustness and performance, not just basic functionality.
Redefining Exploratory Testing
With these few exceptions accounted for, I feel that it is safe to say that Exploratory testing is becoming obsolete or at least less important. While many might want to defend the role of Exploratory testing as being an important part of their everyday work, the very same people would probably agree that it is their team’s overall goal to perform as much Scripted testing as possible. Likewise would most project managers like to see their products being formally tested on all levels before release, which again makes fumbling Exploratory tests a waste of time.
I would therefore suggest a re-definition of Exploratory testing. It should be an activity reserved for the two situations described above and for the case where a tester, who has been newly assigned to the test work, needs to get a quick understanding of how the product is meant to work. Exploratory testing, in the old meaning of the word, should be incorporated into what we normally refer to as Scripted testing. Perhaps on a very rudimentary level, at least in comparison with what Scripted tests normally do, but still in the same context as Scripted tests in terms of repeatability, formal reference and thoroughness. This is the way we are already heading, we now only have to also officially recognize it.
Any Comments ?
