Rony Wolfinzon and Ayal Zylberman
Co-Editor – James Bach
The most well-known definition of exploratory testing (ET) was coined by James Bach and Cem Kaner:
Exploratory testing is simultaneous learning, test design, and test execution
This definition of exploratory testing was the prevailing definition for many years. The new descriptive definition says:
“Exploratory testing is an approach to testing that emphasizes the freedom and responsibility of each tester to continually optimize the value of his work. This is done by treating learning, test design, and test execution as mutually supportive activities that run in parallel throughout the project.”
The difference between ET and other common testing approaches lies in ET’s focus on the skill set of the tester rather than on the methodology and processes being used. Still, many people see ET as diametrically opposed to a structured, well defined test approach. In other words, as a QA manager once told me: “ET is the approach I instruct my testers to use when we don’t have time for structured testing”…
The fact that ET combines the tasks of learning, test design and test execution doesn’t mean it eliminates the need for planning the test. In fact, in some situations ET can be more structured and better documented than other traditional approaches.
The goal of this article is to define the most popular misconceptions about ET, and to explain why we believe these misconceptions are wrong as well as how they should be addressed. This article is based on a panel discussion conducted in Tel Aviv in May 2010, whose members numbered some leading testing experts, including James Bach himself. It also draws on notes prepared by James prior to and following the panel.
Misconception #1: ET doesn’t provide complete coverage of the application.
If you do ET badly, it doesn’t provide accountability! But if you do it well, it provides even better accountability.
Session-based test management is a well-known method for ensuring high accountability by documenting tests using many types of techniques, e.g. video, writing, log analysis, etc.
(More information about session-based testing can be found in James Bach’s website at www.satisfice.com )
Another ET practice that provides improved coverage is “state tables”. This practice is based on linking the session logs to the requirements, thereby creating transparency charts that indicate in what session each test case was tested. It’s important to mention that testing all the requirement test cases doesn’t prove that the system has been checked; however, testing all the requirements using the ET approach does increase confidence that most of the system’s bugs were found.
Misconception #2: ET is not a structured approach.
All testing is structured! The question is: how is it structured? Unskilled exploratory testers often think that ET is not structured because they are testing unconsciously, i.e. different tests are conducted in each test cycle.
ET is not a methodology. In some cases, methodologies, such as Rapid Software Testing and also Agile Testing, apply the ET approach and package it together with a full process. In other cases, a proprietary methodology is prepared based on the specific needs of the organization and the product.
By the same token, we might claim that all structured testing (ST) is unstructured. It is very rare to see in STD’s any information about what the test is testing. Most popular STD templates don’t contain any data about the test, or what requirements or flows they are testing. In session-based testing we use a guide-book that contains all the flows and “business rules” that we are testing. This gives the tester a bird’s eye view of the testing program, and therefore a better understanding of the product he is testing, as opposed to mindlessly executing a test step by step.
Misconception #3: ET testers require a different skill set than scripted testers.
There is no such thing as an "ET tester".
ET doesn’t really change the main activities involved in testing. One way of looking at it is that ET only changes the timing of these activities; instead of designing the tests and then executing them, both activities are performed simultaneously.
Most people that are capable of planning tests are also capable of executing ET. However, if you hire a tester that has only execution skills (monkey testing), he won’t be capable of working according to the ET approach. But then, at least in our experience, he won’t be capable of any other testing either, regardless of the approach being used...
Misconception #4: ET testers don’t need training.
ET is not ad-hoc testing! The exploratory tester needs a large arsenal of testing techniques in order to perform good and efficient exploratory tests. Without the techniques that are provided in each ET training program, testing is inefficient
Misconception #5: ET means lack of visibility and transparency.
One of the main arguments against ET is that it does not allow the management to control what is being tested. Well, actually, we believe this is the main argument against Scripted Testing.
Scripted Testing usually produces a mass of test documents sometimes amounting to thousands of pages or more. In most cases these documents are not inspected thoroughly, and only sample tests get reviewed.
In ET, at the end of each session, a Test Log is produced containing a charter of the test conditions, area tested, detailed notes on how testing was conducted, a list of any bugs found, a list of issues (open questions, product or project concerns), any files the tester used or created to support their testing, percentage of the session spent on the charter vs. investigating new opportunities, percentage of the session spent on creating and executing tests, bug investigation /reporting, session setup or other non-testing activities, and session start-time and duration. Although the list of items included in the log seems long, it usually only amounts to a few pages that are much easier to inspect and control and thus, provide better visibility and transparency on what is being tested.
In one of the organization we worked in, we defined a Test Leader position whose main task is reviewing the Test Logs. In practice, this was the first time a real review was done on what was being tested…
Misconception #6: ET takes more time then Scripted Testing.
The following diagram shows the difference between the ET and Scripted Testing processes:
While studying the system is done only once in ET, Scripted Testing requires that learning take place during several steps: Once, before the test planning (based on the requirements /design), and once more, before executing the test. The tester spends time both on understanding the system under test and on understanding the test documents. As a result, ET requires less time to be spent on learning and, therefore, fewer resources are needed for testing.
Another resources-related issue is the level of detail in the test documents. Although Scripted Testing often requires detailed documentation, when using ET goals can only be achieved by using high-level-of detail descriptions of the conducted tests. This style of documentation saves a lot of time, mainly when reviewing and maintaining the test documentation.
Misconception #7: When using ET, there is no room for Scripted Testing
It would be too arrogant to say that no Scripted Testing is ever needed. The fundamentals of Scripted Testing are very important to any testing project. The great benefit of exploratory testing is that it offers the testers skills and techniques that facilitate their ability to identify ”show stoppers”, and “non requirement based” defects sooner while investing less effort. The intelligent testing project manager must find the correct balance between ET and ST that will lead to the best possible outcome for his project.
Misconception #8: Most organizations don’t use ET.
WE believe that most organizations are simply afraid to admit they are using ET. Let’s take for example a tester who is highly familiar with the software. And let’s say that this tester has a 20 step test, of which the first 10 steps are identical in all other 50 tests he has already executed. Do you really think that this tester will execute the first 10 steps in each test? If you believe, like we do, that he won’t, it follows that what he is doing is simply exploratory testing with a different state each time (AKA condition state chart testing). In that case, why should he waste his time on documenting so many steps, if he is not planning to use them in the future? Examples like this abound, which is why we claim that most organizations do use ET, but are afraid to admit it.
Misconception #9: ET is cannot be applied to complex systems.
The more complex the system is, the more possible test-cases can be created to test it. This is the reason that for complex systems a certain degree of risk analysis is performed when deciding which test cases will be performed and which will be ignored However, once the tests have been selected, they are the only ones that will be executed. As a result, defects residing within the sphere of the rejected tests may be missed.
One of the main advantages of using ET is that it gives the tester the freedom to select different scenarios in each test cycle. This leads to greater coverage of the system; more defects can be found, resulting in higher quality of the system under test. So the bottom line is: the more complex a system is, the more suitable it is for ET!
Misconception #10: ET goes home
This is a well-known fallacy; ET is just a movie. Existence of aliens has never been proved.
