ET testers use more knowledge

Tweet: Exploratory testing: more knowledge used in test design & failure recognition + Finds failures incidentally

Tags: Software testing, Exploratory testing, Participant observation, Grounded theory, Think aloud

Paper: Juha Itkonen, Mika V. Mantyla, Casper Lassenius, "The Role of the Tester's Knowledge in Exploratory Software Testing," IEEE Transactions on Software Engineering, 11 Sept. 2012. IEEE computer Society Digital Library. IEEE Computer Society


B1. Exploratory testing (ET) is increasingly used
B2. Growing evidence that industry testers see value in it
Problem: Not clear why and how ET works


M1. Video & audio recording + field notes of 12 testing sessions where testers thought aloud while testing
M2. Researcher occasionally asked for clarification from tester during the sessions
M3. 30 min interviews after each session
M4. Analyzed with Grounded Theory


R1. Testers recognize failures based on their personal knowledge without test case descriptions
R2. Testers applied knowledge
R3. Failures where found incidentally (in parts/features of system not currently being tested)
R4. Failure symptoms could be classified into 5 types of commision and 4 types of omission symptoms
R5. Failure invocation classified on number of inputs/conditions that interact to create failure
R6. Large fraction of failures did not require complicated test designs or descriptions to be provoked and recognized
R7. Failures related to domain knowledge are straightforward to provoke
R8. Failures related to system knowledge or generic SE knowledge are more complicated to provoke

Raw numbers

D1. 12 testing sessions
D2. 8 different testers in 4 different companies/units
D3. 1.5-2 hours per test sessions
D4. 88 failures found in total
D5. 20% of failures were windfall failures, found incidentally
D6. Main failure types = 24% presentation/layout + 21% incorrect result + 14% missing capability
D7. Number of inputs/conditions to create failures = 45% on 1 condition + 26% on 2 conditions + 14% unclear
Tiny SE - latest SW Eng research in <140 chars