Tweet: Exploratory testing: more knowledge used in test design & failure recognition + Finds failures incidentally
Tags: Software testing, Exploratory testing, Participant observation, Grounded theory, Think aloud
Paper: Juha Itkonen, Mika V. Mantyla, Casper Lassenius, "The Role of the Tester's Knowledge in Exploratory Software Testing," IEEE Transactions on Software Engineering, 11 Sept. 2012. IEEE computer Society Digital Library. IEEE Computer Society
Background
B1. Exploratory testing (ET) is increasingly used
B2. Growing evidence that industry testers see value in it
Problem: Not clear why and how ET works
Method
M1. Video & audio recording + field notes of 12 testing sessions where testers thought aloud while testing
M2. Researcher occasionally asked for clarification from tester during the sessions
M3. 30 min interviews after each session
M4. Analyzed with Grounded Theory
- Q1. Types of knowledge that is used in ET?
- Q2. How do testers apply knowledge in ET?
- Q3. Which failures recognized in ET?
Results
R1. Testers recognize failures based on their personal knowledge without test case descriptions
- R1.1 Personal knowledge = Domain knowledge + System knowledge + General SE knowledge
R2. Testers applied knowledge
- R2.1 as a test oracle to decide if result is correct or not, and
- R2.2 to design tests on the fly.
R3. Failures where found incidentally (in parts/features of system not currently being tested)
R4. Failure symptoms could be classified into 5 types of commision and 4 types of omission symptoms
R5. Failure invocation classified on number of inputs/conditions that interact to create failure
R6. Large fraction of failures did not require complicated test designs or descriptions to be provoked and recognized
R7. Failures related to domain knowledge are straightforward to provoke
R8. Failures related to system knowledge or generic SE knowledge are more complicated to provoke
Raw numbers
D1. 12 testing sessions
D2. 8 different testers in 4 different companies/units
D3. 1.5-2 hours per test sessions
D4. 88 failures found in total
D5. 20% of failures were windfall failures, found incidentally
D6. Main failure types = 24% presentation/layout + 21% incorrect result + 14% missing capability
D7. Number of inputs/conditions to create failures = 45% on 1 condition + 26% on 2 conditions + 14% unclear