Software testers devise
extensive sets of tests and execute them prior to a product's release.
These tests are usually executed on setups that testers have prepared in
their lab environment. Many organizations may have invested significant
sums of money to have the lab infrastructure in place. However, what
generally tends to happen is that when this product is released, the
customer reports issues pretty quickly. So, what happened? What happened
to all the man-hours spent on testing and the investment on expensive
lab equipment? Why did our in-house testing efforts not show up these
defects that a customer seemed to find "easily"? What are we doing
wrong?
This brings us to our theory, i.e. a defect can only be discovered in a system or environment that contains the defect
.
A defect that might show up in a customer environment may not manifest
itself in a sanitized lab environment. Often our lab environment is
setup and controlled based on our view of how the product is likely to
be used. Within the confines of the boundaries we have defined, we
execute our battery of tests and feel confident when the tests run
without reporting issues. However, a customer's environment suffers from
no such boundary constraints and does not find it hard to expose a
defect. Defects may not necessarily be in the product itself. It could
arise from the interactions of the product with its operating
environment, dependencies, usage, etc.
Therefore, unless we are able to replicate in sufficient detail the
customer environment and the likely real-world usage scenarios after the
product is released, we will likely continue to see an increasing trend
of customer reported issues.
