Most software developers and testers are all too familiar with a chart showing how the cost of fixing a bug increases exponentially over the project cycle.
Fixing a problem early (or preventing it in the design phase) is significantly less expensive than polishing it out in the certification test phase. Almost nobody argues against the principle that when it comes to fixing bugs, the earlier the better (for the project and for my health).
To address this principle, the MPLAB® IDE team has taken a highly iterative approach to software development. They send frequent builds "over the wall" to our test group. As new features come online in these test builds, we perform some
early exploratory testing to make sure that each new feature or improvement matches our expectations based on approved requirements and design documents. We often then go back to update our test plans so that they better cover the feature's scope.
So what's the problem? Maybe the chart showing the cost of a fixing a bug increasing over the project cycle doesn't show the entire picture. With the highly iterative development processes that are becoming increasingly more popular with development teams these days,
is there also a hidden cost of reporting a bug too early?By sending the test team very frequent test builds, the development team is giving the test team a very early look at a new feature,
usually before it is anywhere near being finished. At an early stage, most features are often pretty rough with stubbed functionality and a placeholder GUI. What's a test engineer to do?
At this early stage, how can a test engineer really know if an issue is a limitation in the current build or a real bug?
On one hand, I could go ahead and enter a bug report in the tracking system. When the developer sees the report they could just wait for the implementation to be further along and then just mark the issue as fixed. However, there's an associated cost with that approach. Each and every time someone from either the test team or the development team interacts with a bug report, it costs time. Until the issue/limitation gets "fixed" that bug will show up on reports, in project status reviews, and in issue-prioritization meetings, wasting everyone's time as the issue is read and discussed. Often times, these types of issues linger around long after they are actually addressed by the developer through the natural evolution of the feature. Then, later in the project, the developer can't reproduce the issue on their current code base and asks the test engineer to retest the feature. Again, all of this costs time.
Alternatively, I could
make a personal note to myself to go back to retest the functionality later and continue testing by working around the limitation. In later builds, I then go back through my personal list of "retest" items to make sure that the developer didn't miss anything. If the issue still exists, I could then re-evaluate my decision to enter a bug report or leave the item on the list for retest on an even later build. With this approach there is both an associated complication of keeping my own personal list of issues to retest in parallel to the actual bug tracking system and also a risk that I may end up delaying the bug report until it becomes more difficult and expensive to fix.
The MPLAB IDE team tries to address these concerns by providing a build status report with each build.
Each developer is responsible for providing information on the feature/changes that they have put in the build. These reports are supposed to tell the test team what to test and what is not ready for testing. They also use the reports to make us aware of any limitations in the build. These build reports tend to work out pretty well in practice so long as the developer provides sufficiently detailed information and the test engineer is aware of all of the implications of each limitation.
In the end, I believe that erring on the side of reporting a bug too early is safer than sitting on it and possibly reporting it too late.
Good communication between the development team and the test team results in better and more accurate understanding of the state of the software and therefore more effective testing.
What do you think?
From their site
