Thursday, January 11, 2007

What is a bug?

It happens sometimes, when I report a bug, that I have an argument with someone at the receiving end of the reporting chain. The raw happens over the definition of a bug. For instance, there is a new implementation of a consolidated tool. The new tool does almost all the old one did, except X. Therefore I file a bug report saying that X is missing. The ensuing argument runs along the lines of:
- This is not a bug, says the Verifier. You can't say it's a bug because it does not do what you want. It should be downgraded to feature request.
- This is the recommended replacement of the old tool, I retort, and as such it should do at least what the old tool did, plus the new stuff. I insist it is a bug.
- The manual does not mention feature X for the new tool, and then it is not a bug. It's a feature request"
The Verifier's reasoning is technically correct, and it is within the boundaries of his allowed action, so I see no point in arguing much more.
However, I can start a campaign of educating people on bug evaluation, in hope that these notes reach the ones in charge of writing the guidelines, so that this kind of requests is met more kindly next time.

So, what exactly is a Bug?

I start by borrowing some definitions from Ron Patton's excellent book on Software testing. According to that, it's a bug when ...
  1. The software doesn’t do something that the product specification says it should do.
  2. The software does something that the product specification says it shouldn’t do.
  3. The software does something that the product specification doesn’t mention.
  4. The software doesn’t do something that the product specification doesn’t mention but should.
  5. The software is difficult to understand, hard to use, slow, or—in the software tester’s eyes—will be viewed by the end user as just plain not right.
(There is an online sample chapter, The realities of software testing containing the above definition)
Let's comment on each point.

Broken promise (positive)

The software doesn’t do something that the product specification says it should do.
This is a no brainer. If you said that the editor would save my file when I hit the "save" button, and it doesn't, then it's a bug.

Broken promise (negative)

The software does something that the product specification says it shouldn’t do.
This is as simple as the previous one. If you say that the program won't overwrite my painfully composed setup file and instead it silently restores the defaults, then it's a bug.

Collateral damage

The software does something that the product specification doesn’t mention.
Here we enter in tricky territory. There is an extra feature not mentioned in the manual. Sometimes you should be just happy and say thank you, for instance if your word processor can handle simple spreadsheet-like calculations in table cells. But if the same word processor makes a hidden copy of every document you edit for backup purposes, you may get quite angry. Either way, is anything of this sort happens, then it's a bug.

Forgotten specs

The software doesn’t do something that the product specification doesn’t mention but should.
At this point, we start bickering loudly. If your word processor's manual does not mention the "save" option, and such option is nowhere to be found in the menu and task bars, anyone faintly computer-literate would complain that this omission is inexcusable. If such a missing feature happens, then it's a bug.

Catch-all clause

The software is difficult to understand, hard to use, slow, or—in the software tester’s eyes—will be viewed by the end user as just plain not right.

And now the bickering becomes open fight. You, the user, have the right to comfortable usage. The specifications warns you that you may get unpleasant side effects when using a killer application. You understand it and run it anyway, but it happens that the mentioned side effects are causing real damage. Guess what? It's a bug. No matter if it was in the docs. It should have been under the "known bugs" section. For instance, the docs may warn you that a system monitor, to keep you informed of the health of a database, will poll the server using a non-existing user. That sounds fine, but later, when your intrusion detection system rings you at 3am saying that the number of failed login attempts has reached the critical level and as a safety measure the database server was shut down, then, I can't help it, it's a bug.

This latest rule must not be used as an excuse to pass any request as a bug report. It's hard to set a clear boundary, but sometimes, without this fifth rule, many critical bugs would remain latent.

3 comments:

Rosie Sherry said...

Don't borrow definitions, come up with your own meaning.

A bug to me is a possible negative impact on the quality of a system/product. I do not work strictly to specifications, as they are full of questions and doubt. Remember they are created by humnas who are error prone. In additon to this I often work without specifications.

The best tool to use when determining a bug is your mind.

I'm sure my wording of this will develop and improve over time.

www.rosiesherry.com

Lawreence A. Murakami said...

What is your stance on the proper posting of "non-bugs". I recently posted a non-bug on the MySQL forum about an instance where the single quote character "'" is no longer being supported as a quote character for an alias_name in a order by clause. Current documentation says the backtick character "`" is supposed to be used to quote an alias_name so "not a bug" only a change of behavior.

I haven't researched previous documentation for MySQL or DB2 or Oracle documentation, but I am positive it is a change in behavior.

Giuseppe Maxia said...

If you believe that it's a bug, please report it as such in the bugs system (http://bugs.mysql.com).
If it breaks compatibility with existing applications, it may very well be a bug.