I’ve had enough bad experience with submitting bugs in someone’s else software or database, that I don’t bother to do it at all. Some people are responsive and helpfull, some don’t care, but I cannot know that in advance. After a few “bad” times I’m not ecouraged to point errors in somebody’s work via direct contact – it can be simply waste of time. Pointing error in publication is even worse – resistance is enourmous.

I will happily send you an email concerning HMMER. I didn’t think it’s still under development, given dynamics of its releases.

For example, if you know of a bug in HMMER, could you please just report it to me? Likewise if you know of an “artificial domain” (whatever that means) in Pfam. Thanks.

William, I don’t mind small mistakes in papers. If these doesn’t reverse the meaning of the text, I wouldn’t bother to track them. I’m interested in finding fundamental errors – wrong assumptions, misinterpretation of data and such. I believe tracking would be easy – it would take nothing more than a Greasemonkey script, that would inform reader at NCBI page that there are some bugs reported for particular paper. Rest is on reader’s side.

Marcin, I didn’t mean to bash anybody, but at least to raise a flag on errors for the community to be aware of. One example is HMMER, brilliant and widely used package: but it has a bug under certain conditions it misses domains in the analyzed sequence. If the community was be aware of that, I believe community would fix it. See for example cd-hit – currently it is hosted by Bioinformatics.org, not its author.

Roland, I agree it wouldn’t be easy to push this idea forward, but on the other hand I don’t believe in updating data by scientists themselves. It simply doesn’t happen often enough – people change their work, move between continents and I see no chance that they keep their previous work up to date. And that includes also me and my servers, so I’m also to be blamed. It’s a complex issue (you can easily get monet for new service, but hardly to maintain the old one), so I don’t think bug tracking system would solve it completely – maybe a small change in funding system (to reserve money for maintenance of data and services by ourselves) would be a better idea.

About the errors that should have been caught by peer review, I can share two perspectives, that of an author and that of a reviewer. The author often doesn’t bother to go over their manuscript for grammatical errors, feeling that it’s the editor’s job. They get the obvious stuff, but are just as prone to common errors as anyone else.

The reviewers often read for content only, feeling that pointing out grammatical errors is unimportant and not their job.

I don’t know from personal experience, but I’d imagine that this leaves editors somewhat overworked and causes them to miss the occasional mislabeled figure and so on.

It would be nice if readers could submit a “patch”. Simple bug reporting would also work, but since this would certainly cause a sharp uptick in errata, the problem of how to implement and track these corrections needs to be addressed, since the volume would probably be too much for the existing system of simply posting editorial notes or links to errata.

People reading the paper version wouldn’t be well served by a “release early and often” philosophy, certainly!

As you have mentioned, this could be an interesting method to highlight ‘bugs’ (aka errors) in peer reviewed publications and stimulate discussion. Like bug trackers for software, these publication ‘bugs’ could be marked with different priorities, and like software bug trackers, it’s up to anyone who cares to participate to discuss the problem and come to a consensus on the severity, and go about notifying the journal/database responsible (including marking the bug WON’T_FIX if the author officially refuses to fix it). Sometimes these discussions may reveal that there are simply alternate interpretations of the results, which is valuable but probably not considered a bug.

Maybe the extra attention, with the problems identified out in the open, could shame some authors or publishers into correcting serious errors or even retracting the odd paper, rather than just hoping no one will notice their mistake. Think of it as like ‘full disclosure’ for bugs in published science

Database providers can be rather protective of their databases and sensitive to criticism, in my experience. We need more honesty in admitting to errors – they’re not necessarily anyone’s “fault”; most are a result of automated processes.