Image via WikipediaAs you have probably noticed, yesterday’s BioBarCamp was covered in depth over at FriendFeed and additionally Cameron was streaming video live from the event (it’s still available under the same address). One particular session drawed my attention, because it was about measuring impact of scientists. It’s something I have very strong opinion about since couple of weeks, so forgive me this rant.
Peter Binfield (PLoS) and Pedro Beltrao did a great job on presenting current status of the issue and presented potential way to measure impact of a publication (quoting after Shirley – “your article received x citations, viewed x times, received x comments, bookmarked x times, rated x by experts, discussed on x respected blogs, appeared in x news media, etc etc” – instead of single “your article was published in journal with IF of X”). And while two months ago I was really interested in such discussions and willing to help, today I simply don’t care. The reason is simple and is presented in the post title: by any measure, I’m average at most.
That’s absolutely obvious that majority of scientists is at most average by any standard or measure. And that is not going to change, at least not much. Those who are at the top by Impact Factor today, will be at the top by other measure. Those who do some not-that-important stuff like me, will be still pretty average by other measure. One of the reasons may be all kinds of issues with normalization of the field size (there’s too much problems with biological ontologies to believe that dividing science space into fields is going to work much better). Another thing may be relative importance of the field (that’s something different from field size) – human research will always draw more attention than electrochemistry. And I could go on and on – all these issues aren’t novel and have been described and discussed in thousands of blog posts. The point is that even if such new ideal measure is going to be fair, it will not change life of majority of scientists. Not only because some of us do average things, but also because some of us have average money (BTW, I haven’t found much discussion on including in the measure research budget, which surprises me given the fact that amount of money spent on a project correlates pretty much with the IF of the journal it is published in afterwards).
So, I don’t really care if IF stays or not (although people working on improving measuring get my deep respect). Reputation-wise I’m going to be in the middle unless I will make something extraordinary. But honestly to make a scientific breakthrough the last thing I need is a number describing quality of my thinking.
Freelancing science 
Paulo
August 8, 2008 at 14:09
Excellent points Pawel, especially when you mentioned the lack of discussion of where grant money is being spent. Every time you look grants are getting more concentrated in a smaller number of places/people.
I stopped looking at IF some time ago. It doesn’t matter if an article was published in a 0.5 IF journal. If interests me and it is a sound well written publication it is good enough for me, an average guy like you.
Pedro Beltrao
August 9, 2008 at 03:15
These additional measures should also help to filter scientific content not just to evaluate a scientist. Today we evaluate something first by where it was published and only then by reading it. We already sort and discard a lot of things just based on the evaluation of the editors. I am also arguing that having all this information attached to a publication will helps find more relevant science for us.
Pawel Szczesny
August 9, 2008 at 05:30
To find relevant science we need two decent services: search engine and recommendation engine. Algorithms used in both will significantly differ from each other and from an algorithm used to evaluate scientist’s performance (just compare Google, F1000 and IF). I’m not against attaching all meta information to the paper, but I would argue that whatever aim is it should be clearly defined – solving two issues with a single approach is likely to fail.
Danny
August 9, 2008 at 05:31
Impact factor
Perfect!
Yaroslav Nikolaev
August 10, 2008 at 20:49
thanks for the enlightenment Pawel, its a great point!! Definitely no good using same number (IF) across the whole science domain, without normalization on the field, funding, research time frames, etc. But don’t you think a score would become meaningful when applied to a narrow area of research? For example, a search filter in a semantically rich environment that gives a list of experts/groups from a defined research field ranked by some (user-defined) combination of the parameters from the “Shirley’s equation“.
Pawel Szczesny
August 11, 2008 at 08:07
Good question Yaroslav and I have no definitive answer. While it’s true that such scoring would allow for effective searching for big shots’ papers relevant to a particular query, finding them looks possible without it (see JANE, E-lise, Anne O’Tate). My feeling is that we could do better by defining similarity score between authors instead of defining research areas, but until somebody does the implementation of either approach I have no way to prove it’s better.
Yaroslav Nikolaev
August 11, 2008 at 21:31
“research area” was also meant to be a part of the example..Its just the only semantic parameter of the research process (== keywords) which is reasonably defined within existing scientific literature. Having more semantics one would be able to make more elaborate requests than simply finding biggest shots in a particular area..but you’re right, have to see some sort of implementation first to be able to extrapolate on this topic.
Academic Career Links
March 26, 2009 at 21:46
However, some people argue that one can do great science without being genius: see e.g. the famous talk You and Your Research by Richard Hamming.