Do reference management systems encourage sloppy referencing practices?

Over at the Dynamic Ecology blog there’s an interesting discussion going on about “how to keep up with the literature” that’s relevant to all fields, not just ecology.  Spoiler alert: it’s impossible to “keep up” if “keep up” means “read everything”.  But do check it out as there’s lots of good advice in that post.

One of the topics that’s arisen in the comments is about the use of reference management systems such as Endnote, Refworks, Zotero, Mendeley, etc. Everyone has their own preferences as to which to use, and there seems to be advantages and disadvantages to all of them.  However a minority (so it seems) of us don’t use any kind of reference management system, which strikes those who do as very odd.  Personally, I tried Endnote a long time ago, it was ok, then I lost the database when an old computer bit the dust.

I’m not sure how much more efficient/effective I would be as a publishing academic if I was to get back into using a reference management system. One of the supposed advantages of these systems, that they will format references to the specific requirement of a particular journal, seems to me to be a double-edged sword.  I actually find re-formatting references quite relaxing and I think (though I may be wrong) that it develops attention-to-detail and accuracy skills that are useful in other contexts.

Also I suspect, but have no proof, that reference management software is responsible for perpetuating errors in the reference lists of papers that then result in mis-citations on Web of Knowledge, etc.  My suspicion is that this has got worse over time as people rely more and more on reference management software rather than their brains.  These citation errors can have an impact on an individual’s h-index, as I mentioned in a post last year.

By coincidence yesterday I spotted a hilarious example of just this kind of mis-citation that I think can be blamed on a reference management system. This paper of mine:

Ollerton, J., Cranmer, L. (2002) xxxxxxx Oikos xxxxxx

was rendered in the reference list of another paper as:

Ollerton, J., Cranmer, L., Northampton, U.C., Campus, P. (2002) xxxxxxx Oikos xxxxxx

The last two “authors” are actually from the institutional address – University College Northampton, Park Campus! [UCN is the old name for University of Northampton].

Now in theory that shouldn’t happen if an author’s reference management software is doing its job properly, and information has been correctly inputted, but it does happen: errors are not uncommon.  In addition (it seems to me) authors often don’t check their reference lists after they have been produced by the reference management software. That’s sloppy scholarship, but I can understand why it happens: people are busy and why bother if the software is (in theory) getting it right every time?  It also shouldn’t happen at the editorial production end of things, because references are usually cross-checked for accuracy, but again it does, even for top-end journals (in this case from the Royal Society’s stable!)

Again it’s anecdotal but I’m also noticing that reference lists in PhD theses that I examine are getting sloppier, with species names not in italics, various combinations of Capitalised Names of Articles, unabbreviated and abbrev. journal names, etc. etc.

Does any of this really matter?  Isn’t it just pedantry on my part?  Whilst the last statement is undoubtedly true, I think it does matter, because attention to detail at this very basic level gives the reader more confidence that attention has been paid at higher levels, such as citing accurate statistics from primary sources to back up statements, rather than relying on secondary sources, as Andrew Gelman discussed in an old blog post on referencing errors.

But maybe I’m a lone voice here, I’d be interested in your thoughts.

22 thoughts on “Do reference management systems encourage sloppy referencing practices?

  1. ScientistSeesSquirrel

    I think you’re entirely right.

    I recently came across this one: “Sachs, J. L., U. G. Mueller, T. P. Wilcox, J. J. Bull, T. H. E. Evolution, and O. F. Cooperation. 2004. The evolution of cooperation. Quarterly Review of Biology 79:135–160.” Now, if there really is an Otto Francis Cooperation out there, he’s working in exactly the right field…

    I’m as guilty as the next person, I think. I will admit that I rarely get a paper through the proof stage without someone finding some kind of referencing error. But: when I find one, I fix it IN MY REFERENCING SOFTWARE, and thus I never make the same mistake again. I understand why these errors happen. I don’t find checking reference formats soothing like you seem to; I find it stultifying and unrewarding. I’m willing to bet that one this, at least, most people are more like me than like you! These mistakes drive me nuts, but I know why there’s there, and I make them too.

    But here’s the thing: much like you, I can’t decide if I’m just being a pedant. The ones that mess up citation counting etc are a minor nuisance 99% of the time (it’s hard to make a case for widespread serious consequences). This will abate as search algorithms get ever better. Humans searching for literature should only rarely be flummoxed. And things like unitalicized species names and inconsistent capitalization: yeah, those drive me absolutely bonkers – but I can’t honestly figure out why they matter. (Note to any of my students reading this: please expunge that last sentence from your brains).

    Will be interesting to see further discussion on this!

    1. jeffollerton Post author

      “I think you’re entirely right.”

      That’s the first time anyone has uttered those words to me in my life…..I may have them carved on my tombstone 🙂

      But seriously, thanks for your comments. Your example is great, there must be lots out there, looking forward to seeing others.

      It would be a really interesting study to see if there is a correlation between referencing errors and other kinds of errors in papers, e.g. getting stats wrong. One way to approach it would be to see if retracted/corrected papers tend to have more referencing errors than average for uncorrected/retracted papers. No doubt that could be done with some clever programming

      1. ScientistSeesSquirrel

        That’s a brilliant idea (now you need a bigger tombstone to put that on there too). I wonder if the retraction study has been done. That might be a bit of a blunt tool, though. I really like your stats/referencing errors suggestion. That would be a great thesis idea in Science Studies.

    2. Simon Tarr

      I agree with Stephen here with regards to the utility of fixing it in my reference software (I use Mendeley) so that the mistake can’t happen again (unless you just copy and paste incorrect references with no QC!).

      My workflow is essentially like this:
      When I add a citation to a document, I try to make the reference accurate there and then. I find that nibbling away and polishing my reference database as I go along is infinitely preferable to 3 hours of eye-strain while you verify an entire reference list for a paper in one go. After I have finished a piece of work, I then always go through and manually check authors/titles/issue numbers etc. I’ll write a little note within the software to say that I’ve manually checked this entry and it’s correct. The next time I use that reference it takes less than a second to verify its accuracy.

      The benefit of Mendeley (and others) to back up the database online, automatically scan and populate a paper’s metadata and then keep everything in sync across multiple devices outweighs the odd mistake that may creep through.

  2. Dustin Herrmann

    This sounds like a version of the “I’ll stick with my typewriter” argument. Go for it if it works for you, but citation software makes us (the general) more productive, although I make no comment on whether they make users more innovative or happier.

    1. jeffollerton Post author

      Thanks Dustin, but I think you’re wrong on that first count. I know I sound like a Luddite, but I’m really not – I blog and use social media a lot, for instance.

      But regarding your second point that “citation software makes us ….. more productive”, where is the evidence to back up that statement? Do scientists who use reference management systems really produce more papers per year, or have greater funding income, or however else we want to measure “productivity”?

      1. Dustin Herrmann

        Not calling you a Luddite, just observing that the argument has some of the same elements. As far as productivity, the same logic by which computers are more productive than typewriters at document production and modification would apply in this case of manual versus assisted reference management. I do not mean to argue whether emergent properties of productivity (e.g., papers per year, how well grounded papers are in existing scholarship) are affected.

  3. Benno Simmons

    Completely agree with what’s written here. Having tried pretty much every reference management program over the years, I find them universally pretty terrible.

    In my experience the time spent correcting an automatically compiled bibliography is not much less than the time it takes to write it out manually. Plus they all seem to have pretty horrible clunky interfaces.

    Reference management software is still an unsolved problem as far as I’m concerned.

    But they work well for lots of people, so this is just my experience!

  4. Brad Scott

    Coming from the publishing end of the process, I’m not sure I’ve noticed any change in sloppiness of referencing over the years. It has always been a wide spectrum of good and bad. I can think of plenty of cases of old works that have caused lots of trouble when trying to automatically tag the bibliographies since they had the punctuation all wrong.

    For the few bits of writing I do, I use EasyBib, though I do take care to check any imported data against the actual article or book. I’ve been creating digital data for long enough to know how easy it is to get the algorithms wrong.

  5. laurajanegraham

    I think that RMS are very useful for organising and formatting references, but they need to be used with a lot of care because, as you point out in the post, they can generate some nonsense text. I have heard stories of when RMS formatting has gone very wrong. I tend to have a three stage process for dealing with references: 1) whenever I cite a paper in something I’m writing I move it into a folder for that specific paper; 2) when the paper is done I go through each reference in the folder and check that all of the fields are correct; 3) do a sanity check of the final references section (admittedly because I have faith in step 2 I tend to be quite slack about step 3).

  6. angela moles

    I wonder if there are upsides (other than being fast and easy, but yes, error prone) of reference management software that you haven’t mentioned here. For instance, I wonder if using this sort of software make people any more likely to cite ECRs, researchers from developing countries etc rather than just repeatedly citing the big names whose work we know well (e.g. by making it easier to relocate that cool paper you read last year for which you can’t remember the authors’ names).

    1. Elinor

      I also will search through my references on occasion, using the built-in search feature of my reference management software (Zotero).

      Another big advantage of using the software is that I don’t have to worry about revising my references list every time I add or remove a citation in the main text.

      (Like some others who have commented, I check each citation for accuracy as I add it to the software. Occasionally I have to manually fix references for things like computer software or reports in the document, but not having to type out and format every single citation saves me a lot of time.)

  7. xykademiqz

    I am a Latex user so we use Bibtex for references. Essentially, from the same .bib file, different bibliography style files (each journal has its own) helps produce a list of references in a desired format. So the actual restyling is not a big deal, provided you actually have your references in a reasonably correct format. But I have noticed that students will just grab the Bibtex record from either Google Scholar or the article’s journal page, and even these Bibtex records sometimes have inaccuracies, or don’t abbreviate journal name, or list full first and middle names of authors, or have the journal name in all caps, etc. I always check the entry when it’s first made and then it’s accurate forever, but my students don’t care to fix these in the .bib files and I always have to do the cleanup; it drives me nuts.

  8. Helen

    I can’t get my head round endnote etc, though my scholarship is non-existent at the moment so that probably doesn’t matter. On the other hand, correct references do matter – I’ve just marked a pile of essays with so many mistakes, which made finding sources quite difficult at times. And as you said, Jeff, these kinds of errors definitely create the wrong impression.

  9. Pingback: Recommended reads #86 | Small Pond Science

  10. Pingback: One of the reasons why I don’t use reference management software…. | Jeff Ollerton's Biodiversity Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s