What makes a bad reference manager?
Update 2013-11-11: For some statistical data read On the popularity of reference managers, and their rise and fall
Update 2014-01-15: For a detailed review of Docear and other tools, read Comprehensive Comparison of Reference Managers: Mendeley vs. Zotero vs. Docear
At time of writing these lines, there are 31 reference management tools listed on Wikipedia and there are many attempts to identify the best ones, or even the best one (e.g. here, here, here, here, here, here, here, here, … ). Typically, reviewers gather a list of features and analyze which reference managers offer most of these features, and hence are the best ones. Unfortunately, each reviewer has its own preferences about which features are important, and so have you: Are many export formats more important than a mobile version? Is it more important to have metadata extraction for PDF files than an import for bibliographic data from academic search engines? Would a thorough manual be more important than free support? How important is a large number of citation styles? Do you need a Search & Replace function? Do you want to create synonyms for term lists (whatever that means)? …?
Let’s face the truth: it’s impossible to determine which of the hundred potential features you really need.
So how can you find the best reference manager? Recently we had an ironic look at the question what the best reference managers are. Today we want to have a more serious analysis, and propose to first identify the bad reference managers, instead of looking for the very best ones. Then, if the bad references managers are found, it should be easier to identify the best one(s) from the few remaining.
What makes a bad – or evil – reference manager? We believe that there are three no-go ‘features’ that make a reference manager so bad (i.e. so harming in the long run) that you should not use it, even if it possesses all the other features you might need.
1. A “lock-in feature” that prevents you from ever switching to a competitor tool
A reference manager might offer exactly the features you need, but how about in a few years? Maybe your needs are changing, other reference managers are just becoming better than your current tool, or your boss is telling you that you have to use a specific tool. In this case it is crucial that your current reference manager doesn’t lock you in and allows switching to your new favorite reference managers. Otherwise, you will have a serious problem. You might have had the perfect reference manager for the past one or two years. But then you are bound to the now not-so-perfect tool for the rest of your academic life. To being able to switch to another reference manager, your reference manager should be offering at least one of the following three functions (ideally the first one).
- Your data should be stored in a standard format that other reference managers can read
- Your reference manager should be able to export your data in a standard format
- Your reference manager allows direct access to your data, so other developers can write import filters for it.
The latter point is of particular importance if you think about using a web-based reference manager (e.g. CiteULike or Bibsonomy). In this case, chances are that other reference managers cannot access you data, especially not, if the provider of the web-based reference manager decides to turn-off its service. So, you should definitely check if the web-based reference manager allows you to export all your data or allows accessing the data via an API. We emphasize the “all your data” because some reference managers only offer an export function for some of your data. This means, only because you see an export button in your favorite reference manager, you shouldn’t think “All right, there is an export function, everything is great”.
A prominent example for a non-complete export function is Mendeley. Mendeley allows exporting bibliographic data of your PDFs (including tags) as BibTeX file. However, Mendeley does not export the virtual folder structure in which you organize your PDFs. More importantly, you cannot export text you highlighted in PDFs, or comments you made in your PDFs . Imagine, you spent dozens of dozens of hours sorting your PDFs to the folders in Mendeley, and annotated your PDF files but then, in a few years, you decide to use another tool than Mendeley. You would want to export all your data but couldn’t. So, you would have to make a decision: Do you annotate all your PDFs again with the new reference manager or just stick with Mendeley, even if it’s not the best solution (any more)? That’s a dilemma – you are locked-in.
Another example is
Qiqqa and ReadCube. As far as we know, Qiqqa and Readcube store your PDF annotations in a proprietary format which cannot be exported to the standard PDF format. Hence, if you are using Qiqqa or ReadCube (and annotate PDFs) you will be locked-in in a similar way as with Mendeley. [Update 2014-11-26: Please read the comments for more information about Qiqqa]
2. A “must-register” feature that forces you to store your data in the cloud
In the good old days, you downloaded a software, installed it on your computer and your data was all yours. Nowadays, there is the cloud. We won’t discuss here whether storing data in the cloud is generally sensible or not. However, we argue that you shouldn’t use a reference manager that forces you to store your data in the cloud, even if you personally advocate the cloud. Fact is that there is a growing concern about privacy issues relating to cloud storage. As a consequence, some companies ban cloud services from their employees’ computers. For instance, IBM banned services such as Dropbox because of privacy concerns, and from personal experience we know that also some universities are quite skeptical about Dropbox & Co. When you use a reference manager, that stores all your data in the cloud, we see a good chance that someday your university will ban this tool, because the university does not want their employees’ highly sensitive data to be stored in the cloud. We don’t know yet of any universities who did so (do you?), but at least in smaller scale this already happens. For instance, I recently met a researcher at the JCDL conference. He just got a PostDoc position and his Professor told him, he could use any reference manager he wanted, as long as it doesn’t store his and the working group’s data in the cloud. Actually, the researcher was using Mendeley and although he was totally happy with it, he was forced to use another tool, only because Mendeley had no option to work locally, without registration. So, at least if you are considering a long term career in academia, we suggest to use a reference manager that does not force you to store your data in the cloud. Otherwise, you might be forced to migrate to another tool due to to some policies of your boss or university (and migrating is usually very time consuming, or, if there is a lock-in, not possible at all).
3. A “loner feature” that prevents you from collaborating with your colleagues
Sooner or later you will be required to collaborate with other researchers, at least occasionally. Then, your reference manager hopefully will support this or at least it will not prevent you from doing so. There are three factors that could make a collaboration difficult or even impossible.
A) Proprietary data-formats
In the ideal case your collaborator is just using the same tools as you are – but what if not? Then, it becomes crucial that the reference manager of your collaborator can read the data of your tools and vice versa. That means, when selecting your favorite reference manager, you should make sure it can read and write standard data formats and is not using proprietary formats. Mendeley, for instance, will make it difficult (if not to say impossible) to collaborate with researchers using a reference manager other than Mendeley. As explained earlier, Mendeley has no export (or import) for annotations in PDF files. So, you cannot share your annotations with colleagues when they are using PDF editors other than that of Mendeley. Also sharing and working on the same BibTeX file would be difficult. Although Mendeley can import and export BibTeX files, Mendeley does not update existing entries in its database with entries that have been changed in the BibTeX file. This means, if you export Mendeley’s bibliographic data as BibTeX, and your colleague is changing some data in the BibTeX file, you may import the BibTeX file again, but all modified documents are imported as new documents, so you would have duplicate entries in Mendeley.
B) Infeasibility for others to use your favorite reference manager
If the data format of your reference manager is not compatible with the tools of your collaborators, you could still try to make your collaborators using the same software as you are doing. However, even if they wanted, there might be reasons preventing the use of your favorite reference manager.
First, not all tools are available for all operating systems. If you are using a reference manager for Windows, but your collaborator is using Linux or Mac then you have a serious problem. Of course, the collaborator could try to run the reference manager in a virtual machine, or install Windows but this is a very error prone and time consuming task (and costly, if the collaborator has to buy a Windows license). Most likely, not all your potential collaborators would be willing to do so. This means if you decide to use a reference manager not available for all operating systems, you most likely will have to compromise in the choice of your collaborators.
Second, the price of the software might be important. A tool like Endnote costs about 250$ for a single standard license. Even if you think the software is worth it, or if you get the software for free through your university, not all potential collaborators can or want to pay the price (think of students, researchers in developing countries, or researchers who just want to have a quick look at your data).
c) Local databases
If you are using a reference manager that stores your data e.g. in BibTeX format, collaboration is easy. You can just store the BibTeX file in your Dropbox and share it with collaborators. Alternatively, you could setup e.g. SVN which would not only sync BibTeX files but merges them when several users make changes at the same time. Similarly, you could share PDFs via Dropbox and annotate them with standard PDF readers such as Adobe Acrobat, Foxit Reader, … However, many recent reference managers store their data not in individual files (e.g. BibTeX, or PDFs) but in a local database. Databases have several advantages for your daily work but they have one huge disadvantage: You cannot simply copy them to your Dropbox and share them with your colleagues. If these reference managers do not offer some own functionality for collaboration, it will be almost impossible for you to share your data. If the reference manager offers an integrated collaboration function this might even be more comfortable and powerful than e.g. Dropbox. However, it usually comes at a price and that is typically quite high. In case you don’t want to collaborate but to synchronize your data between e.g. your home and office computer, your reference manager should at least allow to change the path of the local database. In this case you can store your database e.g. in Dropbox and use it on another computer.
Therefore, if you look for a reference management tool, check if you can use standard synchronization tools such as Dropbox (or other third party solutions). If you cannot, look at the pricing plans and really think about if you are willing to pay the price. For instance, at time of writing, Mendeley charges 99€ (~130$) a month for their business team package with five users. We are not saying that Mendeley’s service is not worth the money. Actually, their collaboration features are great and far more powerful than e.g. Dropbox. But 99€ a month (i.e. ~1200€ a year!) is still quite a bit of money for five users to collaborate. Think about, if you are really willing to pay this price and whether potential collaborators of yours would be willing to pay their share, too.
Many researchers are selecting their favorite reference managers based on what wonderful features the tools offer. However, we argued that in a first step researchers should think about what ‘features’ the reference manager should not offer. We proposed that reference managers that a) lock you in, b) force you to store your data in the cloud, or c) prevent collaboration, should be ignored right away. Maybe, if you are a Bachelor or Master student, and you are 100% sure that after handing in your thesis, you will never work again with academic literature, you might ignore the advice we proposed. However, if you are planing to work for several years in academia (or if you are uncertain), we suggest to think about what we wrote. Otherwise, chances are that you won’t be happy with your reference manager in the long-run.
It’s probably needless to say that Docear is none of the bad reference managers. Where ever possible we use open standard formats to store your data and all your data is on your own computer and (optionally) on our servers*. Your PDF annotations are stored in the PDF standard format, and any standard PDF viewer will be able to read them, and you can share your PDFs (and annotations) easily e.g. via Dropbox. Your references are stored in BibTeX format which can be read by almost any reference manager and sharing and collaborating with BibTeX files is easy. Mind maps are stored in Freeplane’s XML format, a format that is understood by several other tools and that any developer could easily write an importer for within hours. You may export your mind maps in various other formats (e.g. PDF, PNG, …) and, again, you can share them easily. Even the source code of Docear is publicly available. If we should ever stop developing Docear, anybody could continue our work.
Does all this mean that Docear is the best reference manager? Not necessarily. Only because something is not bad, doesn’t make it automatically good. However, we hope that our argument helps to avoid selecting a reference manager that will make you regret your decision, and we are very keen to hear which reference managers you think belong to the bad category and which don’t. How about BibDesk? Bibsonomy? Bibus? Citavi? CiteULike? EndNote? Mendeley? Papers? Qiqqa? Refbase? RefWorks? Zotero? …? We are also looking forward to hearing any other of your ideas about this topic. Did we forget some no-go features? Do you agree with our list? Are we exaggerating? Were we too biased when writing this article? Let us know!
Update: 2013-11-20: I created a table showing the three ‘negative-features’ for some selected reference managers. Please help us to complete the table. If you know a missing piece, send the information to firstname.lastname@example.org. Also, if you discover errors, please let us know!
 List adopted from http://literaturverwaltung.wordpress.com/vergleich-literaturverwaltungssoftware/
 In Mendeley, you can export each of your PDFs individually and your comments will be added to the last page of the PDF. However, this is hardly a true export of your data and no other tool will be able to recognize the annotations in the PDF as real annotations but only as normal text. This also means that if you export Mendeley’s annotations, you might see them in your favorite PDF viewer but you won’t be able to edit or delete the annotations. For more information on the shortcomings of Mendeley’s PDF export, read the comments.
Update note, November 1st 2013: The “must-registration” feature was added.