Managing a bibliography of BiBTeX entries with Denote

When the Denote package first came on the scene, I thought to myself “Ah! This would be a perfect way to manage my bibliography!”. Some background information, I am a PhD student, and I figured that its file naming scheme would be well suited for saving the pdfs of the different papers I read. What with its tagging system, I could easily find any paper’s pdf provided I gave it sufficiently informative tags.

Which I did. My denote-directory was soon full of pdfs with informative tags and titles to make it easy to search for them again. The only problem was that I never did. And when I did, I found myself wanting the paper’s BiBTeX entry (for when it finally came to cite them in my own work) instead of the actual pdf itself. The latter could be easily gotten online, putting my habit of hoarding pdfs into question. Additionally, when I went to write some notes about a given paper (using Denote, of course), I would make a new note that would link to the pdf file. A link that was almost never followed.

As a result, I decided to rethink how I manage my bibliography, and that is what I try to describe below.

Given their importance to my workflow, my bibliography management system now centres around BiBTeX entries, and not the pdfs of the papers they refer to. Here is a typical workflow of adding a publication to my bibliography:

1. I come across an interesting paper online that I’d like to save to my bibliography.
2. After copying its BiBTeX entry from the webpage to the clipboard, I switch to emacs.
3. I then open up an org-capture template that asks for the BibTeX (which I paste in), a title (defaults to the title of the entry), and some tags.
4. A capture window opens with a Denote note that has the inserted BiBTeX enclosed in an org source block.
5. I then write any notes I may have (if any) and close the capture window with C-c C-c.

To implement this workflow, I first define the org capture template function, which dynamically creates the Denote note template from the given BiBTeX. The word ’template’ reminds me of the denote-templates user option, but the templates we’re creating are dynamic, so we need something different. Come in then org capture template functions to save us.

Here is its definition:

(defun namilus-denote-org-capture-biblio ()
  "Ask the user for a bibtex entry, title, and keywords, and creates a denote note template with:

1. The bibtex included inside an org bibtex source block.

2. The keyword \"biblio\" and the bibtex entry's sanitised key as
part of the denote file's tags. If the bibtex entry entered by
the user is empty or doesn't match the regexp, only the
\"biblio\" keyword is added, along with whatever other keywords
entered by the user."
  (let* ((bibtex (namilus-denote-bibtex-prompt))
         (title (denote-title-prompt (namilus-denote-bibtex-title bibtex)))
         (keywords (append (denote-keywords-prompt) (namilus-denote-biblio-keywords bibtex)))
         (front-matter (denote--format-front-matter
                        title (denote--date nil 'org) keywords
                        (format-time-string denote-id-format nil) 'org)))
    (setq denote-last-path
          (denote--path title keywords
                        (file-name-as-directory (denote-directory))
                        (format-time-string denote-id-format) 'org))
    (denote--keywords-add-to-history keywords)
    (concat front-matter (namilus-denote-bibtex-org-block bibtex))))

The function namilus-denote-org-capture-biblio prompts for the BiBTeX entry, a title (which is auto-completed from the inputted BiBTex), and some keywords. In addition to the keywords supplied by the user, this template function also adds two tags:

1. The ’biblio’ tag, in order to distinguish bibliography entries from other notes in my zettelkasten.
2. The ’sanitised’ BiBTeX key. Sanitisation involves the removal of non alphanumeric characters from the key so it can be used as a Denote file tag.

The motivation behind adding a BiBTeX key tag is so that given that entry’s key, I can easily find its corresponding note.

The functions that read input from the user, sanitise the BiBTeX, extract the key, generate the file tags, and create the org source block are given below. A lot of this code is taken straight from denote.el and refashioned for my purposes. Ah, the principles of software freedom in action!

(defun namilus-denote-bibtex-prompt (&optional default-bibtex)
  "Ask the user for a bibtex entry. Returns the sanitised
version. See `namilus-denote-sanitise-bibtex' for details."
  (let* ((def default-bibtex)
         (format (if (and def (not (string-empty-p def)))
                     (format "Bibtex [%s]: " def)
                   "Bibtex: "))
         (sanitised-bibtex (namilus-denote-sanitise-bibtex (read-string format nil nil def))))
    (if sanitised-bibtex
        sanitised-bibtex
      (error "Invalid BiBTeX"))))


(defun namilus-denote-sanitise-bibtex (bibtex)
  "Returns a santised version of BIBTEX. Sanitisation entails remove
all non alpha-numeric characters from the bibtex key, and
 returning this updated bibtex entry. If BIBTEX is not a valid
 bibtex entry, returns nil."
  (when (string-match "@.*{\\(.*\\)," bibtex)
    (let* ((key (match-string-no-properties 1 bibtex))
           (sanitised-key (replace-regexp-in-string "[^A-Za-z0-9]" "" key)))
      (replace-regexp-in-string key sanitised-key bibtex))))



(defun namilus-denote-bibtex-key (bibtex)
  "Returns the bibtex key from BIBTEX."
  (when (string-match "@.*{\\(.*\\)," bibtex)
    (match-string-no-properties 1 bibtex)))

(defun namilus-denote-bibtex-title (bibtex)
  "Returns the bibtex title from BIBTEX."
  (when (string-match "\\s *title\\s *=\\s *{\\(.*\\)}," bibtex)
    (match-string-no-properties 1 bibtex)))


(defun namilus-denote-biblio-keywords (bibtex)
  "Returns a list of strings \"biblio\" and the key from the BIBTEX
entry, otherwise, just returns a list consisting of the string
 \"biblio\"."
  (let ((bibtex-key (namilus-denote-bibtex-key bibtex)))
    (if bibtex-key
        `("biblio" ,bibtex-key)
      '("biblio"))))

(defun namilus-denote-bibtex-org-block (bibtex)
  "Returns a string representing an org `bibtex' source block
encompassing BIBTEX, a string of a bibtex entry."
  (concat "#+begin_src bibtex\n" bibtex "\n#+end_src"))

Next comes adding the org capture template, which is relatively straightforward:

(add-to-list 'org-capture-templates
             '("B" "Bibliography (with Denote) BibTeX" plain
               (file denote-last-path)
                      #'namilus-denote-org-capture-biblio
                      :no-save t
                      :immediate-finish nil
                      :kill-buffer t
                      :jump-to-captured nil))

This method of bibliography management I think is better than keeping just a plain .bib file, but is similar to having one large org file with each reference being a top level heading i.e the guiding principle behind Nicolas Rougier’s org-bib-mode package. Here the approach is “one-file-per-entry” instead of “one-heading-per-reference”, with all headings in the same file. Peter Prevos’ package denote-citar also allows for taking notes about a particular BiBTeX entry, but assumes you already have a citar-bibliography full of BiBTeX entries, and requires the use of the citar package as well. Prevos’ package was certainly an inspiration for this post, with the focus of the workflow I’ve described being to quickly save a BiBTeX entry and its corresponding notes, as well as create .bib files on-the-fly from them.

By using Denote to create these note files, we can also benefit from the features that Denote has to offer like filtering by title and keywords, as well as linking between related entries. This is not to say that you can’t do those things with a single org file i.e by using regular org heading tags and heading links, but I chose this workflow because I like having the standalone note files. You might then say “exporting a subtree would also result in a single file for a given heading”, but hey, that’s the beauty of emacs - the choice is up to you…