▶ YouTube Summaries

• YT :: https://www.youtube.com/watch?v=fGjH82DeF7c
• Original title :: Emacs: new 'denote-data' cache development work

A spontaneous live stream (it was raining, no dog walk) in which Prot walks through the design of denote-data, a new optional in-memory cache for Denote. Denote core stays agnostic and reads through replaceable function variables. The cache is a hash table of cl-defstruct entries, built by a background Emacs process so the UI doesn't freeze. Along the way he covers hash tables, structs, sentinels, map-hash vs mapcar, and answers chat questions on his window manager and the Elisp book.

Approach: Denote stays agnostic

By default Denote reads files straight from disk and knows nothing about any cache. The new work is an opt-in layer: when enabled, the same operations read from memory instead, which is faster for long file lists and anything that needs file contents.

The mechanism is a set of variables that each hold a function. For example, a public function such as denote-directory-files calls whatever function the "how to get files" variable holds. The default walks the file system recursively; an alternative can read from a cache, a database or anything else. The same is done for known keywords and other lookups. More abstractions can be added as needed.

Consequence: anyone can write other backends (Xapian, SQLite, etc.) without Denote having an opinion. Prot is happy to widen the entry points if someone has a concrete use case. The cache itself will be a standalone package, denote-data, living on a branch for now and only here for ease of development.

Data model: structs in a hash table

Each note file is a struct, and a hash table maps identifier to struct. Prot demos cl-defstruct in the scratch buffer: defining slots generates a constructor and named accessor functions, which is essentially a plist-get wrapped in a readable function.

Hash tables give roughly constant lookup time regardless of size, so they pay off at hundreds or thousands of notes (sluggish only for tiny sets). Chat asked about oref and EIEIO classes; Prot uses structs because he knows them from his TMR package, being "a one trick pony", while noting EIEIO or a database could work in principle.

Slots in the struct: path, title, keywords, signature-like file name data, and the expensive ones - file text, forward links and backlinks. The first group comes from file names only and is trivially fast. The latter needs opening each file and scanning it.

Building the cache asynchronously

Reading file contents is slow: open the file, regex-search forward for links, and for backlinks effectively grep the whole Denote directory for each identifier (the most expensive part, growing with directory size). The demo shows Emacs frozen for a minute or two when done synchronously.

The solution: spawn a batch Emacs process that loads Denote, builds the hash table and prints it to standard output. A process sentinel watches the output buffer for the "finished" event, then reads the buffer and instantiates the data in the main Emacs. A separate stderr buffer collects messages, and there's a check for errors. Meanwhile the user keeps working.

Benchmark comparison: with file contents off, benchmark-run over five repetitions is near instantaneous (about half spent on GC). With file contents on, one run takes minutes. Prot notes the richer data (whole text, links, backlinks) is the reason.

He wrote a TODO to use a process filter to report progress (e.g. every minute or every N files read) instead of only a "started" and "done" message.

denote-data-mode

The minor mode instantiates the hash table, stores the original values of the entry-point variables so they can be restored on disable, and plugs in cache-reading functions. A user option decides whether file contents are read; default is currently on, final default undecided.

Live demo: enabling the mode starts the background build, echo area reports start and completion, and Emacs stays usable. Then in scratch:

• denote-data-get-identifiers returns all identifiers.
• Get an entry by identifier and read a slot through its accessor, e.g. the path.
• Use maphash to find files that have backlinks and whose identifier starts with "2026", pushing keys onto a list. Chat noted putting the cheap string-prefix check first would be faster; Prot agreed.

Excursus: why the let and push with maphash

Chat asked why a let is needed. maphash (like mapc) runs for side effects and returns nothing useful, whereas mapcar accumulates return values. So to collect results you create an accumulator in a let, push into it, and return it (reversing if order matters). The idiomatic equivalent is dolist. mapc is still handy for terse side-effect calls such as killing a list of buffers. Prot was not fully sure maphash doesn't accumulate and said he'd double-check.

Remaining work and open questions

• External changes: what happens when files change outside Emacs.
• Identifier changes: conceptually understood, approach not settled.
• Duplicate identifiers: a hash table keeps one entry per key, so two files sharing an identifier (e.g. an Org file and its HTML export in the same directory) would overwrite each other. Prot doesn't want to just call it user error and wants stricter handling.
• Backlinks slot currently holds xref data (files plus matches), good for a backlinks buffer but not for "which files link here". Plan: a separate simple slot or function.
• Progress reporting during cache build, a query interface, and possibly in-buffer completion backed by the cache (too costly without one).
• Visualization (graphs) is possible on this data; Peter Prevos's denote-explore already does some.

Prot stresses Denote core doesn't need a cache: file names are cheap to read, and he writes every day without one. The cache anticipates heavy users and extra use cases.

Documenting struct slots

Prot adds docstrings to each slot with a keyboard macro, then types via the :type option (used only for documentation). He had trouble finding cl-defstruct in the manuals: it isn't in the Elisp manual but in the CL emulation manual, which he thinks the Elisp manual should link to. Type syntax is unquoted, e.g. :type list, :type string. Complaints in chat and from Prot: Emacs docs need more examples.

He would like a way to express nested types (an alist whose first element is a string and the rest something else) - something like defcustom types (repeat directory, alists with element shapes) or Racket contracts. Since it is documentation only, he lets it go.

Chat and misc

• Release: no deadline for the denote-data package; likely within a week or two. Changes to denote.el merge to main in coming days while still unreleased - a good time to get involved because later changes would need backward compatibility.
• Setup: Debian with Herbstluftwm (not BSPWM), same setup for close to a decade; chosen after swapping for a wider 1080p monitor, where a manual tiler makes sense. His dotfiles configure BSPWM, herbstluftwm and i3 alike via sxhkd.
• Elisp book: no plans for additions yet, though async examples could fit; lack of reader feedback makes it hard to judge difficulty.
• Databases: he has no experience with Xapian or SQLite; someone else can build denote-xapian and similar. A chat link to a Xapian-from-Elisp package was noted as interesting.
• Contact: email, Signal or Telegram for the philosophy-minded viewer.
• Avoids consult-info on streams because previews might leak something.