Using rdfind To Deduplicate Obsidian Dropbox Backups

(Last updated )
Obsidian Tips

EDIT 2022-03-01: The following still works but I can’t recommend using the aut-o-backups plugin. Its lack of features and configurability really makes it unusable for any real-world use-cases.

This might be of interest for anyone who is using the aut-o-backups plugin to automatically backup their Obsidian vault to Dropbox. The plugin is intentionally leaving out a lot of the complexity that is normally involved in dealing with backups, like pruning (only keep x backups) and deduplication. This post is about the latter.

The tool rdfind can be used to deduplicate the backups and in so doing safe A LOT of storage (especially if you have included binary files like images).

Dropbox is able to follow symlinks (or soft links) so it’s possible to deduplicate the created backups by converting duplicate files into symlinks pointing to only one file.

The following has been tested and is used on a Mac.

(Please make a backup of your backups before attempting any of this!)

  1. Open the terminal
  2. Install rdfind with Homebrew if you haven’t done so already: brew install rdfind
  3. Navigate to your backups folder: cd ~/Dropbox/Apps/Obsidian\ Backups
  4. Run rdfind: rdfind -makesymlinks true .

You will get output looking similar to this:

rdfind -makesymlinks true .
Now scanning ".", found 2475 files.
Now have 2475 files in total.
Removed 0 files due to nonunique device and inode.
Total size is 467948959 bytes or 446 MiB
Removed 13 files due to unique sizes from list. 2462 files left.
Now eliminating candidates based on first bytes: removed 10 files from list. 2452 files left.
Now eliminating candidates based on last bytes: removed 0 files from list. 2452 files left.
Now eliminating candidates based on sha1 checksum: removed 2 files from list. 2450 files left.
It seems like you have 2450 files that are not unique
Totally, 223 MiB can be reduced.
Now making results file results.txt
Now making symbolic links. creating 
Making 1406 links.

This can be run on regular intervals (once an hour or so) to deduplicate your backups. I use Keyboard Maestro for this:

Good Apps - Obsidian

(Last updated )
Good Apps

Obsidian is a personal knowledge base and note taking app with a lot of flexibility and extensions that make it very powerful. If you like to take notes and be in control of your data, then Obsidian might be the right fit.

What makes Obsidian so valuable for me is that it allows me to consolidate a lot of things into one app. Like information about a project and its tasks. Reference materials about sociology and solutions to programming problems. Everything can be made to fit into Obsidian and thanks to its extensibility and configurability, the app can be adapted to my evolving needs over time.

Most people point out that Obsidian is an app for people who like to link their notes together. If you rename a note in Obsidian, all the links to it will be renamed, too. Linking notes, so that they become chains and/or trees of notes unlocks emergent qualities that you wouldn’t be able to access without linking them.[1] It is also available as an iPhone app, that can do almost all the things that the Mac app can do. So you can take all your stuff with you as well.

Obsidian might not be for you, because it might just be a little bit too much. But it might also just be the tool you were looking for. If you happen to have a tendency to react skeptically to overly hyped apps like this one: Ignore the knowledge graph visualization stuff[2] and also the personal knowledge base singularity talk and just try it out. Obsidian is great. Don’t let the sometimes overly boastful rhetoric in this space keep you from at least trying it.

  1. For example: You will find notes that, taken together, inspire a new creative thought. You might also rediscover an old solution to a problem you currently have. Or you might notice a pattern in otherwise disparate information.  ↩

  2. I think that visualizing one’s notes could at some point be helpful for maintaining and refactoring, but we are not there, yet.  ↩

The Early Riser Day 47 🎙🌅

Podcast > The Early Riser

Any problems you may have in a particular area of your life are the result of the system that produces these.

I don’t think I managed to talk about it eloquently, but what I tried to say: This is an interesting perspective, however you might not be in complete control of the system that produces problems. So you may have to satisfice and maybe even accept some things as realities that may have looked like solvable problems at first.