I was refactoring my .zshrc recently, and found these existing options for managing the command history:

setopt INC_APPEND_HISTORY
setopt SHARE_HISTORY
setopt HIST_IGNORE_DUPS
setopt HIST_IGNORE_ALL_DUPS
setopt HIST_SAVE_NO_DUPS
setopt HIST_IGNORE_SPACE

I think I copied these from somewhere whenever I last modified the file. But this time, I wanted to apply a more systematic approach to setting options, where I was sure I knew exactly what each option in the file was doing.

I stared at these options for a bit, and thought to myself:

Some of these look similar to each other, and I don't know how each one is different despite comments. And most importantly, do I even need all of these?

For the purposes of this article, I'll be dividing the options into categories:

Appending history

The default behavior for writing to the history file is to write all commands from a session in bulk at the end of the session. I'm looking for a way to append commands from different sessions as they're entered to the history file, aka sharing the same file across sessions.

The solution for this was simple, RTFM:

APPEND_HISTORY <D>
  If this is set, zsh sessions will append their history list to the history file, rather than replace it. Thus, multiple parallel zsh sessions will all
  have the new entries from their history lists added to the history file, in the order that they exit.

INC_APPEND_HISTORY
  This option works like APPEND_HISTORY except that new history lines are added to the $HISTFILE incrementally (as soon as they are entered), rather than
  waiting until the shell exits.

SHARE_HISTORY <K>
  This option both imports new commands from the history file, and also causes your typed commands to be appended to the history file (the latter is like
  specifying INC_APPEND_HISTORY, which should be turned off if this option is in effect).

One of the first things that stood out was the fact that only one of these options needs to be set, INC_APPEND_HISTORY has the functionality of APPEND_HISTORY and part of SHARE_HISTORY works like INC_APPEND_HISTORY. APPEND_HISTORY didn't do what I wanted, so it was up to me to decide of making a choice between the latter two.

I'm primarily concered around writing commands to the file, so that they're available in any sessions I start after it, not so much existing shell sessions (which SHARE_HISTORY does), so I chose INC_APPEND_HISTORY for now.

Another option I found in the man page was INC_APPEND_HISTORY_TIME, which works like INC_APPEND_HISTORY but appends the commands to the file once they've completed, which I thought was cool.

Managing duplicates

The default behavior is to keep duplicates. I'm looking for a way to store only the most recent version of a command and delete all instances of it from the file, as the command is entered. Starting with the man page again:

HIST_IGNORE_ALL_DUPS
  If a new command line being added to the history list duplicates an older one, the older command is removed from the list (even if it is not the
  previous event).

HIST_IGNORE_DUPS (-h)
  Do not enter command lines into the history list if they are duplicates of the previous event.

HIST_SAVE_NO_DUPS
  When writing out the history file, older commands that duplicate newer ones are omitted.

HIST_IGNORE_DUPS is a subset of HIST_IGNORE_ALL_DUPS, and so the choice is between HIST_SAVE_NO_DUPS and HIST_IGNORE_ALL_DUPS.

HIST_SAVE_NO_DUPS

Just going by the description and names, HIST_SAVE_NO_DUPS should have worked, but it didn't:

> setopt HIST_SAVE_NO_DUPS

> tail -2 ~/.zsh_history
setopt HIST_SAVE_NO_DUPS
tail -2 ~/.zsh_history

> tail -2 ~/.zsh_history
tail -2 ~/.zsh_history
tail -2 ~/.zsh_history

> # it's saving duplicates :o

If I close the above session and view the file in a new session, it removes the duplicate tail -2 command:

> tail -5 ~/.zsh_history
m ~/.zshrc
source ~/.zshrc
setopt HIST_SAVE_NO_DUPS
tail -2 ~/.zsh_history
tail -5 ~/.zsh_history

I'm probably misunderstanding how the option works. I thought "writing out the history file" meant each time the command got appended, now that I've set the appending to be that way. But it looks like the removal of duplicates happens only at the end of the session, irrespective of the append behavior.

I looked at the zsh source code for evidence of this, and it turns out this option is only referenced in a function called hend in Src/hist.c, indicating the end of history related operations. This seems like something that would run at the end of a shell session.

HIST_IGNORE_ALL_DUPS

Setting this option seemed to work, sort of.

> setopt HIST_IGNORE_ALL_DUPS

> tail -2 ~/.zsh_history
setopt HIST_IGNORE_ALL_DUPS
tail -2 ~/.zsh_history

> tail -2 ~/.zsh_history
setopt HIST_IGNORE_ALL_DUPS
tail -2 ~/.zsh_history

> # that worked :D

While it avoided adding immediate repeated commands (like HIST_IGNORE_DUPS), it removed older instances only once the session was closed (like HIST_SAVE_NO_DUPS).

Then I looked back at the man page, and noticed something I hadn't noticed before: HIST_SAVE_NO_DUPS makes changes to the history "file", whereas HIST_IGNORE_ALL_DUPS makes changes to the history "list".

How is a history "list" different from a history "file"? The history list stores commands for a particular shell session, before they're written to the history file. Keeping in mind the default behavior for saving history, having a temporary list per session makes sense. However, it looks like this list is in use even when the append behavior is changed.

To see how this option affects the list, we can view it using the history command:

> setopt HIST_IGNORE_ALL_DUPS

> echo hello
hello

> echo hello
hello

> history -2
 1394  setopt HIST_IGNORE_ALL_DUPS
 1395  echo hello

Commands don't repeat in the history list, and hence aren't repeated in the history file too! To confirm that it works for all older instances of a command, I tried running a command that appears slightly early on in the list:

> history 0 | grep 'echo test'
 1413  echo test

> echo test
test

> history 0 | grep 'echo test'
 1428  echo test

The line number changed, which means that the older instance was removed from the history list. However, the history file still has the older duplicate:

> cat ~/.zsh_history | grep '^echo test$'
echo test
echo test

This is due to the same reason as for why HIST_SAVE_NO_DUPS didn't work - the removal of duplicates from the history file happens only once a shell session ends.

In a nutshell, HIST_IGNORE_ALL_DUPS works like HIST_SAVE_NO_DUPS with the added functionality of removing dupes in the history list. While I expected a shell option to remove older dupes from the file as they were added, this option seems like a resonable alternative.

Fin.

Phew, that was an unexpectedly long adventure! But my history config has now reduced from 6 lines that I wasn't sure about, to 3 lines that I can confidently reason about!

setopt INC_APPEND_HISTORY
setopt HIST_IGNORE_ALL_DUPS
setopt HIST_IGNORE_SPACE