set -o pipefail Failed My Clip Script on Some Nights and Not Others: Exit Status 141, head and grep -q

A three-line script that prints the oldest clip in a folder passed every test I gave it with a small folder. With 1000 clips it failed 22 times out of 100. With 2000 clips it failed every time. Each failure was exit status 141, with nothing on stderr and nothing on stdout.

It ran under set -euo pipefail, a common first line for scripts like this, and its pipeline ended in head -n 1. This post is about what those two do together, why the failure comes and goes with folder size, and how to keep pipefail anyway.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it stores files. Every result comes from the scripts shown, run on this machine:

GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
sort (GNU coreutils) 8.32
head (GNU coreutils) 8.32
find (GNU findutils) 4.8.0
CPUs: 2

The script

The folder is fake: empty files, each touched one minute after the last, so the oldest is cam1_00001.mp4.

#!/bin/bash
# mk.sh N : make N empty clip files, one minute apart, oldest first.
n=$1
rm -rf clips && mkdir clips
for i in $(seq 1 "$n"); do
  f=$(printf 'clips/cam1_%05d.mp4' "$i")
  : > "$f"
  touch -d "@$((1761000000 + i * 60))" "$f"
done

The script asks find for each clip's modification time and path, sorts by time, and keeps the first line:

#!/bin/bash
set -euo pipefail
# Print the oldest clip, so a cleanup job knows where to start.
oldest=$(find clips -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1)
echo "oldest: ${oldest#* }"

How often it fails

This runs the script 100 times per folder size and counts non-zero exits. list_bytes is what sort tries to hand to head.

#!/bin/bash
# Run oldest.sh 100 times per folder size and count the non-zero exits.
for n in 50 90 100 200 500 1000 1500 2000 3000; do
  ./mk.sh "$n"
  bytes=$(find clips -name '*.mp4' -printf '%T@ %p\n' | wc -c)
  fails=0
  for k in $(seq 1 100); do
    ./oldest.sh > /dev/null 2>&1 || fails=$((fails + 1))
  done
  echo "clips=$n  list_bytes=$bytes  failed=$fails/100"
done

Output:

clips=50  list_bytes=2150  failed=0/100
clips=90  list_bytes=3870  failed=0/100
clips=100  list_bytes=4300  failed=3/100
clips=200  list_bytes=8600  failed=6/100
clips=500  list_bytes=21500  failed=4/100
clips=1000  list_bytes=43000  failed=22/100
clips=1500  list_bytes=64500  failed=30/100
clips=2000  list_bytes=86000  failed=100/100
clips=3000  list_bytes=129000  failed=100/100

Three bands. Up to 90 clips: 0 failures. From 100 to 1500 clips, it failed some of the time, and an earlier run gave different counts in that band. From 2000 clips up, it failed every time.

The middle band is the bad one: a job tested on a few dozen clips works, gets scheduled, and then fails on some nights and not others once the folder fills up.

What 141 is

The Bash manual: "When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status." 141 minus 128 is 13, and signal 13 is:

#!/bin/bash
./mk.sh 3000
./oldest.sh
echo "oldest.sh exit status: $?"
kill -l 13
set -o pipefail
find clips -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1 > /dev/null
echo "PIPESTATUS: ${PIPESTATUS[*]}"

Output:

oldest.sh exit status: 141
PIPE
PIPESTATUS: 0 141 0

The first line is all oldest.sh printed: its echo did not run, so no partial result and no message. PIPESTATUS shows which part died: find 0, sort killed by SIGPIPE, head 0. head had already printed the wanted line.

head -n 1 reads until it has one line, prints it, and exits. Its end of the pipe closes. The Linux pipe man page says what happens to the process still writing: "If all file descriptors referring to the read end of a pipe have been closed, then a write(2) will cause a SIGPIPE signal to be generated for the calling process." Its default action ends the process. Nothing was wrong with the data; sort was still talking when head hung up.

Without pipefail nobody notices: the pipeline takes head's status, 0. With it, per the manual, "the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status". That is sort's 141, and set -e ends the script.

Why the folder size decides it

Whether sort gets the signal depends on whether it still has a write to make after head has gone. Tracing sort's writes on a 1000-clip list:

#!/bin/bash
./mk.sh 1000
find clips -name '*.mp4' -printf '%T@ %p\n' > list.txt
strace -e trace=write -o sort.trace sort -n list.txt | head -n 1 > /dev/null
cut -c1-72 sort.trace

Output:

write(1, "1761000060.0000000000 clips/cam1"..., 4096) = 4096
write(1, "0000000000 clips/cam1_00096.mp4\n"..., 4096) = -1 EPIPE (Broke
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8043, si_uid=5924
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_TKILL, si_pid=8043, si_uid=592
+++ killed by SIGPIPE +++

sort writes in 4096-byte blocks. Slowed by strace, its second write found head gone. Without the tracer it is a race:

• At most one block (the 50 and 90 clip lists are 2150 and 3870 bytes). sort makes one write and is done before head can close anything. 0 failures.
• More than one block, but less than the pipe holds. The pipe man page gives its capacity: "Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes)." Every block fits, so sort can usually write them all before head exits. Usually. When the scheduler runs head first, the next write hits a closed pipe.
• More than the pipe holds (the 2000 clip list is 86000 bytes). sort has to wait for head to read before it can write the rest. head reads the first line and leaves. The next write hits a closed pipe, every run.

The man page adds "Applications should not rely on a particular capacity". Your thresholds will differ with CPU count, load and versions. The three bands will not.

The quieter version: grep -q in an if

grep -q stops reading at the first match, for the same reason head stops at one line. Put it in an if:

#!/bin/bash
set -euo pipefail
# Is there at least one clip from camera 1 in the folder?
if find clips -name '*.mp4' | grep -q 'cam1_'; then
  echo "cam1 clips found"
else
  echo "no cam1 clips"
fi

Run 100 times on a folder of 10 clips, then on a folder of 3000, all of them from camera 1:

#!/bin/bash
for n in 10 3000; do
  ./mk.sh "$n"
  wrong=0
  for k in $(seq 1 100); do ./grepq.sh | grep -q 'no cam1' && wrong=$((wrong + 1)); done
  echo "clips=$n  said 'no cam1 clips' $wrong/100 times"
done

Output:

clips=10  said 'no cam1 clips' 0/100 times
clips=3000  said 'no cam1 clips' 100/100 times

With 3000 matching clips it said there were none, every time. grep -q matched and exited 0; find, still listing, was killed by SIGPIPE; pipefail made the status 141; if read that as false.

set -e does not save you here. The manual lists the cases where it does not exit, and one of them is a command that is "part of the test in an if statement". So the script runs the wrong branch and exits 0. If the else branch sends an alert that camera 1 stopped recording, it sends it every time the folder is large.

Same script, different status: when the parent ignores SIGPIPE

A process can be set to ignore SIGPIPE, and child processes inherit that. Then, per the pipe man page, the write "fails with the error EPIPE" instead, and the program decides what to do. Python ignores SIGPIPE itself. Its subprocess documentation says: "If restore_signals is true (the default) all signals that Python has set to SIG_IGN are restored to SIG_DFL in the child process before the exec." Run the script both ways from Python:

import subprocess
# Python ignores SIGPIPE itself. restore_signals decides whether the child gets it back.
subprocess.run(["./mk.sh", "3000"], check=True)
for restore in (True, False):
    p = subprocess.run(["./oldest.sh"], capture_output=True, text=True,
                       restore_signals=restore)
    print("restore_signals=%s  exit=%d  stdout=%r" % (restore, p.returncode, p.stdout))
    print("  stderr=%r" % p.stderr)

Output:

restore_signals=True  exit=141  stdout=''
  stderr=''
restore_signals=False  exit=2  stdout=''
  stderr="sort: write failed: 'standard output': Broken pipe\nsort: write error\n"

With the default, the same silent 141. With the signal left ignored, sort gets the error, prints Broken pipe, and exits 2. One script, one folder: 141 and silence from a terminal, 2 and a write error from a parent that leaves SIGPIPE ignored. I only tested Python as the parent; check other launchers before reading a 2 as a disk problem.

Fix one: a last command that reads everything

A reader that reads to the end of its input cannot cut anything off. Here awk keeps the smallest timestamp it has seen and prints that line at the end, so no sort is needed:

#!/bin/bash
set -euo pipefail
dir="${1:-clips}"
# awk reads every line before it prints, so nothing upstream is cut off.
oldest=$(find "$dir" -name '*.mp4' -printf '%T@ %p\n' |
  awk 'NR == 1 || $1 < min { min = $1; line = $0 } END { print line }')
echo "oldest: ${oldest#* }"

The tempting fix is to delete pipefail. Here is the same pipeline as oldest.sh without it, run against the 3000-clip folder and against a folder that does not exist:

#!/bin/bash
set -eu
dir="${1:-clips}"
# The same pipeline as oldest.sh, with pipefail removed.
oldest=$(find "$dir" -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1)
echo "oldest: ${oldest#* }"

#!/bin/bash
./mk.sh 3000
for s in oldest_awk.sh oldest_nopf.sh; do
  fails=0
  for k in $(seq 1 100); do ./$s > /dev/null 2>&1 || fails=$((fails + 1)); done
  echo "$s on 3000 clips: failed=$fails/100, prints: $(./$s)"
done
echo "--- a folder that does not exist ---"
./oldest_awk.sh no_such_dir; echo "oldest_awk.sh exit status: $?"
./oldest_nopf.sh no_such_dir; echo "oldest_nopf.sh exit status: $?"

Output:

oldest_awk.sh on 3000 clips: failed=0/100, prints: oldest: clips/cam1_00001.mp4
oldest_nopf.sh on 3000 clips: failed=0/100, prints: oldest: clips/cam1_00001.mp4
--- a folder that does not exist ---
find: ‘no_such_dir’: No such file or directory
oldest_awk.sh exit status: 1
find: ‘no_such_dir’: No such file or directory
oldest: 
oldest_nopf.sh exit status: 0

Both pass on the big folder, 0 failures in 100. On the missing folder, the awk version stops with status 1, because pipefail still counts find's failure. The other prints oldest: with nothing after it and exits 0, and a cleanup job reading that line carries on with an empty path. Removing pipefail trades a false failure for a false success, and the false success is the worse of the two.

Fix two: let the first command stop itself

To ask whether there is at least one match, tell find to stop at the first match with -quit. There is no pipe, so there is nothing left writing into a closed one, and the assignment keeps find's own status where set -e can see it:

#!/bin/bash
set -euo pipefail
# find stops at the first match itself, so nothing is left writing into a closed pipe.
first=$(find clips -name 'cam1_*.mp4' -print -quit)
if [ -n "$first" ]; then
  echo "cam1 clips found, e.g. $first"
else
  echo "no cam1 clips"
fi

#!/bin/bash
for n in 10 3000; do
  ./mk.sh "$n"
  wrong=0
  for k in $(seq 1 100); do ./grepq_fix.sh | grep -q 'no cam1' && wrong=$((wrong + 1)); done
  echo "clips=$n  said 'no cam1 clips' $wrong/100 times"
done

Output:

clips=10  said 'no cam1 clips' 0/100 times
clips=3000  said 'no cam1 clips' 0/100 times

Right answer on both folders, every time.

Checklist

• Search for pipefail scripts whose pipelines end in head, grep -q, grep -m or anything else that stops reading early.
• Read 141 in a log as an upstream command still writing when the reader left, not as a crash.
• Test with more output than one pipe holds, not a handful of files.
• Prefer a last command that reads to the end, or a first command that stops itself, over removing pipefail.
• Check what an if around such a pipeline does; set -e will not stop it.
• If a job acts differently under a supervisor, check whether it starts with SIGPIPE ignored.

Things I did not test

• Other shells. This is bash 5.1; other shells may lack pipefail or report signals differently.
• Other sort builds. The 4096-byte block is GNU coreutils 8.32 here. Other builds may write other sizes, which moves the bands.
• Real clips. The files are empty; only the length of the list matters here.
• cron or systemd. I did not check whether either starts jobs with SIGPIPE ignored.
• A loaded machine. I only ran on an idle one.

Sources. Quotes are from the Bash manual (Exit Status section and The Set Builtin), the Linux pipe(7) man page, and Python's subprocess documentation. Outputs are pasted unedited.

If you want an old phone recording clips in the background so there is a folder like this to clean up, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Exit 141 under pipefail means the reader left early, not that the writer broke. Fix the reader.