GNU tar Exit Status 1 on a Folder That Is Still Being Written: Zero-Padded Clips, set -e, and || true

A nightly tar of a folder of camera clips can finish, write an archive, and still hand back a clip whose tail is nothing but zero bytes. GNU tar does tell you. It tells you with exit status 1, and exit status 1 is the one most backup scripts handle badly: set -e treats it as a crash, and || true treats it as nothing.

This post is about that middle status, what produces it when the folder is still being written to, and how to keep the files that are still open out of the archive in the first place.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it stores files. Every result comes from the scripts shown, run once on this version:

tar (GNU tar) 1.34

The three exit statuses

The GNU tar manual lists three outcomes: "Successful termination", "Some files differ", and "Fatal error". For an archive being created, the middle one means "some files were changed while being archived and so the resulting archive does not contain the exact copy of the file set". In the runs below, a missing input gave status 2 and a file changing under tar gave status 1.

That middle status is not an error and not a success. It means the archive exists and something in it may not be what you think. A script that collapses it into either of the other two is wrong some of the time.

How the tests fake a recorder

Getting tar to read a file at the exact moment another process writes to it is a race, and races make bad demos. GNU tar has a checkpoint hook, --checkpoint=N --checkpoint-action=exec=..., which runs a command every N records while tar works. I used it to append to, truncate, or delete a clip part way through the run. It stands in for a recorder that has not closed its file yet. It is not a recorder, and real timing will differ.

The setup makes three clips of 3000000 bytes and backdates them by 20 minutes:

#!/bin/bash
# Three finished clips, sizes fixed so every run prints the same numbers.
rm -rf /tmp/tar_demo && mkdir -p /tmp/tar_demo/clips /tmp/tar_demo/backup
cd /tmp/tar_demo
for n in 1 2 3; do head -c 3000000 /dev/urandom > clips/clip$n.mp4; done
touch -d '20 minutes ago' clips/*.mp4
ls clips

Test one: a clip that grows during the run

The nightly job runs under set -e, so any non-zero status stops it. After tar, the script would upload the archive and prune old ones.

#!/bin/bash
# The nightly job, run with set -e. The checkpoint hook stands in for a
# recorder that is still appending to clip3.mp4 while tar reads it.
set -e
cd /tmp/tar_demo
tar czf backup/clips.tgz \
  --checkpoint=50 \
  --checkpoint-action=exec='sh -c "head -c 1000 /dev/urandom >> clips/clip3.mp4"' \
  clips
echo "upload and prune would run here"

Run it and compare clip3 in the archive with clip3 on disk:

#!/bin/bash
HERE=$(cd "$(dirname "$0")" && pwd)
cd /tmp/tar_demo
bash "$HERE/t1_changed.sh"; echo "job exit status: $?"
echo "--- clip3 in the archive vs on disk (bytes)"
tar tvzf backup/clips.tgz clips/clip3.mp4 | awk '{print "archive:", $3}'
stat -c 'disk:    %s' clips/clip3.mp4

Output:

tar: clips/clip3.mp4: file changed as we read it
job exit status: 1
--- clip3 in the archive vs on disk (bytes)
archive: 3005000
disk:    3017000

The upload line did not print. set -e stopped the job at status 1, so tonight's archive was written and then not sent anywhere, and the old archives were not pruned. If the job's alerting checks whether an upload happened, this looks the same as a dead disk.

The archive also holds a shorter clip3 than the disk does. tar recorded the size when it reached the file and read that many bytes. Anything appended after that is simply not in tonight's archive. For a clip that is still recording, that is expected. The point is that status 1 is the one signal that it happened.

Test two: a clip that shrinks during the run

The worse case is a file that gets shorter while tar is reading it. Here the hook truncates live.mp4 to 1000000 bytes once, at checkpoint 100, after tar has started on it.

#!/bin/bash
# Truncate live.mp4 once, at checkpoint 100, while tar is part way through it.
cd /tmp/tar_demo
head -c 3000000 /dev/urandom > clips/live.mp4
tar cf backup/live.tar --checkpoint=1 \
  --checkpoint-action=exec='sh -c "[ \$TAR_CHECKPOINT = 100 ] && truncate -s 1000000 clips/live.mp4; true"' \
  clips/live.mp4
echo "exit status: $?"
rm -rf restore && mkdir restore && tar xf backup/live.tar -C restore
stat -c 'restored size: %s' restore/clips/live.mp4
printf 'non-zero bytes in the last 1000000: '
tail -c 1000000 restore/clips/live.mp4 | tr -d '\0' | wc -c
rm clips/live.mp4

Output:

tar: clips/live.mp4: File shrank by 1976512 bytes; padding with zeros
exit status: 1
restored size: 3000000
non-zero bytes in the last 1000000: 0

tar had already promised the archive a 3000000-byte file in the header, so it kept that promise by padding with zeros. The restored file is the original size. Its last 1000000 bytes contain no non-zero byte at all. A media player may open it and stop early, or may not open it. Either way, the restore looks complete by size, and only the status and a line on standard error say otherwise.

This is also status 1, the same as test one. The status does not tell you which of the two happened. Only the message does.

Test three: a clip deleted during the run

A prune job or a recorder that rotates files can remove a clip after tar has listed the folder and before it opens the file. The status depends on how the file got onto tar's command line.

#!/bin/bash
cd /tmp/tar_demo
reset() { for n in 1 2 3; do head -c 3000000 /dev/urandom > clips/clip$n.mp4; done; }
reset
echo "--- folder named, clip3 deleted mid-run"
tar czf backup/r1.tgz --checkpoint=50 --checkpoint-action=exec='rm -f clips/clip3.mp4' clips
echo "exit status: $?"
reset
echo "--- same files fed as a list on standard input"
printf 'clips/clip1.mp4\0clips/clip2.mp4\0clips/clip3.mp4\0' |
  tar czf backup/r3.tgz --null -T - --checkpoint=50 --checkpoint-action=exec='rm -f clips/clip3.mp4'
echo "exit status: $?"
reset
echo "--- same files named one by one"
tar czf backup/r2.tgz --checkpoint=50 --checkpoint-action=exec='rm -f clips/clip3.mp4' \
  clips/clip1.mp4 clips/clip2.mp4 clips/clip3.mp4
echo "exit status: $?"
reset

Output:

--- folder named, clip3 deleted mid-run
tar: clips/clip3.mp4: File removed before we read it
tar: clips: file changed as we read it
exit status: 1
--- same files fed as a list on standard input
tar: clips/clip3.mp4: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
exit status: 2
--- same files named one by one
tar: clips/clip3.mp4: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
exit status: 2

Name the folder and a vanished file is status 1, with the folder itself reported as changed. Feed the files as a list, which is what a find pipeline does, or name them one by one, and the same deletion is status 2 with Exiting with failure status. Two scripts that archive the same clips can disagree about whether the same night was a failure.

Test four: turning the warning off

The usual next step is to silence the message. On this version:

#!/bin/bash
cd /tmp/tar_demo
HOOK='sh -c "head -c 1000 /dev/urandom >> clips/clip3.mp4"'
echo "--- --warning=no-file-changed"
tar czf backup/w1.tgz --warning=no-file-changed --checkpoint=50 --checkpoint-action=exec="$HOOK" clips
echo "exit status: $?"
echo "--- --ignore-failed-read --warning=no-file-changed"
tar czf backup/w2.tgz --ignore-failed-read --warning=no-file-changed --checkpoint=50 --checkpoint-action=exec="$HOOK" clips
echo "exit status: $?"

Output:

--- --warning=no-file-changed
exit status: 1
--- --ignore-failed-read --warning=no-file-changed
exit status: 1

Both runs printed nothing and both returned status 1. The first switch removed the one line that said which file changed, and the status stayed. The second switch did not change the status here either.

The manual's warnings page says the file-changed keyword "applies only if used together with the ‘--ignore-failed-read’ option". The tar 1.35 release notes add a fix: "Fix --ignore-failed-read to ignore file-changed read errors as far as exit status is concerned." I tested 1.34, so on 1.35 or later the second run may return 0. I did not check. That is a reason to be careful: on a newer tar, that pair of switches could hide a changed or zero-padded clip completely, status included.

The same warnings page also says warning messages "do not affect the exit code of tar command". On this version, the file-changed case did. I would trust the status you observe on your own tar over either sentence.

Test five: the || true fix

The other common fix is to stop set -e from firing at all. Here the folder name has a typo.

#!/bin/bash
# The common "fix": swallow the status. The folder name is misspelled.
set -e
cd /tmp/tar_demo
tar czf backup/nightly.tgz clipz || true
echo "job carried on"
stat -c 'archive size: %s bytes' backup/nightly.tgz
tar tzf backup/nightly.tgz | wc -l

Output:

tar: clipz: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
job carried on
archive size: 45 bytes
0

|| true hides status 2 as well as status 1. The job carried on with a 45-byte archive holding no files, which would then be uploaded and would count as the newest backup when the pruning step runs.

What I would run instead

Two changes. First, keep files that are still being written out of the archive, so status 1 becomes rare instead of nightly. Second, handle the three statuses as three different things.

#!/bin/bash
# Archive only clips nobody has written to for 10 minutes, then treat
# tar's three outcomes as three different things.
cd /tmp/tar_demo || exit 2
out="backup/clips-$(date +%F).tgz"
find clips -type f -name '*.mp4' -mmin +10 -print0 |
  tar czf "$out" --null -T -
st=${PIPESTATUS[1]}
case $st in
  0) echo "archive complete: $(tar tzf "$out" | wc -l) clips" ;;
  1) echo "WARNING: a clip changed during the run, archive kept, check it" >&2 ;;
  *) echo "FAILED with status $st, archive discarded" >&2; rm -f "$out"; exit "$st" ;;
esac

The -mmin +10 test picks only clips nobody has modified in the last 10 minutes. -print0 with --null -T - hands tar the exact names, spaces and all. The case reads tar's own status from PIPESTATUS, not the status of find. Because the names arrive as a list, a clip deleted after find saw it is a status 2 (test three), and the script throws that archive away rather than keeping a partial one.

The test adds a fresh live.mp4 next to the three settled clips:

#!/bin/bash
HERE=$(cd "$(dirname "$0")" && pwd)
cd /tmp/tar_demo
for n in 1 2 3; do head -c 3000000 /dev/urandom > clips/clip$n.mp4; done
touch -d '20 minutes ago' clips/*.mp4
head -c 3000000 /dev/urandom > clips/live.mp4   # still being written
bash "$HERE/nightly.sh"; echo "exit status: $?"
tar tzf backup/clips-*.tgz

Output:

archive complete: 3 clips
exit status: 0
clips/clip2.mp4
clips/clip3.mp4
clips/clip1.mp4

Three clips archived, live.mp4 skipped, status 0. It goes into tomorrow night's archive, once nothing has written to it for 10 minutes.

Pick the number of minutes from your own recorder. It has to be longer than the longest gap between two writes to an open clip. If the recorder writes rarely, 10 is too short.

Checklist

• Do not run a nightly tar under bare set -e without handling status 1 yourself.
• Do not append || true to it. That also swallows status 2.
• Keep the message. --warning=no-file-changed removes the one line that names the file.
• Archive files that have stopped changing, chosen by age or by a rename the writer does when it closes the file.
• Read the message on status 1. file changed and File shrank are different problems with the same status.
• Restore one archive now and then and play a clip from the end of it, not the start.

Things I did not test

• tar 1.35 or later. The --ignore-failed-read change is from its release notes, not from a run.
• A real recorder. The hook appends, truncates or deletes at a checkpoint, which is not how any real writer times its writes.
• bsdtar, BusyBox tar, or tar on a phone. Their statuses and messages may differ.
• Compression failures. The manual says a failed compressor makes tar exit with that program's status. I did not trigger one.
• Network or FUSE-mounted folders. Not tried.

Sources. Quotes are from the GNU tar manual (single-page version and its warnings section) and the tar 1.35 release announcement. Outputs are pasted unedited.

If you want an old phone recording clips in the background so there is a folder like this to back up, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Status 1 is tar saying the archive exists but is not the folder. Read it as exactly that.