The attacker deleted everything. At 2 a.m., /proc still remembered.

The incident below is a fictional scenario (all IPs are from the RFC 5737 documentation ranges). The techniques are real. Only run them on systems you own or are authorized to examine.

"Delete the file and the evidence is gone." Attackers believe it. Plenty of investigators quietly do too.

If you've ever closed a case because the logs were empty, the history was wiped and no suspicious file was left on disk, this one is for you. From exactly that state, you can sometimes get the deleted malware back, intact.

What you'll learn

• Why a file you rm'd is not actually gone
• How to find a deleted-but-still-running process with one line
• How to recover the binary byte for byte and keep it admissible as evidence
• How to reconstruct execution even after .bash_history is wiped

Written for SOC analysts, incident responders, sysadmins and CTF forensics players. If you know basic Linux commands, you can follow along.

The story at a glance

Let's start at 02:41.

02:41 — An empty /dev/shm

The alert fired at 2:41 a.m. Our public web server, web03, was sending small, perfectly regular requests to 198.51.100.5. That is what C2 beaconing looks like.

I logged in and checked the usual drop spots.

$ ls -la /dev/shm /tmp
/dev/shm:
total 0
drwxrwxrwt  2 root root   40 Oct 16 02:41 .
drwxr-xr-x 18 root root 3980 Oct 16 00:12 ..

Nothing. .bash_history was empty too. If I had stopped here, this case would have ended as "cause unknown."

The one thing the attacker didn't know

The attacker's logic was sound: drop a payload in /dev/shm, run it, rm it, wipe the history. Disk scanners see nothing.

But rm doesn't erase data. It removes a name (a directory entry). The inode and its bytes live on for as long as anything still holds them open, and a running process holds its own executable.

[directory entry] ──┐
   ↑ cut by rm       ├─→ [inode] ─→ [the actual bytes]
[running process] ───┘
   ↑ not cut

The moment the attacker ran the payload, they handed their evidence to the kernel.

02:47 — Hunting the "deleted" process

ls -l /proc/*/exe 2>/dev/null | grep '(deleted)'

lrwxrwxrwx 1 www-data www-data 0 Oct 16 02:45 /proc/2411/exe -> '/dev/shm/.update (deleted)'

There it was: PID 2411, running as www-data, executable already deleted. The attacker couldn't kill it without dropping their C2 connection, so they left it running.

02:52 — Getting the file back, byte for byte

/proc/<PID>/exe is more than a symlink. It is a direct path to the executable the kernel is still holding.

mkdir -p /evidence
cp /proc/2411/exe /evidence/recovered_payload.bin
sha256sum /evidence/recovered_payload.bin | tee /evidence/recovered_payload.sha256

Hash it the moment you acquire it. That hash is your chain of custody: proof the evidence hasn't changed since you collected it.

03:05 — Capture everything else about the process

P=/proc/2411; D=/evidence/proc_2411; mkdir -p $D
tr '\0' ' ' < $P/cmdline > $D/cmdline.txt   # the real command line
cp $P/maps    $D/maps.txt                    # what code is loaded
ls -l $P/fd > $D/fd.txt                      # open files and sockets
ls -l $P/cwd                                 # where it runs from

fd showed an open socket to 198.51.100.5:443. The payload, its destination, and its launch details all came out of something the attacker believed was gone.

Order of volatility matters. Don't kill the process or reboot first. The moment you do, this evidence is gone for good. Collect first.

The history was empty. The record of execution wasn't.

.bash_history is a self-report the shell writes on exit, trivial to wipe. auditd is the kernel recording executions directly.

ausearch -m EXECVE -ts today | grep -E 'dev/shm|\.update'

The attacker erased their diary, not the kernel's log.

What the attacker thought they deleted vs. what remained

My favorite row is the last one: the attempt to hide becomes the clue.

A one-line detection

ls -l /proc/*/exe 2>/dev/null | grep -q '(deleted)' && \
  echo "ALERT: deleted-but-running executable on $(hostname)"

It can fire on legitimate software right after a package update, so correlate it with execution from /dev/shm or /tmp.

Takeaways

• rm removes a name; a running process keeps the bytes.
• Hunt with ls -l /proc/*/exe | grep deleted, recover with cp /proc/<PID>/exe.
• Hash on acquisition. Collect volatile evidence before you kill anything.
• Shell history is self-reported; auditd is the kernel's record.

Attackers only need one way in. Defenders get a quiet advantage in return: even the act of covering tracks leaves a trace.

This is adapted from Scenario A of my Japanese book on server defense for SOC, CSIRT and forensics (40 chapters, Windows and Linux, MITRE ATT&CK-mapped, with a laptop-sized lab).