If you have ever run a HuggingFace model download on RunPod and come back to find your cache gone, this is why.
The default is wrong for rented GPU pods
HuggingFace sets HF_HOME to ~/.cache/huggingface when no environment variable overrides it. On a standard Linux machine that is fine your home directory persists.
On RunPod, your home directory is on the container volume, which is ephemeral. When the pod stops, it is gone.
RunPod has two storage layers: the container volume, which is ephemeral, and the network volume mounted at /workspace, which persists across pod restarts. HF_HOME defaults to the wrong one.
This means every model download, every tokeniser, every dataset you pulled via datasets.load_dataset() gone on pod restart. You pay to download them again on the next run.
On large models that is not a minor inconvenience. A 7B model download at pod startup adds minutes and real cost every single time.
The fix is one line in your startup script:
export HF_HOME=/workspace/.cache/huggingface
/workspace is RunPod’s persistent volume mount. One line and your cache survives every restart.
Why this is easy to miss
There is no error message. HuggingFace downloads the files successfully to the ephemeral path. Your training run completes. The problem only surfaces when you restart the pod and the download happens again and even then it looks like a slow startup, not a misconfiguration.
Silent failures are the hardest to debug because you are not looking for them.
Automating the check
If you want this caught automatically before every job rather than relying on remembering to set the variable, ComputeFence checks it as part of its pre-flight scan:
uvx computefence doctor
It flags HF_HOME pointing to ephemeral storage and prints the exact export command for your platform. It also checks GPU visibility, Accelerate config, checkpoint directory persistence, and disk headroom the other silent failures that cost compute budget without a log entry.
The two-line pre-flight for every RunPod job
export HF_HOME=/workspace/.cache/huggingface
uvx computefence doctor
The first line fixes the cache. The second confirms everything else is configured correctly before the GPU bill starts.
GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence
Top comments (2)
This is the kind of failure mode that never shows up in benchmarks but quietly eats real money. The point about it being silent is what stings most — a re-download masquerading as a slow startup is exactly the kind of thing you'd never catch in logs. For anyone serving models with vLLM on rented pods, the same trap applies to LoRA adapters and tokenizer files too; pointing the cache at the persistent volume should honestly be the platform default.
Exactly right on the vLLM and LoRA adapter point I had not thought about that specific case but it is the same failure mode one level deeper.
The adapter files download to the default cache, pod stops, they are gone, next run re-downloads silently.
The platform defaulting to persistent storage would solve it but until that happens the export line needs to be in every startup script.
Appreciate you flagging that the vLLM and adapter case is exactly the kind of real-world variant I want to document.