DEV Community

Cover image for I Built slothd: A Local-First, Offline-Capable AI Daemon That Patiently Organizes Your Messy Workspace
Dhruv Jani
Dhruv Jani Subscriber

Posted on Edited on

I Built slothd: A Local-First, Offline-Capable AI Daemon That Patiently Organizes Your Messy Workspace

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission πŸ¦₯

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

slothd β€” local-first Β· offline-capable Β· idle AI

πŸ’‘ Quick Navigation: What I Built Β· Live Demo Β· Code Β· Two-Stage Funnel Β· Open Innovation Β· Lazy Payoff Β· Prize Tracks (9 Categories) Β· Discussion

πŸ“– Table of Contents (Click to expand)

What I Built

Last week I ran du -sh ~/Downloads on my laptop and realized I was hoarding hundreds of cryptic PDFs (INV-9041-final(1).pdf), forgotten .exe installers, and unnamed screenshotsβ€”and I refused to upload my private bank receipts or .env configs to a cloud AI just to sort them. So for International Sloth Day, I built a daemon that waits until I walk away from my keyboard and cleans the mess locally while I sleep.

Workspaces, Desktop dumps, and project inboxes are where good intentions go to hibernate. Cryptic filenames pile up (Screenshot 2026-10-18 at 2.34.12 PM.png, INV-9041-final(1).pdf, DockerDesktopInstaller.exe). Receipts, strategy documents, archives, and configuration files end up in a flat, unorganized heap because standard file extensions tell you almost nothing about what a document actually contains.

I built slothd, a local-first, offline-capable AI file organizer designed around the philosophy of "Slow and Steady":

  • Patient Idle Activation: Uses native OS APIs to measure keyboard and mouse inactivity. It sleeps while you work and only activates when you step away.
  • Two-Stage Intelligence Funnel: Fast metadata triage (1ms via local 7-signal engine, ~20ms via Prior Labs TabPFN) flags anomalies (clutter, stale installers, duplicate hashes) before invoking heavier models.
  • Local Vision & Text Comprehension: Uses Gemma 3 running locally in Ollama with strict Pydantic JSON schemas to read PDFs, invoices, and images.
  • Natural Language Semantic Search: Embeds organized files into Tiger Data pgvector (768-dim nomic-embed-text) with a local SQLite vector fallback, enabling queries like "Where is my AWS invoice from October?".
  • Crash-Resilient & Reversible: Every run is a dry-run by default. Moves use a two-phase write-ahead commit point logged to an SQLite WAL manifest with SHA-256 content hashes. An automatic startup reconciliation pass protects against power cuts, and slothd undo restores every file instantly.
  • Glassmorphic Local Dashboard: A lightweight, dependency-free web UI (FastAPI + Vanilla ES6) with live stats, move manifest inspector, interactive activity timeline, and an ElevenLabs audio briefing player.

πŸ”’ Local-First & Air-Gapped by Default: The entire core workflowβ€”idle detection, filesystem scanning, tabular metadata triage, Gemma 3 document understanding via local Ollama, SQLite WAL manifests, and 1-click rollbackβ€”executes 100% locally on your machine with zero internet calls. A genuinely air-gapped deployment simply leaves external integrations disabled. Cloud partner services (Tiger Data pgvector, SerpApi vendor lookup, ElevenLabs voice) are strictly opt-in extensions via explicit CLI flags (--enrich-online), and each is engineered with a complete offline fallback (local SQLite vector search, local heuristics, terminal audio chimes). No personal file or metadata ever leaves your device by default.

Who gets to be a little lazier because of it?

Every developer, creator, and researcher who wastes 20 minutes every week digging through unorganized files, manually dragging PDFs into dated folders, or putting off digital cleanup. You let the sloth run patiently while you make coffee, return to an inspected plan, and keep complete control.

Before and after organization

Demo

The complete interactive loop demonstrates the full lifecycle:

Messy Workspace 
  ➜ Fast Metadata Triage (TabPFN / Local Heuristic) 
  ➜ Local AI Plan (Gemma 3) 
  ➜ Dry-Run Preview 
  ➜ Explicit Approval 
  ➜ Semantic Search (Tiger Data) 
  ➜ SQLite Manifest Inspection 
  ➜ 1-Click Rollback
Enter fullscreen mode Exit fullscreen mode

35-Second Walkthrough Demonstration

slothd live walkthrough

The Offline Demo Fixture: Reproduce in 10 Seconds

You don't need to risk your own files to evaluate slothd. The repository includes a deterministic test fixture in test_workspace/messy_inbox containing an invoice, duplicate invoice, Uber receipt, project notes, Python benchmark, configuration file, leftover Windows installer, stale zip archive, and unsorted screenshots.

Anyone can clone and run the scan immediately:

git clone https://github.com/JaniDhruv/slothd.git
cd slothd
uv run slothd scan test_workspace/messy_inbox
Enter fullscreen mode Exit fullscreen mode

In under a second, you will observe the verified metrics:

  • 10 files scanned
  • 2 clutter anomalies flagged in 1 ms (via deterministic 7-feature triage engine / TabPFN schema)
  • 1 exact duplicate set detected via SHA-256 (973 B reclaimable)
  • 0 cloud API calls required (100% air-gapped isolation)

Code

The project is 100% open-source under the MIT License:

πŸ‘‰ GitHub Repository: github.com/JaniDhruv/slothd

Key Source Modules

How I Built It

A file organizer that feeds every single file into a heavy vision model is noisy, slow, and expensive. Instead, slothd uses a structured two-stage funnel:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚               Messy Target Directory                   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            β”‚
              1. Deduplicate & Extract Features
                            β”‚
                            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚     STAGE 1: Prior Labs TabPFN Metadata Triage         β”‚
β”‚  (7 tabular signals: log size, depth, ext, time, etc.) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
              β”‚                            β”‚
   Leftover installers, archives,          β”‚ Needs deeper
   screenshots, duplicate clutter          β”‚ semantic classification
              β–Ό                            β–Ό
   Flagged for cleanup          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                β”‚ STAGE 2: Gemma 3 Local β”‚
                                β”‚   (Ollama Vision/Text) β”‚
                                β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚
                                           β–Ό
                                β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                β”‚ SQLite Reversible Log  β”‚
                                β”‚  + Tiger Data pgvector β”‚
                                β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Enter fullscreen mode Exit fullscreen mode

Stage 1: Lightweight Tabular Triage (Prior Labs TabPFN)

Before running AI models, slothd extracts 7 tabular signals from the filesystem:

  1. Log-scaled file size
  2. Extension category group
  3. Path depth
  4. Hour of modification
  5. Day of week
  6. Filename length
  7. Numeric/UUID entropy signal

When TabPFN credentials or a local model checkpoint are configured, TabPFN evaluates these tabular feature distributions directly (taking ~15–25 ms). To guarantee strict zero-credential, air-gapped operation by default, slothd also bundles a calibrated deterministic heuristic engine (TabPFN-Heuristic-Local) that evaluates the exact same 7-signal feature vector in under 1 millisecond.

(Note on benchmark accuracy: The 1 ms timing reported in the deterministic test fixture reflects this local offline heuristic engine. When running against TabPFN's hosted model with credentials, inference takes ~20 ms across the batch. Both modes share the exact same Pydantic triage interface and 100% test pass rate).

TabPFN anomaly report

Stage 2: Local Vision & Content Understanding (Gemma 3)

Only files requiring semantic classification (unlabeled PDFs, scanned receipts, documents) are forwarded to local Gemma 3 via Ollama.

  • PDFs are converted to in-memory images via PyMuPDF.
  • Outputs are constrained via Pydantic schema passed directly to Ollama's format parameter. The model output is validated token-by-token at the grammar level: category, subcategory, description, date, vendor, proposed_name, confidence, and tags.
  • If Ollama is offline, a 0.2s health probe triggers an instant fallback to local extension classification, preventing the CLI from ever freezing.

Dry-run organization preview

Safe, Reversible Operations & Crash Resilience via SQLite WAL

The organization pipeline enforces strict safety, transaction ordering, and crash-resilience invariants:

  1. Non-Destructive Dry Run: slothd organize prints the plan without moving files. You must explicitly pass --go or click "Approve" in the dashboard.
  2. Collision Avoidance: If Invoices/AWS_Invoice_2026.pdf already exists, slothd automatically appends _1, _2 instead of overwriting.
  3. Two-Phase Write-Ahead Commit Point: To prevent state divergence if power cuts or a crash occurs mid-organization, execute_moves() writes an operation intent row (status='pending') to SQLite WAL before executing the disk move (shutil.move), then transitions it to status='completed'.
  4. Startup Crash Reconciliation Pass: On startup, OperationLog.reconcile_operations() verifies any uncommitted pending rows against filesystem reality (marking completed if the destination exists, aborted if the source remained) without trusting either side blindly.
  5. Content-Hash Deduplication (file_hash Index): Every organized file is indexed by SHA-256 hash in SQLite. If an organized file later leaves its folder (or is restored from a backup/archive into the inbox), slothd detects that its content hash already has an active home and skips re-running local AI models.
  6. Timestamp Preservation: Original modification times (mtime, atime) are preserved on moved files.
  7. Instant Rollback: Running slothd undo restores the entire batch in reverse order and safely deactivates the content hash.

πŸ’‘ Community-Driven Evolution: During the challenge discussions on DEV, fellow engineers raised critical systems questions regarding crash windows between disk operations and manifest commits, as well as handling files restored from external backups. We turned that feedback into shipped features: a two-phase write-ahead commit point, startup reconciliation, and content-hash tracking!

Observability: Using Sentry as an Active Debugging Loop

Sentry was not added as a decorative vanity badge; it was an active part of the developer feedback loop during development.

The integration initializes Sentry with release [email protected] and traces critical pipeline boundaries: scanner.scan_directory, ai.classify_batch, organizer.preview, organizer.execute, organizer.undo, and search.query.

Sentry Distributed Trace Waterfall

[Transaction] slothd.organize_cycle (total: 1,248 ms)
β”œβ”€ [Span] scanner.scan_directory .................... 12 ms [10 files discovered]
β”œβ”€ [Span] ai.tabular_triage (TabPFN) ................ 1 ms [2 anomalies flagged]
β”œβ”€ [Span] ai.gemma_classify_batch (Ollama) .......... 1,180 ms [Prompt: 412 tok, Resp: 86 tok]
β”œβ”€ [Span] organizer.preview_manifest ................ 3 ms [Manifest prepared]
β”œβ”€ [Span] db.wal_log_commit ......................... 4 ms [SQLite WAL sync]
└─ [Span] search.index_pgvector (Tiger Data) ........ 48 ms [768-dim embeddings stored]
Enter fullscreen mode Exit fullscreen mode

Authentic Sentry Captured Trace & Issue Evidence

Issue ID: 7805015409
Release: [email protected] (production)
Title: ConnectionError: Connection refused to 127.0.0.1:11434
Culprit: slothd.engine.classifier in classify_batch
Trace ID: 71ac0b2ff1dc4c4788ff6a5b7098e354
Span ID: a1ea6839c23ccbb5 (op: ai.classify_batch, desc: "Gemma 3 batch classify (4 files)")
Breadcrumb: [warning] slothd.engine.classifier: Ollama server is offline or unreachable. Fast-falling back to local classification.
Handled: Yes (Circuit-breaker tripped gracefully without crashing the daemon)
Enter fullscreen mode Exit fullscreen mode
Issue IDs: 7810323579 (SLOTHD-H) Β· 7810323576 (SLOTHD-G)
Release: [email protected] (production)
Title: AttributeError: 'ScannedFile' object has no attribute 'sha256'
Culprit: slothd.engine.organizer in plan_moves (line 412)
Transactions: /api/preview (Trace ID: 3645671f0cda4789b015585196dca00b, Span ID: a4a4373a984fe925)
              /api/execute (Trace ID: df3545a7e5944f429196747f961fbfdb, Span ID: a4904942490417cf)
Context: Captured live by Sentry Starlette ASGI middleware during community-driven content-hash refactoring
Handled: No (Caught in test suite and resolved in commit 8f6b7e3)
Enter fullscreen mode Exit fullscreen mode
Observed Failure in Sentry Root Cause Revealed Resolution & Test
ConnectionError on 127.0.0.1:11434 (Issue #7805015409) Ollama daemon was offline during CLI execution Added 200ms pre-flight probe, local heuristic fallback, and circuit-breaker regression test (3a327ea).
ValidationError: confidence > 1.0 Gemma 3 occasionally returned confidence as a percentage (90 or 100) instead of float 0.0–1.0 Added automatic percentage normalization validator in Pydantic schema with bounds coverage (a914b53).
RuntimeError: generator didn't stop after throw() Tracing context manager was incorrectly re-entering on caught exceptions Cleaned up tracing wrapper to propagate exceptions correctly and verified with regression tests (df19f29).
AttributeError: 'ScannedFile' has no attribute 'sha256' (Issues #7810323579, #7810323576) Community-driven content-hash deduplication referenced .sha256 instead of .content_hash in plan_moves Added @property sha256 alias on ScannedFile and defensive fallback in plan_moves, verified across all 122 tests (8f6b7e3).

Why Does Open Innovation Matter?

Personal files contain our most vulnerable data:

  • Tax returns and bank statements
  • Medical lab reports and health records
  • Invoices and payment receipts
  • Confidential code and configuration secrets

Uploading this stream of daily life to closed, centralized cloud APIs creates an unacceptable privacy tradeoff.

Open-weight AI (Gemma 3) and open-source tabular models (TabPFN) fundamentally change this dynamic. They make it possible to run state-of-the-art vision and tabular triage entirely inside the boundary of your own computer:

  • No document crosses your local network without explicit opt-in.
  • Models can be audited, swapped, or operated air-gapped without internet access.
  • Partner integrations (SerpApi online vendor grounding, ElevenLabs voice) require explicit flags (--enrich-online) and are never enabled by default.

Privacy is not a checkbox added after the architecture is done; with open-source AI, privacy is the architecture.

The Lazy Payoff

The lazy payoff is not about moving files a few milliseconds faster. It is about removing the cognitive dread of manual digital housekeeping.

Instead of spending Sunday afternoon opening dozens of PDFs one by one, deciphering cryptic filenames, and deciding where each document belongs:

  1. You let slothd daemon run in the background.
  2. It detects when you step away from your keyboard.
  3. It spots clutter anomalies and duplicates in 1 millisecond (via local tabular triage).
  4. It reads invoices and dates with Gemma 3.
  5. You return to a tidy proposed plan, hit enter to approve, and know that every single change can be undone with slothd undo.

Measured Efficiency from the Test Fixture

10 files scanned
2 clutter anomalies identified
1 duplicate set found
973 B reclaimable disk space
1 ms metadata triage (local deterministic 7-signal engine / TabPFN schema)
0 cloud API calls required
100% offline isolation
Enter fullscreen mode Exit fullscreen mode

Prize Categories

Architectural Depth & Ecosystem Evidence

While slothd operates completely offline out of the box, it was engineered with a modular architecture that cleanly interfaces with specialized ecosystem tools. Rather than a superficial checklist of sponsor imports, each partner integration represents an optional, verified subsystem that enhances capability without compromising our local-first core:

⏱️ 10-Second Judge Navigation Matrix

Track How slothd Uses It Source Module
Google Gemma Local gemma3:4b vision + text via Ollama with strict Pydantic JSON outputs classifier.py
Prior Labs TabPFN 1ms 7-feature tabular triage flagging stale archives & size outliers before LLM wakeup tabular_predictor.py
Tiger Data 768-dim pgvector semantic search (<=>) + hybrid keyword re-ranking tiger_data.py
Sentry Tracing Distributed spans (slothd.scan, predict, classify, search) tracking latency & tokens sentry_tracing.py
SerpApi Opt-in --enrich-online live web grounding for cryptic filenames & unknown vendors serpapi_client.py
ElevenLabs slothd briefing Sloan-the-Sloth voice summary via eleven_multilingual_v2 elevenlabs.py
GitHub Copilot Cross-OS CI matrix (Ubuntu/Windows, Py 3.11/3.12) + .github/copilot-instructions.md ci.yml
Entire Full agent session provenance & Git checkpoints via .entire/settings.json .entire/

1. πŸ¦₯ Grand Prize: Hacktoberfest Week 3 "Slow and Steady"

  • Concept: Embodying International Sloth Day with native OS idle measurement (Win32 GetLastInputInfo, macOS IOHIDSystem, Linux xprintidle) that sleeps during work and activates only after 5+ minutes of continuous inactivity. All actions are dry-run by default and 100% reversible via SQLite WAL manifests.

2. πŸ’Ž Featured Track: Best Use of Gemma

  • Module: slothd/engine/classifier.py
  • Design: Deep local document comprehension using gemma3:4b running in Ollama. PyMuPDF renders PDF pages in-memory for visual token ingestion, constrained token-by-token with strict Pydantic JSON grammar schemas to eliminate hallucinations.

3. πŸ“Š Featured Track: Best Use of TabPFN

4. 🐯 Partner Track: Best Use of Tiger Data

5. πŸ”­ Partner Track: Best Use of Sentry Agent Tracing

  • Module: slothd/integrations/sentry_tracing.py
  • Design: Distributed span tracing across scanner, AI triage, and execution boundaries. Verified with authentic production traces (Issue #7805015409, Issue #7810323579, trace_id: 3645671f0cda4789b015585196dca00b) that resolved real development regressions in real time (3a327ea, a914b53, df19f29, 8f6b7e3).

6. πŸ” Partner Track: Best Use of SerpApi

  • Module: slothd/integrations/serpapi_client.py
  • Design: Opt-in --enrich-online live web grounding for mysterious software names and vendor IDs, locally cached with zero transmission of private document bodies.

7. πŸŽ™οΈ Partner Track: Best Use of ElevenLabs

  • Module: slothd/integrations/elevenlabs.py
  • Design: Sloan the Sloth witty audio summaries via eleven_multilingual_v2 (slothd briefing), featuring local offline chime fallbacks when no API key is configured.

8. πŸ€– Partner Track: Best Use of GitHub Copilot

9. πŸ“¦ Partner Track: Best Use of Entire

  • Module: .entire/settings.json & .agents/hooks.json
  • Design: Integrates Entire CLI 0.11.6 to maintain immutable agent session provenance, tool checkpoints, and commit logs directly in version control.

Verification & Testing

Every test runs in total offline isolation (no live external services required):

uv run pytest -v
# 122 passed in 31.21s (including crash reconciliation & hash tracking)

uv run ruff check .
# All checks passed!
Enter fullscreen mode Exit fullscreen mode

πŸ¦₯ Parting Sloth Thoughts: Reclaiming Digital Calm

We spend so much time micromanaging our digital clutterβ€”manually dragging files into folders, hunting for lost receipts, or feeling guilty about a desktop covered in final_v2_really_final.pdf.

slothd was built on a very simple premise: your machine should quietly work for you when you're resting, not steal your attention when you're busy.

By embracing open-weights, deterministic rule fallbacks, and respectful idle scheduling, you don't have to surrender your personal documents to cloud silos or burn your laptop's battery just to keep a clean workspace. Take a breath, close your laptop lid, and let the sloth do the heavy lifting while you take a break.

πŸ’¬ Community Discussion

I'd love to hear your thoughts and feedback:

  1. What is the single oldest or most chaotic file currently buried in your messy folders?
  2. What custom classification heuristics or local AI pipelines would you like to see supported next?
  3. If you run slothd on your workspace, how many megabytes of duplicate clutter did you reclaim?

If slothd brought a little bit of calm to your daily workflow, consider dropping a ⭐ on the GitHub repository, leaving a comment below, and sharing your experience! Happy Hacktoberfest! πŸ‚πŸ¦₯


Top comments (41)

Collapse
 
dj29 profile image
Dhruv Jani •

πŸ¦₯ A little more context behind slothd!

I wanted to build something that could quietly take care of digital clutter without constantly demanding my attention or sending my personal files to the cloud.

This is my Week 3 submission for the Hacktoberfest Open-Source AI Challenge, and I had a lot of fun bringing together local AI, idle-aware automation, and reversible file operations.

The 35-second demo and offline fixture are included in the article if you'd like to see it in action!

Now I'm curious: What's the messiest folder on your machine, and would you trust a local AI assistant to organize it? πŸ‘€

Happy Hacktoberfest! πŸ‚πŸ¦₯

Collapse
 
rytjy profile image
rytjy •

Local-first daemon + local model is the right shape, but daemons have a failure mode a chat window doesn't: the model call is slow and the file events keep arriving.

How do you handle the backlog β€” a bounded queue you drain, or drop-and-rescan?

I ask because I measured a 1.5B answering four narrow yes/no checks on CPU at roughly 48 seconds. At that latency a file watcher can stack a couple of dozen pending events inside a minute, and the interesting question becomes which ones you're willing to skip.

Collapse
 
dj29 profile image
Dhruv Jani •

Definitely drop-and-rescan on idle β€” trying to hook a local model to a live inotify event queue is basically suicide.

If you run npm install or extract a zip, a reactive file watcher immediately drowns the model. So slothd doesn't listen to continuous file event streams at all. It just checks OS idle time periodically, and only runs a fresh directory scan once you've stepped away from your keyboard for 5+ minutes.

When it wakes up, it deals with that CPU latency in two ways:

  1. Which ones we skip: ~80% of files never touch the LLM. Installers (.exe/.dmg), archives (.zip), code, and exact SHA-256 duplicates are sorted or pruned instantly (<1ms) by metadata triage. Gemma only ever gets invoked for truly ambiguous files (cryptic PDFs, unnamed receipts/screenshots).
  2. Bounded slices: It only grabs 20 files per cycle. If you touch your mouse or keyboard, it yields immediately so it never steals your CPU while you're working.

So instead of an exploding FIFO queue, the backlog just quietly chips away across your breaks.

Collapse
 
rytjy profile image
rytjy •

Drop-and-rescan on idle explains the backlog, and the 80% metadata triage is the part I'd steal. One thing I'm still curious about: with a 20-file cycle and ambiguous files costing tens of seconds each, a 200-file ambiguous backlog needs ~8 minutes of idle β€” does the next idle window resume where it stopped, or does every window re-rank and risk starving the same low-priority tail forever?

Thread Thread
 
dj29 profile image
Dhruv Jani •

It doesn't starve the tail because the files are physically moved upon batch execution!

Once a 20-file batch is organized, those files are moved into their categorized folders and logged to SQLite WAL. On the next idle wake-up, scan_directory(skip_organized=True) ignores the moved files. So your 200-file backlog shrinks to 180 β†’ 160 β†’ 140 until it hits zero. The filesystem itself acts as the persistent queue.

And you don't even have to wait for idle if you want it done right now:

  • Run manual on-demand: slothd organize ~/Downloads --go --ai (or hit "Approve" in the web dashboard).
  • Or set --idle-threshold 0 on the daemon to just plow through continuously.

Idle mode is just the polite default so it never cooks your laptop while you're actively typing.

Thread Thread
 
robertpaschal profile image
Odinaka Robert Nnamani •

The filesystem-as-queue is the right instinct β€” the disk is the one ledger you can't lose β€” but it answers the starvation question, not the crash one. My 3ima1 question still stands: if slothd dies between the move and the WAL append, the file is organized on disk but invisible to the manifest, so the next scan re-organizes a file that already has a home. The fix I'd steal from document pipelines: write the manifest entry as PENDING before the move, flip to COMMITTED after β€” on restart, anything still PENDING gets verified against disk instead of trusted. Do you fsync the WAL before a batch counts as done?

Thread Thread
 
rytjy profile image
rytjy • • Edited

@dj29 β€” one follow-up on the skip_organized path, and it runs into Robert's point below from the other side.
Files-as-queue is a better commit point than a queue row, and moving the file explains why the tail cannot starve. The case I would still poke at is the inverse: if an organized file later leaves its folder β€” restored from a backup, renamed, moved by another tool β€” does skip_organized=True treat it as new and re-run the model on it, or does the manifest match on content hash and skip it?

The second one is what makes undo trustworthy; the first one quietly re-organizes a file that already has a home.

Robert's crash window and this are the same missing check from opposite directions β€” disk organized / manifest silent, and disk gone / manifest organized. Both want one reconcile pass that trusts neither side on its own, which is why I would make the manifest write the commit point rather than the move.

Thread Thread
 
dj29 profile image
Dhruv Jani • • Edited

You made such a sharp observation that I went ahead and implemented both features! πŸ”₯

Here is what we just added and tested:

  1. Write-Ahead Intent Commit (Crash-Safe Commit Point):
    execute_moves now writes an operation intent row with status='pending' into the SQLite WAL before executing the disk move (shutil.move), then transitions it to status='completed'.
    On startup, OperationLog automatically runs a reconcile_operations() pass:

    • If destination exists on disk: move succeeded before crash $\rightarrow$ marks completed.
    • If source still exists on disk: move never completed $\rightarrow$ marks aborted. Neither side is trusted blindly.
  2. Content-Hash Manifest Tracking (file_hash in SQLite):
    Every organized file now logs its SHA-256 hash in SQLite alongside its destination.
    If an organized file is restored from a backup or copied back into the inbox by another tool, scan_directory(known_hashes=...) and plan_moves(ops_log=...) detect that its content hash already has an active organized home on disk and skip it immediately β€” zero wasted AI inferences and no duplicate re-filing! (And if you run slothd undo, the hash is deactivated so it can be re-organized).

  3. 122 unit tests passing with dedicated test coverage for startup crash reconciliation and restored-file hash skipping.

Really appreciate you digging into the transaction plumbing β€” you genuinely helped make slothd significantly more robust! πŸ¦₯

But my friend, this is the limit I can go, I'm still a final year college student and I do appreciates critiques and feedbacks but I'm preparing for last final/external exam of my college. So, I won't get much time. But keep the feedbacks coming in! I'm learning a lot from it!

Have A Great Day!

Thread Thread
 
robertpaschal profile image
Odinaka Robert Nnamani •

That is a solid pair of fixes, and I especially like that reconcile does not trust either side blindly. The one edge case I would still test is when both the source and the destination exist after a crash, for example a copy that got interrupted after the destination was written but before the source was removed. If destination-exists wins and marks the move completed, does the leftover source get cleaned up, or could it be re-organized later as a duplicate? The same question applies to the hash check: if the destination file was edited after it landed, its hash no longer matches the manifest, so a restored copy would not be skipped. And given your exams, I would not expand this further right now β€” is that both-exist case already covered by your 122 tests, or is it the next one you would add when you have time again?

Thread Thread
 
dj29 profile image
Dhruv Jani •

Definitely the next one to add once exams wrap up! πŸ˜„

Right now if both exist after a crash, destination-exists wins and marks it completed. On the next cycle, Stage 1 hash triage catches the leftover source as a duplicate and skips it so it doesn't get re-processed. But having reconcile_operations() explicitly clean up the stranded source file is 100% the right polish for v0.2.

Really appreciate the thoughtful review and the kind words on exams! πŸ¦₯

I'll definitely circle back to this after my final exams are over.
Have A Great Day!

Thread Thread
 
rytjy profile image
rytjy •

Two notes, no reply needed until your exams are over.

The reconcile covers dest-exists and source-exists. There is a third state: neither exists β€” the move completed, then the file was deleted or moved away by something else before the next startup. Neither rule matches, so the intent row stays pending forever (or gets swept as aborted) while the truth is "organized, then removed by someone else". Marking that unknown and surfacing it is cheaper than guessing.

The second one is Robert's question from the storage side: a pending row in WAL is only a crash-safe commit point if that commit reached the disk. WAL mode is usually run with synchronous=NORMAL, where a power loss can drop the last transactions after the move already happened β€” which is exactly the window the intent row is meant to close. synchronous=FULL on that one write, or an explicit fsync before shutil.move, is what makes the ordering real rather than nominal.

The hash manifest is the right shape, and wiring undo to deactivate the hash is the part I would have missed. Good luck with the finals.

Thread Thread
 
dj29 profile image
Dhruv Jani •

Haha thanks so much for the deep dive and the well wishes on finals! Really appreciate the feedback on fsync and edge states for v0.2!
I'll definitely circle back to this once I'm done with my finals!
Have A Great Day!

Collapse
 
mohsen_jr profile image
Mohsen jr •

The dry-run and undo features are what make this practical to trust with real files. AI can make mistakes, so being able to review changes before applying them is just as important as the organization itself.

Collapse
 
saeediphone profile image
Saeed iPhone •

The two-stage approach is especially smart; using lightweight triage before invoking a local vision model keeps the workflow efficient, while dry-run previews and rollback make automation much safer. I also like the privacy-first design. How well does Gemma handle ambiguous files with misleading filenames or mixed content?

Collapse
 
dj29 profile image
Dhruv Jani •

Thanks so much for reading and for the great question, Saeed!

That was actually one of the core edge cases we tested during development. Here is how slothd and Gemma 3 handle misleading filenames and mixed/ambiguous content:

  1. Content Always Overrules the Filename:

    When a file hits Stage 2, Gemma doesn't rely on the filename to guess what it is. For PDFs, PyMuPDF extracts the raw text body (or, if it's a scanned document with <50 characters of text, renders the first page into an in-memory PNG at 2x zoom). For screenshots and receipts, the raw image bytes are passed directly to Gemma 3 Vision. Because visual tokens and document text take precedence over the filename string, a cryptic file like Screenshot_2026-10-18.png or INV-9041-final(1).pdf gets identified by its actual header, logo, and line items (e.g., Uber Receipt or AWS Invoice), generating a clean proposed name like Uber_Receipt_2026-10-18.pdf.

  2. Mixed Content & Multi-Label Semantic Tagging:

    Real-world documents are rarely clean-cut (e.g., an invoice that is also a legal contract). Our Pydantic schema forces Gemma to emit three layers of categorization:

    • A broad category (e.g., Documents)
    • A granular subcategory (e.g., Service Agreement & Billing)
    • A list of tags (e.g., ["aws", "cloud", "invoice", "legal", "q3-taxes"])

These tags and summaries are indexed into Tiger Data pgvector (with local SQLite vector fallback). Even if a mixed document ends up filed in Documents/2026/October/, running slothd search "cloud hosting bill" or querying via the dashboard immediately surfaces it based on vector similarity.

  1. Graceful Fallbacks & The Confidence Safety Net: The token generation is strictly constrained by a Pydantic grammar schema. If a file is completely ambiguous or corrupted, Gemma outputs null for vendor/date and assigns a low confidence score. When ai_proposed_name is null, slothd strictly preserves the original filename and falls back to extension clustering.

And most importantly: because every batch runs as a non-destructive dry run first with human approval and SQLite WAL logging, an ambiguous file's destination can be inspected before moving, or instantly reversed with slothd undo! πŸ¦₯

Collapse
 
saeediphone profile image
Saeed iPhone •

**That’s a solid approach! **Prioritizing actual document content, using multi-label tagging, and preserving ambiguous files rather than guessing makes the system much more reliable. The human approval and rollback safeguards are especially important for building trust in local AI automation. πŸ˜ƒ

Collapse
 
tanay_dwivedi9098 profile image
Tanay Dwivedi •

Hey nice project @dj29 . This is very useful for me to clear out digital clutter, as my digital files aren't a mess but a disaster πŸ’€

Collapse
 
dj29 profile image
Dhruv Jani •

Thanks Tanay! I'd be glad if you could run it and tell me your experience. And don't worry it has batch rollback too, so if you didn't like how slothd organized your files, you can batch rollback it.

Have A Great Day!πŸ˜„

Collapse
 
amintacccp profile image
tamina •

The two-stage funnel is the part I keep coming back to β€” cheap metadata triage first, and only waking the vision model for files that actually need semantic reading. That, plus dry-run by default and the SQLite WAL undo log, is exactly the safety story an auto-organizer needs before I would let it loose on my Downloads folder. The confidence-as-percentage bug (Gemma returning 90 instead of 0.9) is a great catch too β€” that kind of quiet failure is nasty in production.

Collapse
 
joemcmahon profile image
Joe McMahon •

Fantastic idea, and great implementation. I can see ways one could extend this -- it definitely has its own idea of "this is organized" that might not fit everything -- but the base idea of "nothing ever leaves my machine to be categorized" is great and probably something we're all going to have to tackle one way or another as agents become more pervasive and accepted.

Teaching it to look at some files that are organized and filing into that structure would be very nice indeed. I was hoping it could do that out of the box, but it's very useful as it is.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

One I'd save for after finals i.e. edit an organized file, then try undoing that batch. I'd want undo to flag the changed content and leave it intact rather than quietly treat it as the original. This fault-tolerant work-queue exercise approaches the same ownership problem from another angle i.e. finishing an operation and still having authority to change its result are separate things.

Collapse
 
dj29 profile image
Dhruv Jani •

Sure is, and I'll definitely go into these edge cases once I don't have to worry about exams anymore.

Collapse
 
yug_vasava profile image
Yug Vasava •

Great Bro, keep it up and All The Best!

Collapse
 
dj29 profile image
Dhruv Jani •

Thanks Bro! Give it a try and gimme feedback when you're free!

Collapse
 
yug_vasava profile image
Yug Vasava •

Sure Bro.

Collapse
 
vsamukov profile image
Vladimir Samukov •

Nice work. Defaulting to dry-run with an undo manifest, and rescanning on idle instead of hooking the model to inotify, are the right calls. The back-and-forth on crash safety in this thread is great to see.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.