I have an MCP server that digests a GitHub repository into markdown so a model can read it without cloning anything. Directory tree, plus the contents of the files that matter.

I pointed it at a real repository and got this back:

Error: result (109,668 characters) exceeds maximum allowed tokens

Not a truncated answer. Not a partial digest with a warning. Nothing. The tool call failed and the caller was left with an error message where a repository summary should have been.

The cause was a constant I had written months earlier without thinking about it:

MAX_FILE_BYTES = 32_000
TOP_N_FILES = 10

Ten files, each capped at 32,000 characters. I had reasoned that ten files is a reasonable amount of a repository to show. Which is true, and also not a size limit at all. Ten files is anywhere from a few hundred characters to 320,000 depending on whose repository you point it at.

The obvious fix, and why I did not just do it

The obvious fix is to add a total character cap. Pick a number, fill it, stop.

The problem is picking the number. I could have reasoned my way to one. 50,000 sounds fine. 32,000 sounds fine too. So does 64,000. Any of them sounds fine, which is a good sign that reasoning is not the tool for the job, and is exactly how TOP_N_FILES = 10 got there in the first place.

So before changing anything I measured. Six repositories, spanning the range I would realistically encounter: two small ones of my own, a mid-sized MCP server, a docs-heavy list repo, and two large real codebases.

A fifty-fold range between the smallest and largest output, from the same tool with the same settings.

Three things fell out of that table that I would not have guessed.

The tree alone can eat everything

Look at the tree column for uv and the python SDK. 65,530 and 57,306 characters, just to list filenames.

I had been thinking of the budget as a cap on file contents. But on a repository with 1,600 files, the directory listing on its own is larger than any sensible total. If I had capped only the file content, uv would have blown past any budget I set before fetching a single file.

The tree needed its own cap, at a fraction of the total, and a note saying how many entries it left out.

The ranking was picking the wrong files

This is the one that actually mattered, and I only saw it because I printed what was being selected rather than just how big it was.

Here is what my tool chose to show about uv, a Rust build tool:

27,194  test/ecosystem/airflow/pyproject.toml
32,048  test/ecosystem/home-assistant-core/pyproject.toml
27,517  test/ecosystem/pandas/pyproject.toml
 9,156  test/ecosystem/jupyterlab/pyproject.toml
 8,757  test/ecosystem/black/pyproject.toml

Eighty-nine thousand characters of other projects' dependency lists, scraped out of uv's test fixtures. The only uv code that made it in was three main.rs files, all of them thin entry points, and Cargo.toml ranked below every one of those fixtures.

And here is awesome-mcp-servers:

32,048  README-fa-ir.md
32,048  README-ja.md
32,048  README-ko.md
32,048  README-pt_BR.md
32,048  README-th.md
32,048  README-zh.md
32,048  README-zh_TW.md

The same README in seven languages, 224,000 characters of it.

The cause was one line:

if name in PRIORITY_NAMES:
    score += 1000

Any file called pyproject.toml got a thousand points, wherever it sat. A vendored copy six directories deep in a test fixture scored exactly the same as the one at the root describing the actual project.

This is why the measurement mattered. If I had shipped the budget on its own, the digest would have gone from 207,867 characters of the wrong files to 40,000 characters of the wrong files. Smaller, still useless, and now looking deliberate.

The file it wanted most came back empty

One more, which I would never have found by reading code.

In the awesome-mcp-servers run, the top-ranked file scored 1200 and returned zero characters:

     0  score=1200   README.md
32,048  score=200    README-fa-ir.md

GitHub's contents API will not serve a file over 1MB. It does not return an error. It returns 200 OK with an empty content field and encoding: "none". So the base64 decode succeeds, produces an empty string, and the file silently disappears.

That README is 1,616,144 bytes. The single most important file in the repository was being dropped without a word, and seven translations of it were filling the space instead. The fix is to notice encoding: "none" and refetch through the blobs endpoint, which serves up to 100MB.

What I changed

A total budget, not a file count. 40,000 characters, configurable. Spent on the tree first, then files in priority order, until it runs out. The number of files now falls out of what fits instead of being pinned in advance.

A cap on the tree, a quarter of the total, with a line saying how many entries it omitted.

A per-file ceiling of 40% of what remains, so one large file cannot crowd out everything else.

Ranking by position, not just name. The priority bonus now decays with directory depth, so the root manifest beats a vendored one. Test and fixture directories lose points, vendored ones lose more, examples lose only a little because sometimes an examples directory is the best documentation a project has. Files in the repository's primary language, which the API already tells you, gain some.

Here is the same table after:

uv's digest now leads with Cargo.toml and README.md. awesome-mcp-servers leads with its actual README. My own small repo got bigger, because it was only ever showing 10 of its 22 files and now shows all of them.

The thing I would tell past me

I nearly shipped a one-line change. Add a constant, clamp the output, close the issue. It would have passed review, passed tests, and produced a tool that was confidently wrong in a smaller font.

What stopped it was spending an hour printing what the tool actually produced against real inputs. Not unit tests, which only check what I already thought to assert. Not reading the code, which is where the bug had been sitting unnoticed for months. Just running it against six real repositories and looking at the output.

If you are building anything that assembles context for a model, you are making size and selection decisions whether you notice them or not. Print what your tool actually sends, against real inputs, at real sizes. The numbers are frequently not what you expect, and the interesting failure is rarely the one you set out to fix.

The tool is pyarchana/gopher, an MCP server that fetches, caches and digests context for Claude. The measurements above live in #5 and #14 if you want the full before and after.