Ship the Seam, Not the Feature

Picture the review call. The candidate has shipped a tidy feature; the tests pass; the README has screenshots. Then you ask one question: "Add a third provider. Fifteen minutes. I'll watch."

Silence. They open the adapter, find the HTTP call copy-pasted into two route handlers, and the fifteen minutes become a tour of nine files. That silence is the signal you actually wanted from the take-home, and no amount of feature polish produces it on its own.

Below is a task you can hand out tomorrow: the prompt, the rubric, a reference solution, and the failure modes that show up in almost every submission.

The prompt, handed over verbatim

Build a small service around one interface, Model, with two implementations: one that talks to a hosted model and one that is fully offline. Ship make conformance, which must pass with no network access and no credentials. The interview continues as a live extension: in fifteen minutes you will add a third implementation while we watch, without editing existing tests.

That last clause is the whole task. Everything else is scaffolding to make the clause fair.

Why the drill beats a feature review

Features are graded by output. Seams are graded by the cost of the next change, and cost is invisible until someone forces a change. A candidate can copy a working chat loop from a tutorial and still not know where a timeout belongs. Watching them add a third adapter tells you in ten minutes what a two-hour architecture conversation only lets them describe.

The drill is also kind. You are not asking them to be clever under pressure; you are asking them to have made one boring decision early. Anyone who wrote a real port will look calm. Anyone who didn't will look busy, and both are useful information.

The reference solution

The port is two methods, and the docstring is part of the contract, not decoration.

# ports.py
from typing import Iterator, Protocol

class Model(Protocol):
    name: str

    def stream(self, prompt: str, *, timeout_s: float = 20.0) -> Iterator[str]:
        """Yield text chunks until the stream ends.

        The adapter owns the deadline: if timeout_s elapses before the
        first chunk, raise TimeoutError before touching the network.
        """
        ...

    def health(self) -> tuple[bool, str]:
        """Return (ok, reason). Never raise; never retry."""
        ...

Write the sentence about who owns the deadline, or the drill becomes an argument. Half the candidates who fail the extension fail because they assumed the caller enforces timeouts and the reviewer's new adapter assumes the opposite.

The conformance suite is deliberately small. It tests the contract, never the model.

# tests/test_conformance.py
import pytest

CASES = ["say hi", "return the word: seam"]

def check_contract(make_model):
    model = make_model()
    ok, reason = model.health()
    assert isinstance(ok, bool) and isinstance(reason, str)

    for prompt in CASES:
        chunks = list(model.stream(prompt, timeout_s=5.0))
        assert chunks, "a stream with no chunks is a bug, not an empty answer"
        assert all(isinstance(c, str) for c in chunks)
        assert "".join(chunks).strip()

    with pytest.raises(TimeoutError):
        list(model.stream("hi", timeout_s=0.001))

That last assertion is the one people argue with, which is exactly why it belongs in the contract and not in a reviewer's head.

The hosted adapter stays dumb on purpose. Only the payload builder knows the wire format, and it sits in one function so the extension drill never touches it.

# adapters/hosted.py
import os
from typing import Iterator

class HostedModel:
    name = os.environ.get("MODEL_NAME", "unset")

    def __init__(self) -> None:
        self._url = os.environ["MODEL_BASE_URL"]
        self._key = os.environ["MODEL_API_KEY"]

    def stream(self, prompt: str, *, timeout_s: float = 20.0) -> Iterator[str]:
        if timeout_s <= 0.5:
            raise TimeoutError("deadline expired before dispatch")
        # Wire format elided: copy it from your provider's current docs,
        # not from a blog post. Keep it in this method and nowhere else.
        yield from self._transport(prompt, timeout_s)

    def _transport(self, prompt: str, timeout_s: float) -> Iterator[str]:
        raise NotImplementedError

Note what the environment variables do not contain: any invented endpoint or model name. Those come from whoever runs it.

Then the loop you actually run:

python -m venv .venv && . .venv/bin/activate
pip install pytest
make conformance        # offline adapter only, no network
MODEL_BASE_URL=... MODEL_API_KEY=... MODEL_NAME=... make conformance

The rubric

Five rows, weighted so the drill dominates. Anything below 25 on the seam row is a no, regardless of how good the demo looks.

"Zero edits to tests/" is doing a lot of work in that table. It is the cheapest proxy you will find for a boundary that actually holds.

Where free access changes the arithmetic

Take-homes that require a paid API key quietly filter your candidate pool by credit card. The candidate who would have done the best work is often the one who did not want to spend twenty dollars to be evaluated, and you never see them.

That is the gap free model access closes: the hosted adapter can be exercised without a purchase order, and a free server option lets you deploy the reference implementation once so the reviewer opens a URL instead of provisioning a laptop. Both sides can run the same drill on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The limits matter more than the access. Free tiers are shared and busy; they are the wrong place to grade latency, throughput, or answer quality. Treat the hosted run as a smoke test, keep the offline adapter as the source of truth for pass/fail, and read the current quota page yourself before you promise a candidate anything. Quotas move, and a blog post that quotes one is wrong by the time you read it.

Who should not use this

Skip it if the role has no integration surface to speak of, and skip it if nobody can spare forty-five minutes of live review, since the drill is the only part that pays for itself. Also skip it if your team's own codebase has no seam to point at. Candidates notice hypocrisy faster than they notice rubrics.

Four ways it breaks

The adapter that leaks. httpx calls live inside route handlers, retries live in the view, and the port is a formality. Adding a third provider means editing request-handling code, and the fifteen minutes evaporate. You will spot this in ninety seconds by grepping for the HTTP client outside adapters/.

The fake that lies. The offline adapter returns a hardcoded string and never exercises the timeout branch, so the suite is green and the contract is untested. Ask during the drill which assertion would fail if the deadline check were deleted. A candidate who knows will point at it immediately.

The timeout nobody owns. The port is silent about deadlines, so the reviewer's third adapter raises asyncio.TimeoutError and the shared test expects TimeoutError. This one is partly your fault for shipping a vague prompt; fix the docstring, not the candidate.

Config sprawl. One env var read in five places. The third adapter needs a new one, and now every file knows about configuration. The fix is boring, which is the point: one settings object, injected at construction.

Run the drill on your own reference repo first. If adding a fourth adapter takes you twenty minutes, the prompt is not ready to grade anyone.

If you want the low-friction version of the hosted adapter, the free model access and free server option are enough to run this end to end with a candidate this week.