Some questions about a startup idea can be answered with a conversation or a cheap web page. Others need a working product. A lot of wasted effort comes from mixing the two up.
Joel Gascoigne started coding Buffer before he'd checked that anyone wanted it. He says so himself, in a February 2011 post about getting from idea to paying customers. Partway in he stopped and put up a two-page site instead. Then he wedged a pricing page between the two pages, so a visitor had to click a plan before leaving an email address for a launch notice. A small number of people clicked paid plans, and that was enough to send him back to the code. Seven weeks of evenings and weekends later he launched, on 30 November, and by his account the first paying customer turned up within four days.
Figma went the other way. Its launch post talks about three years of silence. Dylan Field has said that when Figma raised its first round, it had no user metrics and the product was still in stealth.
People cite both stories as proof of opposite rules: test first, or build first and let the market judge the finished thing. Both founders ran companies that survived, so neither story tells you much about the odds. Put side by side, though, they show two founders stuck on different questions. This article is about working out which question you're stuck on, and what the cheapest test of it would be.
Two risks under one label
Rob Fitzpatrick's The Mom Test splits risk in two, at least as summarised by study guides. Customer risk asks whether people want this and will pay. Product risk asks whether it can be built. When most of the risk sits in the product, those summaries say, no amount of customer conversation settles it. His examples include video games and a farm fertility monitor.
Buffer was mostly customer risk. Queuing tweets wasn't technically scary; the open question was whether anyone cared. Figma's early browser-based design tool is a simplified example of the opposite. People could picture wanting it, and nobody knew whether a browser could do the job. That contrast is mine, not something any of the sources says. Real ideas usually carry some of each, so the practical question is which one is the bigger unknown right now.
Not all evidence weighs the same
Here is what common signals can and can't tell you.
| What you got | What it tells you | What it doesn't |
|---|---|---|
| A compliment | Goodwill, maybe real interest | Behaviour, or intent to buy |
| An interview about what they do today | This person has the problem and how they cope | That the market is big, or that they'd switch to you |
| A signup or waitlist | Your message landed in that channel | That they'd use it or pay |
| A pricing-page click | Which plan caught their eye | That anyone would pay. Nobody did |
| A signed pilot or letter of intent | Commitment, as strong as its terms (scope, date, price). Some letters are non-binding | That it holds up beyond the pilot |
| A payment | Willingness to pay for this offer, under these conditions | Renewal, or whether a discount, a friendship or a one-off need did it |
Repeat usage belongs on that list too. It's strong evidence for a product people are meant to use often, and you need a product before you can see it.
Fitzpatrick's own test, again via summaries, is whether a conversation ends in commitment or advancement: the other person gives up time, reputation or money, or agrees to a next step. A compliment costs nothing to give. Treat it as a weak signal until behaviour backs it up.
Don't read the table as a ladder. A payment for a service you deliver by hand says a lot about willingness to pay and very little about whether people will put up with your software. The strongest evidence is whichever kind matches the claim you're testing.
What you can test without a product
Interviews about past behaviour tell you whether the problem is real and how people cope. They're poor at predicting what someone will do with something new, and they're easy to steer without meaning to. A landing or pricing page tests whether a message pulls attention in one channel, and it tests willingness to pay only if money changes hands. A manual or concierge service shows what the job really involves and gives you honest data on price, but it hides everything about scale. A demo video or clickable prototype helps when an idea is hard to explain and says nothing about retention. A scoped paid pilot, a pre-order (with explicit delivery and refund terms) or a letter of intent gets you commitment from a specific buyer, though early buyers often differ from later ones. A technical spike tells you whether the hard part is possible and nothing about demand.
Look at Buffer with that in mind. The pricing-page click wasn't a payment, and the product had little technical risk. Gascoigne is open that his early "validated" was a handful of emails and some feedback. The better evidence, a paying customer, arrived after he'd built and launched. So the sequence wasn't "validate, then build." He ran a cheap test for customer risk, then built the smallest thing that could produce stronger evidence.
Food on the Table shows the manual route. In accounts of the Lean Startup method, an early customer got weekly personal service in place of software. Manuel Rosso's talk on concierge MVPs frames that service as a way to learn rather than to scale, and advises writing code only when it makes learning cheaper. The takeaway is narrow. Manual delivery shows what the job involves and whether people accept a high-touch version. It can't tell you whether software will do the job.
One rule covers every test before a product exists: say what doesn't exist yet. A landing page should describe what customers will get and when, and if you take money, the refund terms should be plain. Buffer's pricing page collected emails for a launch notice, which is a long way from pretending the product was live.
When building is the experiment
Some questions don't yield to conversation. Two cases show how, and both come with the usual survivor caveat.
Dropbox is awkward to describe and obvious once you see it. Drew Houston's own slide deck lists a three-minute screencast on Hacker News in April 2007 that brought "lots of immediate, high-quality feedback", then a private-beta video in March 2008 after which the waitlist jumped from 5,000 to 75,000 in a day. In a 2011 interview he said they'd expected 15,000 at most, and that they used a closed beta because they didn't want to risk a buggy product on all 75,000. One secondary account says a working prototype sat behind the first video. I couldn't confirm that from Houston himself. What the evidence supports is narrower: a demo helped communicate a hard-to-explain product and showed interest among early adopters. It doesn't show willingness to pay, retention, or that the audience generalised. A waitlist is interest, not commitment.
Figma is the cleaner case. The open question was whether a professional design tool could run in a browser at acceptable speed. Field says people who had tried WebGL before warned them off. According to Sequoia's profile, the closed beta came in December 2015 and the first dollar was charged in summer 2017. Interviews couldn't settle feasibility. That took technical experiments and a working prototype, whatever customer learning went on alongside. It was also affordable only because of a founding pair with deep graphics skills and investors willing to fund years in stealth. Few solo founders have both.
What the two cases share, and this is my inference, is that building was justified by one specific unanswered question. If your riskiest assumption can't be tested credibly without a working prototype, build, and build only the part that tests it.
How much validation is enough?
There's no credible number. The nearest research is a line of randomised trials by Camuffo, Gambardella, Spina and colleagues. In the 2020 trial (Management Science, 116 Italian startups in the paper's panel sample), some founders were trained to treat their idea as a theory, form hypotheses and test them, while a control group followed its instincts. The authors report that the treated group pivoted more and earned more revenue.
The 2024 replication in the Strategic Management Journal pooled four randomised trials covering 759 firms. Its abstract reports a positive effect on idea termination and a nonlinear effect on radical pivots: treated firms ran a few pivots rather than none or many. The authors read this as more efficient search plus more "methodic doubt". So the 2020 headline, more pivoting, didn't hold up as stated. The clearer result is more decisive stopping and more deliberate changes.
Both are training programmes for early-stage founders, and neither compares interview-first with MVP-first. They support something narrower: tests built to be able to say no seem to help founders decide. That reading is mine, not a conclusion the authors draw about MVPs.
In practice four things move the answer: what a wrong build would cost, how easily you could undo it, how reachable your customers are, and how much is still unknown. A weekend project you can bin needs very little evidence. Six months of work for buyers you can't easily reach needs a lot more.
A way to decide (my synthesis, not an established method)
Five questions, in order. Skip any you can already answer with evidence.
- Which assumption is riskiest? That the problem exists, that a specific group cares, that your solution works, that it's feasible, or that they'll pay.
- What's the cheapest test that could come back negative? A conversation, a manual service, a prototype, a technical spike, a pilot, a payment.
- What result would change your mind? Write it down before you run the test. If no result would make you stop or change course, you aren't testing anything.
- What's still unknown afterwards? Three paid pilots say nothing about what the fourth customer costs to find.
- What's the next smallest step? Keep learning, build a limited version, run a paid pilot, change the customer, or stop.
A rough rule for interview versus experiment: when several people independently describe the same recent problem and workaround, consider moving from broad discovery to a more specific test. More interviews can still help if you're exploring a different segment or an unresolved objection. Otherwise the next useful question may be whether someone will commit to a concrete step.
An experiment for a solo founder: the scoped pilot ask
This suits a B2B or prosumer idea.
- Hypothesis: People in one narrow segment who already spend time or money working around this problem will agree to a specific, paid pilot.
- Test: Talk to people who've shown the problem in past behaviour. Describe exactly what they'd get, by when, and what isn't built. Ask for a concrete step: a start date, a price, or a deposit with clear refund terms.
- Before you start: Write down how many yeses would make you proceed and how many would make you rethink. Choose a threshold that fits your market, your access to buyers and the cost of being wrong. Don't borrow a universal number.
- Record: For each person, their current workaround, what it costs them, what they said (paid, agreed in writing, asked to wait, declined) and why.
- Limits: You'll mostly reach people like you, a pilot price isn't a product price, and delivering by hand hides software costs.
- Next decision: A yes counts when it's a defined step, such as an accepted scope, timeline and price, not verbal encouragement. Enough of those means deliver by hand and note which steps repeat, since those are what to automate. Repeated "interesting, not now" means urgency or the offer hasn't been shown. Ask why: wrong buyer, unclear offer, budget timing, missing trust, or a problem that isn't pressing. Then narrow the segment, change the offer, or stop.
Back to Buffer and Figma
The two stop looking like opposites once you ask what each founder didn't know. Gascoigne didn't know whether people cared, and a small page could probe that. Field didn't know whether it was possible, and that took building. Both are survivors telling their own stories, so hold the conclusions loosely.
Before your next coffee chat or your next hour in the editor, write down the assumption most likely to sink you, and pick the cheapest test that could prove it wrong.
What's your counterexample? Have you built first and been right to, or run a validation test that said yes while the product still failed? What did you measure, and what would you do differently?
Top comments (16)
cheapest test that could come back negative. most validation plans skip the part where no is an allowed answer.
Yeah, and I think that's the part people struggle with most. It's easy to keep tweaking an idea until the feedback looks positive. Harder to accept that the idea itself might need to change.
tweaking until feedback looks positive is the trap. changing the idea itself is the harder call.
True. Sometimes we're so focused on making the idea work that we forget to question whether it's worth pursuing in the first place.
yeah, the worth-pursuing question comes before the tuning. most threads skip straight to tuning.
Yeah, and by the time you realize the idea itself might be the problem, you've already spent weeks trying to make it work.
weeks spent before questioning the idea itself. that is the sting.
And sometimes the sooner you ask that question, the cheaper the answer is.
sooner asked, cheaper answered.
The split between customer risk and product risk is the useful frame here. Most founders who come to us with a full feature list have product risk in mind, while the thing that usually fails first is whether anyone will change their current workaround.
Odinaka's line about the cheapest test that could come back negative is worth keeping. A landing page with a real call to action, or a manual pilot run by the founder for three customers, can answer that before any code exists. Revenue Operator's distinction between market proof and language proof also matters: if customers describe the problem in different words than your page uses, the page test will undersell a real need.
Where a build does make sense first is when the product itself is the test, for example a workflow nobody can judge from a mockup. Even then, one narrow workflow is enough to learn from.
The Buffer example, where a pricing page sat between the two pages so a visitor had to click a plan before leaving an email, is a good model for the cheap test. Before building, I'd check three things: market proof (are similar products selling, with review counts as a rough proxy), language proof (collect 10 posts where people describe the problem in their own words and reuse those words on your page), and one behavior test such as a fake-door pricing page or an honest pre-sale. If two of the three are weak, change the buyer or the problem before you build. If the risk is mostly in whether it can be built, the Figma route makes more sense, but for most solo makers it's customer risk.
I like the distinction between market proof and actual behavior. Seeing similar products sell is a useful starting point, but it doesn't tell you whether your specific buyer has the problem badly enough to pay for your solution. That's where the behavior test becomes important.
Exactly. Similar products selling tells you the category is real, not that your version is the one people pick. The cheapest behavior test I know is a page with the price on it and a real checkout or a waitlist button, then counting clicks from people who don't know you. Have you run one of those on anything you've built?
Haven't tried that exact test yet. The part about using people who don't know you makes sense, though. It's easy to get positive feedback from people who already know you, but that doesn't tell you much about whether a stranger would actually pay for it.
thinking a product is an MVP from pen and paper, in my humble opinion, is a road heading for demise. there is no place for arrogance in the world of product, at least not at a small scale. at the same time that you build it, there can be someone in the world other then you preparing to already publish it. an mvp MAY become an mvp after several people have tested a pilot functional enough to provide feedback, and when you receive their feedback it should be more than decent to merely start to believe you have an mvp. an mvp becomes an mvp after it had provided best usage in it's area. Nadia Comăneci was the first gymnast to earn a perfect 10.0 in Olympic history and became at her time the only MVP in her department, but she still had to firstly run her routine to prove she actually is an MVP. Same with product. Be humble in your journey.
I agree that real users need to test it before you can know if it actually works for them. I see an MVP a little differently, though. I don't think it has to prove it's the best in its category. It needs to solve one real problem well enough for people to use it, so you can learn what to improve next. That's where their feedback matters.