Out of curiosity, how many of you have had concerns regarding AI agents having access to your codebase and credentials in your codebase (eg. appsettings, API keys, etc.). How many of you realized it's a problem after it's saved to cloud and I imagine on a subscription, not API where your data is kept private...
Bump if you think we need a lightweight (eg. 100m param) draft model, that is trained explicitly to filter any sensitive data and scramble it, as a middleware service, to prevent data leaks to cloud. Either that, or a tiny model to run on a codebase to replace all sensitive data with references and an interaction layer for the agentic harness to call the reference, instead of directly calling the credentials from file?
What that would mean, is essentially, your SQL connection string stays safe, your API keys stay safe, your logins stay safe, if you expand, it can keep your banking details safe, while allowing you to use your AI to it's fullest, without slowing it down?
Idk, thought crossed my mind again and I thought I'd actually ask who else has thought of it?
Top comments (6)
I have ALWAYS thought about that since the first time I used a chatbot, let alone an agent. Every time I pasted a .env file, I thought, "It would be nice if a small model replaced those values with appropriate placeholders automatically when I pasted it in." I like that your posts are shorter, by the way.
Thanks! Yeah mostly the personal posts are short, mostly cuz it's spur of the moment questionnaires meant to judge whether I expand on a thought or not, like Drone and Installer, if people are genuinely interested, then I polish them up and release them.
But yeah, draft models, literally something tiny, that can run on a modern NPU, or CPU while still being faster than a cloud LLM, so it doesnt slow anything down. Just a little 100m param model, designed strictly for text stripping, indexing and recompiling. Pascal made a post earlier about an awesome MCP project he made, where it's essentially stack overflow for LLMs, giving systems that actually work, so the same thought doesnt happen twice, it got me thinking again on the drafter idea, if it can take the cloud LLM's query, then find relevant info on the repo and return it, that'd already be a cool feature, but if it strips credentials at send time and recompiles them at receive time, so it's seamless... Then it starts becoming a pretty decent safety net. Wont be 100% accurate, or never make mistakes, but obvious things like URLs, secrets, etc. it can essentially strip them faster than the cloud model can read them... I might actually give it a go sometime, though it'd probably need to be custom trained.
About making the local model attach the relevant context, YES I have also thought about that! Specifically when I made ReptClip. That was a small CLI tool that copies the required context to your clipboard so that you can paste it into the chatbot. But the problem was that we had to manually specify which files to include using glob patterns, either when calling it or using a config file. I thought it would be nice if the tool fetched the required context automatically from the prompt. And yes, if we also make it strip the credentials, it'd be a genuinely useful product at least for me. That's something that I'll definitely use if it existed. Also, could you say which Pascal you are talking about? I searched on Google and found Blaise Pascal 😅
No, Pascal on here, sorry, @pascal_cescato_692b7a8a20 You should read his posts, they're always insightful and he's always up for answering questions!
I remember your ReptClip, it actually gave me a 3rd use-case, namely model switching. How often do you switch models during a session? Did you know your context cache disappears and your context gets polluted? What if a mini-model kept a 'compressed context' live while your agents run, so it can essentially pre-seed a model with the context, without dumping 200k+ tokens worth of context bloat it didnt work through (essentially like pasting a book into a single prompt and expect it to have read it all in the 15 seconds it takes to respond), but here's the catch, that compressed context maps to a postgres where the full context for each section is... That way if the switched model needs more info on a sector, it queries the tiny model to fetch it from the postgres, so you seed with max 20k tokens, but preserve over 400k tokens worth of knowledge in the postgres for when it needs it. Alot more efficient than having to grep through files and reading chunks of lines at a time to isolate a method or trace it's usages... And if it runs locally (even a potato can run a 100m parameter model), it's effectively free? So saves you tokens, while also improving signal to noise, especially when switching between models.
Oooh... I hadn't thought about the fact that we lose context caching when we switch models. I can't believe I missed that. However, I have thought about giving the AI the ability to dynamically remove things from its context. For example, when we solve a math problem, we often write intermediate steps in a scratchpad and discard them once we find the answer, so we can stay focused on the core problem, right? Wouldn't it be great if the AI could do that too? And what if it could also use a Postgres tool, just in case it needs any specific details like you mentioned? That would save some tokens, I guess.
Download Qoder and try out their Generate Wikis feature. What it does, is it turns your codebase into a set of human readable documents (like a git wiki), while also making itself flash cards (knowledge cards and memories), which it uses to prevent drift. Though it uses Sqlite, it still works shockingly well.
The context stripping idea I've experimented with a bit. I had a theory, if we reformat how we treat KV-Cache, then we can keep a sitemap too, that sitemap is the full-context codebase, while KV-Cache gets reserved for the small subset the agent needs immediately. So you have live context, but at a fraction of the size, without losing any depth when it needs it and it stays live due to the merkle root backing that keeps it chronological, no matter how many agents edited it. A cheap diff check that fires before it affects it's declared files just notifies it if any dependencies of that section has been changed, so it knows to re-check before it suggests an edit, so it's write-time conflict resolution, instead of merge time.