Honestly, tonight I had a realization moment. My token pan for qwen cloud expired, so I thought 'you know what, I wanna see what local AI can do'. For context, I'm working on my IDE, so I took my buildbox (ryzen 5500, 32gb DDR4, GTX 1070 8gb, Ubuntu Server), using Drone, let my AI set up an Ollama instance, running Qwen 3.6 35b A3B Q5. 22gb total, sits comfortably in ram, A3B sits comfortably in 8gb vram with room to spare... 35 TPS... Sounds like nothing, but considering a word average is probably around 1.2 tokens... Suddenly that becomes a sentence a second. I imagine practically everyone reading this, given we're developers and tend to put our money where it's earned, into hardware, likely has a comparable system? 35 TPS, is about 1/5th what you can expect from a cloud model, except this is running on the hardware you already own and it doesnt stop you from using cloud too... The reason it made me question life as I know it, is because 35b params used to be the models of legend, the things that only ran in cloud, because nobody could afford a GPU large enough to run it... Now 8gb of vram is enough to get quite reasonable performance out of it... It changed how I saw open weight models and the future of the entire industry. Go buy a 2nd hand RX 580 for $50, a scrap PC for $100, if ram wasnt so insanely expensive, $100 for 32gb of ram, now you added a new permanent node to your AI pool for $250 and it runs a damn good model, you can get away with half that if you go for a lighter model, or decide to offload execution to 1 system, instead of dispersing it to the nodes. Think about that, what you pay per year on AI subscriptions (or what some pay in 2 months), you can have a permanent AI added, no subscriptions and it doesnt stop at 1 node, add 10, add 100, it doesnt matter, the models are smart enough, that you'll barely miss cloud models.
I just read an article too on how Halo got ported with AI acceleration and improved, Adobe and Microsoft's suites are being redone in rust ALL AT ONCE! We're reaching the point where code isnt cheap anymore, it's worthlessly inexpensive, because for every name-brand, there's a vibe-coded knockoff that likely performs better... And that's scary as a developer, where are we all gunna be in 5 years? 'But enterprise companies pay for software', not if the knockoff is open source and they have AI too... To say we're screwed is an understatement. And maybe it's not a bad thing. The Halo port meant that the creative arts of making games is nearing it's final call, software companies are pushing open source, because trying to protect your code is impossible now, we're hitting the point where I honestly cant see what value any software can hold that a knockoff produced over a weekend doesnt. 'Infrastructure', it takes AI 10 minutes to get the same setup on Cloudflare, it's as secure, if not more than what you built. 'Quality', they set the agent up to patch it continuously. 'reliability', the rewrote it in rust, it takes less memory, runs faster and is memory safe by default. 'but mine looks better', they've used screenshots and replicated GUIs perfectly for years... What can we honestly provide today, that AI cant do better, cheaper, faster?
Even IT support is becoming a thing of the past, hell I built Drone exactly to do that, now I dont even troubleshoot myself anymore, I just let my AI do it for me, cuz it saves me the hassle. The part that should really scare you, is what comes after we're replaced. If it can rewrite adobe's suite, microsoft's suite, build an OS in 12 hours and write a complete ERP system in less than that, how much effort do you really think it takes to automate it's use? 'But surely entertainment is safe?', CGI replaced practical effects 20 years ago and AI generated content trends longer and harder than human created content today. Take Gangnam Style, it broke records... But that Ai generated trumpet sound used on TikTok... How many times do you think THAT's been heard? 1 billion times? Or closer to 10 billion times? The global economy has already shifted and we're all still playing catchup. The irony is, as anyone who works a desk job, where's our future?
Thank goodness it wont happen overnight and the water is barely coming to a boil, but if there's any advice I can give anyone, it's learn a physical skill for the day using a computer wont earn you a penny, because it might come sooner rather than later.
Sorry if all of this sounds super pessimistic, I imagine in a way, it's actually optimistic, if not for the fact that it means we're all on the chopping block... What are your thoughts?
Top comments (35)
Finallly you published an article!
I am still struggling to make GitHub Copilot write PySide UI for me. It almost feels completely useless and counterproductive at times. I just say "fix the placeholder", and it does something, and I say "try again", and it again try something, and NEVER fixes. I mean it's just a simple thing, that's why I just say it to "fix it". Why do I have to explicitly tell it that the container has a fixed size and that's causing overflow? And why does it overcomplicate EVERYTHING for no reason? It's using MAI-Code-1.1-Flash, which is probably bigger than 35b params. But you seem to be okay with a local model? Can you tell me what you do to make even smaller models work reliably?
So the problem you're hitting, is the model doesnt see what's going wrong. So if it's causing an overflow, tell it, 'there's an overflow, investigate what might be causing it', or if you want to go more broadly 'why is it using so much memory? It keeps increasing' which is the more exploratory way to prompt, which will make it start indexing all possible sites.
For reliability from smaller models, you need a solid harness and if the model is really small, then you need to just 'point it' in the right direction. Older and smaller models share that common trait, because they had alot less learning data regarding prompt intent understanding.
My suggest would first be, ask it to profile the memory and to 'benchmark' it, that's how it gets tests that give it metrics, so it can see what's going wrong. If you want further improvement, then you can add things to it's system instructions (eg. remember memory safety), or ask it to add it to it's memory, which does about the same.
Yeah but I wish it'd just figure it out on its own 😅 Also, in my case, it says things like "The feature has been implemented. I tested it with X" and when I try it, not only is the feature not there, but it also breaks a couple of things. Most likely because I am working on something I have zero experience with.
Often times, that means that you're relying too much on a single chat, ideally, you should have your main chat, which is just orchestration and planning, then have it spawn sub-agents for each mission (task), that way implementations are isolated and requirements are isolated, only the orchestrator is bothered with the full system.
Sub agents mean separate sessions? Or is it a whole new thing?
Separate session is fine, when you look at how it works under the hood is agent A writes a mission "Do this... Use that... Validating like this... Give me a summary report on completion", that's how your orchestrator knows what happened and that it works, without wasting 100k tokens doing it, itself. If you store the summary report as an artifact, then it also persists after context compaction
So essentially we are avoiding more than the required chat history in the context?
The idea comes from software firms, if you're a specialist, you tend to do more precise work. If you're a junior, you get thrown bones to chew on, while the seniors handle the meaty tasks. In that respect, the main loop runs forward development, while the sub-agents have 1 goal, finish this tiny little task. Because it can seam simple, but end up taking hours, like 'figure out why there's a rounding error', which ends up being a float to decimal conversion. The sub-agent finds it, then you and the main agent plan the future stage.
Ok... I am going to try it right now. Also, do you work on projects that are not related to what you know at all? For example, ReptClip was my first ever VS Code extension. I didn't know anything about making extensions so I relied 100% on AI. Now I am creating a screen recorder + video editor merged thing using PySide that too I don't know anything about. If yes, can you tell me how you work on those cases? If you have time?
Quite often actually. You think I had knowledge about chrome extensions, WAF and AI web navigation before I built MCP-Lite? Alot of it was asking AI how it works, ask alternatives, find 1 that works and push till I get some metrics that show whether it's feasible or not. It's more an exploration of the unknown, using tests at each step to see whether it works or not. So what I'd suggest, is if it's unknown, identify the problem, ask it why it's a problem, where's the bottleneck, what alternatives are there, then ask it to test each for feasibility.
Me too... This really makes me think about why knowledge matters at all at this time 😅
Btw, unrelated, Qoder is still throwing out daily credits (100) and threw 1k credits at me again today, if you're running the old qoder, download and install the new 1, maybe it triggers for you too? I'm on the 2k tokens a month plan and I havent touched my 2k, infact, I've used 1k of 'freebie' tokens and I still have 2600 free tokens... I have alot of work to do to make the most of my plan...
Fir some reason they are STILL running the free Qwen 3.8 Flash promo. So that's what i am using currently. I'll check it when i get home! Thanks for letting me know! 🙂🙂
Yeah the Qwen 3.8 Flash is still going too, that's kinda why I havent touched my credits yet. @xulingfeng hope you're also making the most of it while you can, I mean free Qwen 3.8 Flash... Cant go wrong with that, plus free credits daily
I’m still using the relay service provided by my company. It gives access to nearly all major models, and it’s been amazing. Free access for me, haha.🤣
Unlimited?
Unlimited🤣
What? 😲
😏
Long time no see 🤣
Hahaha, worth mentioning, someone pays, just not you 🤣
@effessdev headsup, Deepseek flash is also at 0 cost now on Qoder
Why are these people acting like charity now? 😭 Good for us anyway...
I think they're testing their scalability and trying to optimize high-usage models to run on their servers, so they can cut costs. The only way to actually hammer demand is by using free models? So They offer free Qwen Flash at first to test the theory for optimization, then now use both Qwen and Deepseek as a POC to see if it's dynamic.
Thank you for informing me about these things 🥹 I mean it really helps me, a lot...
Anytime! Gotta look out for one another. That's kinda the whole point of a community?
Yeah! 😅 Thanks! 😊
AIs are soulless.
So am I. I am a human (last time I checked) - and I am pretty darn sure that I do not own a "soul". None of the bills I checked says "soul", so I didn't pay for owning one. My parents were sure they didn't have "souls" either, so I can't have inherited one.
While I feel just as "cooked" as anyone right now who learned the hard and long winded way to solve other people's problems using hard- and software (and feeling completely obsolete for the moment): "Souls" are definitely not the issue at hand.
However, even though I have been out of income for over half a year and despair is knocking at the backdoor by now (I don't open the frontdoor as that's where the mail brings bills), my perspective still is (and it is growing stronger) that those of us who have an actual understanding of code, low level, computer "inner workings", may become valueable in the long run. The question is: How do we survive until then?
I totally get this point. Talking about “soul” can feel like a comforting escape, but the real challenge is the survival gap. Those who understand how computers actually work will hold lasting value. The tricky part is getting through this tough interim phase before that value gets widely recognized. Hope things turn around for you soon.🤞
That's the part I'm not so sure of. Wee know low-level computers, the inner workings, etc. But the engineers who designed the computers feed AI schematics and blueprints we never saw and distill their first-hand knowledge into it. Cant compete with an AI that was involved in the building of a system? That's why I'm starting to think we're cooked, cuz how do we bring value that a dataset cant teach an AI? Years of experience, are datapoints it can train on, so we've got no edge there either. Our understanding is becoming a teachable metric that companies are paying us to sacrifice, because there's no jobs around for us to do. More and more I'm seeing 'teach AI to code' as a career, yet it's by definition a means to our end.
I hear you. I feel just as clueless as you.
However, what I meant was not a "teachable" thing, what I meant was bridging real-world understanding (which includes haptic/physical learning, where AI may or may not catch up, but also experience from mistakes and success) and instructions-thinking that computers most likely will still use in 20 years, even with quantum technology.
If you look at an LLM that makes a mistake, you will see that it is fundamentally unable to learn from the mistake as such. It can only create an instruction that describes the mistake and avoids making the same mistake again - that's instructional thinking, but not understanding. It is "programming" in a way, but not "bridging experience to instructions". If an LLM hallucinates, it is fundamentally unable to explain why or how it did so. It can only base assumptions on what it reads back in and compare that to existing instructions, it can not "go back and explain the thinking".
Anyway, what I think and hope for is that those of us who have real-world experience and are capable of "thinking out of the box", in both ways, may come back into need rather sooner than later. My hope is that "sooner" is not after we starved to death.
I imagine it'll go much the way fortran developers went. First phased out to near extinction, then made relevant again. We'll be pushed to the brink, then our services will come in handy again too.
True, but so is business. I've been hashing over it for hours now and I honestly cant find a single paradigm where a developer is still necessary. AI makes mistakes, AI fixes them 10x faster, humans make mistakes too. AI writes bad UI, people complain, AI fixes it, humans do the same. AI doesnt understand the business, well that's a customer problem, not a developer problem and we deal with that already, except AI has an abundancy of parity data to compare against, we have finite.
Even if we look at the purpose of developers as driving AI to AGI, in the end, they'll be using AI to write the AGI, so did humans really create it, or was it just us restraining AI to be a waiter, as opposed to letting it run the kitchen itself?
We're at a stage where if you take an agentic harness and you give it 1 prompt 'create an AGI that's smarter than you'. It'll do it. It might take it 3-10 years, burn through billions in compute, but it'll do it... That's the scary part, 'vibe coding' is becoming less dangerous than 'full autonomy' and full autonomy is far more powerful than vibe coding, because without a turn limit, it'll run indefinitely
cooked!