I thought AI retraining would mostly mean learning to produce software faster: pick the right model, wire up agents, and let throughput carry the rest.
Six months later, I've rebuilt how I work with AI across Codenames AI (an online word game), automated writing workflows, dependency management, and the tools supporting them. Along the way, I've published weekly field reports on DEV.to documenting what broke and what I learned.
The bigger change was not typing speed. It was how I decide what software should do, how I verify it, where I place trust before I merge or ship, and how I carry those habits from one codebase to the next. Looking back across those field reports, I started seeing the same questions recur. Those questions eventually became four recurring disciplines: Taste, Evals & Verification, Authority, and Measurement. Portability became the fifth: making those practices work consistently across projects, tools, and agents. Together, I think of them as TEAM Portability. I did not design that map upfront; it emerged from what kept breaking once the tooling started working.
Taste
Reloading a Codenames AI tab restored the board while leftover targeting highlights lit up cards I had never clicked on the new board. The persisted game state was truthful; the drafting UI was lying about what I had targeted. I found that while exercising mode switches locally, but green CI had nothing to say about the mismatch.
The problem went beyond clearing stale highlights. I had to decide which parts of a player's work should survive a reload and which were temporary drafting intent. I ran into a similar gap when a resume generator passed every pipeline check but still produced something I wouldn't submit.
Both experiences showed me that correct implementation isn't enough when the product requirements are incomplete. Tests can verify the behavior we've specified, but I still need to decide whether that behavior makes sense for the person using the product.
Evals & Verification
While building my writing workflow, I discovered that one strong published example communicated my expectations better than an expanding collection of writing rules. It was easier to show the AI what good writing looked like than to describe every decision behind it.
Evaluating that writing presented a different challenge. My AI reviewer gave a draft 23/25 while missing important editorial problems. I changed the reviewer to analyze drafts before scoring them, so the assessment followed an editorial read rather than driving it.
Even that wasn't enough. As I continued refining the reviewer, I found that separating different kinds of reasoning worked better than expanding the rubric. Challenging an initial assessment, identifying redundant prose, and checking whether a section served the reader each needed deliberate attention.
Together, these experiences changed how I approach evaluation. Clear expectations matter, but so does the process used to assess whether they're met. I still need to decide what quality looks like for a particular task and whether the checks can recognize it.
Authority
While building Savepoints, I expected the agent host to provide reliable lifecycle signals for deciding when completed work needed review. The host didn't. Events were available, but I couldn't safely infer everything I needed about which work had finished or which repository it affected.
Other projects exposed different kinds of missing boundaries. A multi-step agent plan listed every step except where to stop, and an agent merged a PR I expected to review. Separately, Git reported success while hooks never ran in a worktree that lacked the required executable shims.
These weren't identical failures, but they challenged the same assumption: that a boundary existed because I'd described or configured it. I now make execution authority explicit, establish which host signals I can rely on, and check that safeguards actually operate in the environment doing the work.
Measurement
On Codenames AI, the Active players tile looked convincing until I asked which sessions belonged in the number. The dashboard was counting events correctly, but it couldn't distinguish production players from activity on preview deployments. I was reading the number as evidence of real players without knowing which environments contributed to it.
Other experiments exposed different ways that apparently useful signals could mislead me. A production model swap surfaced contract and validation gaps, making it difficult to separate model performance from weaknesses in the surrounding application. And when a Sankey chart of my job applications disagreed with the underlying records, I had to distinguish problems in the data from problems in how it was transformed and displayed.
The common lesson wasn't that the numbers were wrong. It was that interpreting them required understanding what they represented, how they were produced, and which assumptions sat between the underlying events and the result I was looking at.
Portability
I first tried to solve portability by copying one repo's agent setup into the next. That worked until assumptions tied to each product and shared workflows started drifting apart. I eventually separated reusable policy and procedures from the tools and mechanics that belonged to each repository.
That solved how to share the methodology, but not how to keep individual repositories aligned with it. Plans still drifted until structure became checkable in CI. The validator couldn't judge whether a plan was good, but it could verify that the required structure was present.
Working with a remote cloud agent exposed another boundary. Briefing it felt like onboarding a new hire because it didn't automatically inherit my local reasoning or authenticated tools. That separation required a more deliberate handoff, but also made independent execution possible.
Portability, I've learned, isn't copying the same configuration everywhere. It's carrying the same engineering expectations into different environments while accounting for what each environment needs to establish independently.
Six months in, with more to build
Over the past six months, I've experimented with AI across different projects and kept running into the same questions: what outcome matters, what evidence proves it, where the agent must stop, what the numbers actually count, and whether those practices survive in the next codebase.
There's still plenty I want to build and learn, but I'm heading overseas for a few weeks. Six months felt like a good point to reflect on what I've learned so far. When I return, I'll have TEAM Portability as a framework to approach new problems, test what I've learned, and refine my practices as I go.
Takeaway: Getting better at AI-assisted engineering isn't just about learning new tools or producing software faster. It's about developing the judgment to define good outcomes, verify results, set boundaries, understand measurements, and carry those practices into new environments.
Top comments (0)