Know thine enemy: A critical engagement with AI-assisted software development

I have long been a skeptic of large language models, generative AI, and other forms of very large statistical models of surveilled human digital activity. It’s not that I don’t believe they work, or that they won’t change people or society, or even that they aren’t impressive feats of engineering — of course they work, and of course they are changing the world, as technologies often do. It’s that they promote learning avoidance, they’ve invited mass surveillance in trade for productivity and profit, they make the hard parts of implementing software (comprehension, coherence) even harder, and they make the important parts of designing software (requirements), too easy to do poorly. Worst of all, the very notion of modeling human behavior and history with data denies our non-digital, emotional reality, and all of the unencodeable possibilities of our future selves.

And yet, the recent impacts of LLMs in many domains are not ignorable. Whether I think it is moral or not, people are delegating email writing to language models, searching and summarizing document collections, generating art, avoiding learning, getting addicted, and sometimes dying. As much as I don’t like this turn of events, and will continue to speak against it, I can’t ignore this reality.

I also can’t convincingly speak against generative AI without understanding what people see in it, or how capitalists are positioning it. If we are in a war for humanity’s soul, then it serves no one for people on the front lines of its future to be ignorant of the weapons of the other side. And so I set out three months ago to dig into one of the domains I know best — software engineering — to develop my own intuitions about where programming is headed, how AI-assisted software development is different, and how it is changing practice.

I had authentic reasons to dive in. As a hobby, I maintain several open source projects (Wordplay, Adminima, Bookish.press, and Reciprocal.Reviews), and so I write a lot of code, review a lot of pull requests, fix a lot of bugs. And most importantly, I do that in my spare time, which is scarce. The shiny promise of AI-assisted software development marketing is that I could either 1) get more done in the same amount of time, or 2) get the same amount done, but get some of my weekend back. The question I wanted to answer was whether either was true, how, and what negative externalities it would have on the people and communities around me. To find out, I subscribed to a Claude Max subscription, stocked up on some Claude API credits, installed the Claude CLI and VS Code extension, and shifted my software development workflow from wetware focus set against EDM, Jazz, and Blues to distracted prompts and code review. For all of this, I used Opus 4.8 and more recently, Fable.

Three months in, I have a few observations, specially about software development, but also about generative AI more broadly as a technology.

I built more in less time, except when I didn’t

I’ve been writing computer programs since I was 12, which means I have 24 years of practice shaping requirements, planning architectures, and realizing them. That meant that my approach to using Claude Code was primarily one of heavy front-loading of defensive written design specification. I’ve been burned enough times to know that anticipating as much as possible prior to implementation is paramount to preventing churn and defects. And I’ve seen enough architectures to be able to imagine in my mind what I intended to build, what design patterns, algorithms, control flows, and data flows are appropriate to meet requirements, and what the hard parts about that work are likely to be, all before writing a line of code. That expertise meant that I’m a best case developer for using these tools. This was not an evaluation about how the tool might be used by novices or learner, it was an expert analysis.

Because my plans were meticulous, and my environment set up with rich context, and my architectures clean, for most tasks, Claude Code had a reasonable implementation about 90% of the time. In each of those cases, it was often sloppy about following the conventions in my repository, despite them being documented and ubiquitous. I often had to tell it to avoid creating code clones, as it would overlook existing functionality, especially in the 300K LOC Wordplay. But most of the issues I had to repair in these cases were minor comprehensibility and maintainability problems.

The 10% of cases, however, were often abysmal. Because Claude could not feasibly store the entire context of my repositories, and detailed architectural descriptions were rarely sufficient for their nuances, defects were usually deeply embedded assumptions. It generated algorithms with performance issues that would only be obvious with visibility into how the application was used. It embedded particular software quality assumptions about usability, accessibility, learnability, and other difficult-to-encode properties, presumably carried over from the broader patterns of implementation on the internet. It created test cases that encoded faulty requirements, resulting in over-stated correctness and completeness. A few times, it even clobbered data and deleted uncommitted in-flight edits in unrecoverable ways.

The only reason I found any of these problems was because I would deeply code review every line of code it generated, and provide multiple, sometimes dozens of rounds of critical feedback, hammering the implementation’s shape into something shippable. This work made programming feel less like building something and more like a telephone game, where I describe something, get some kaleidescopic version of it back, and then iterate until it created something sufficient. It was not a path to creating something beautiful, or intellectually coherent, but rather, something that was functional but unrefined. Much like the way that any other form of capitalist political economy settles for commodity, because it is cheaper and faster.

It freed my time, but at the expense of my attention

The programming I’ve done for my whole life has always demanded immense, uninterrupted periods of focus. Writing software is cognitively demanding work. When we do it, we fill up our working memory with an incredibly nuanced context of a problem, engage in reason, logic, judgement, and creativity. This process is one of ever-elusive flow, and can be quite joyful, much like an artist entering her studio for an immersive session of creation, and coming out having created something beautiful that did not exist before. These experiences are only possible with focus.

Using Claude Code to create software was a decidedly less joyful and less focused experience. The user experience is generally one of writing requirements in natural language, waiting for Claude to ingest sufficient context to make a plan, reviewing and refining the plan, waiting for it to execute the plan, and granting permissions for riskier operations with side effects. Once it is done, then begins a process of code review, feedback, and refinement, until it meets the originally stated requirements (or the process surfaces new requirements, as in normal programming).

The problem with this process is that it ruthlessly divided my attention. I needed to monitor its planning so it wouldn’t skip key context and generate a flawed plan. I needed to read its plans carefully, and often do my own read of the source it processed, to ensure the plan was sensible. During implementation, it would interrupt every few seconds to minutes, asking for permissions. And during code review, rather than analyzing something I had written myself, and already mostly understood, I had to code review unfamiliar code. Net, this was usually faster than creating it myself, but it meant that rather than a couple of hours of learning and focus, it was thirty minutes of distraction, while I tried to do other things like chores, or email. This was even worse if I was working on two, three, or more sessions in parallel, with interruptions every minute.

Because Claude was building and I was not, I did get some time back. My all day Saturday sessions did shrink to the mornings, giving me back the rest of the afternoon to do other things. But the mornings were now boring and fragmented instead of creative and empowering. And worse yet, I left each session with a compulsion to keep going — there was a power in being able to build so much, so quickly, and if I was spending a full day before, why not build 2–3x as much as I was before? Anthropic releasing its “remote control” feature, allowing one to guide a session from my smartphone, only made it worse, because then I could be distracted anywhere, rather than just my laptop.

This of course has leaked into my relationships and other work. Rather than being present with my wife at a meal, my mind was on the permissions notification in the Claude app. Rather than have focused sessions for writing during the work week, I let Claude notifications leak into my reading, writing, and even advising. This meant creating a much more intentional practice of mindfulness, as there was always an algorithm demanding my expertise.

The only thing that curbed any of this, aside from my own mindfulness practices, was money. Claude limits subscriptions to fixed session periods of token consumption, and API limits were bound by my wallet. This arrangement meant that much of attention was structured as an extractive, almost abusive relationship between Anthropic and my mind: respect limits, and I could be well, but submit to any desire for more making, and it meant more money, less focus, and less joy.

It warped my design process

I have a Ph.D. in human-computer interaction. It is in my nature and my training when I’m designing software to pause, listen, learn, and question before making software. I ask whether we should build it, why we should build it, who we are building it for and with. Implementation, and even software architecture, necessarily comes after long periods of learning with people and communities, about their needs, wants, and desires. The making itself should have the voice of the stakeholders who will use it, and any design process that isolates them from this making is fraught.

Claude Code was designed with a different premise: that it is an individual who decides. There is nothing about the product that acknowledges or supports collective work, or even design work for that matter. It certainly interfaces fine with things like GitHub and version control, but these are far from collective projects; they are expert spaces that make little room for other voices. This means that the path of least resistance when using it is to isolate: talking to people only slows you down, and you have tokens to burn!

My own processes have been in conflict with that, and so bringing Claude into them has created friction. There has been a sense that shipping is just a prompt away, when it used to be that the cost of engineering was high enough that finding the right design was essential. With the cost of engineering being lower, the product creates the illusion that getting the right design can come later, because that right design will only be a prompt away as well. The reality, of course, is that once something has shipped, and people come to use it, software becomes a sticky new reality. What racing to release actually means is racing to assume.

I’ve had to create all kinds of barriers around my process to address this friction. They’ve included:

• Mandating elaborate articulation of the “why” in pull requests, to surface assumptions, not only from others, but from myself
• Explicitly and methodically interrogating the assumptions built into an implementation, to verify that they don’t violate my understanding of the requirements.
• Limiting Claude use to implementation far downstream from design. It comes only after a long process of requirements gathering, and I don’t use it for requirements gathering itself.

As with my attention, all of these require discipline, and everything about the technology incentivizes short circuiting these principles.

If I put on my most narrow of computer science hats, I am of course impressed. Automated program synthesis was a mere dream a decade ago; now it is real. The motivations behind the research that led to it, whether natural language processing dreams driven by Star Trek’s depictions of speech-based computer programming, or software engineering research dreams about ever faster development, have been realized. Computer science has long wanted to automate software development, and LLM-based development represents the largest leap towards that vision since the field started.

Yet, as with the entire history of computer science, this engineering feat ignores nearly every aspect of impact on people, communities, and the world. I see clearly how it robbed me of focus, of joy, of money, and of values over the past three months. Others like my colleague Emily Bender have well documented how it will rob the world of sustainability, land, learning, rights, and freedom. None of this is particularly surprising at one level — domination, extraction, and dehumanization are, after all, what the broader political economy of capitalism is designed to do—but now there is a tool to do it ever faster, shielded by the guise of progress. And many of us, directly or indirectly, will pay an immense price in order to race toward that collapse.

That conclusion leaves me with a moral question: should I still use it? I liken that question to other similar questions like “Should I keep driving, when cars are the leading cause of death in the U.S. ?”, “Should I keep flying, when they are our biggest source of carbon output??”, “Should I teach at universities, when they are structured as projects of industrialized, neoliberal white supremacy? It also reminds me of Audre Lorde’s oft-repeated quote,

“The master’s tools will never dismantle the master’s house. They may allow us temporarily to beat him at his own game, but they will never enable us to bring about genuine change.

One can drive, fly, and teach as part of liberatory work, and perhaps even use LLMs for the political skirmish here and there, but the tools themselves are not the project. The project is changing our social systems, and there is likely no room in those systems for bullshit machines built for exploitation and greed.

For me, I think that means a harm reduction approach. There are going to be times when me using Claude Code, and other LLMs, is a strategy for survival in a system that has no interest in my survival. But I will live under no illusion that it is a tool for liberation, just like building more roads and airports will not help us build stronger communities — tearing them down will.

All of this, of course, is my own moral calculus. When we consider students learning computer science, for example, I think the moral considerations change. Students, for example, have to decide whether they want a career in industries that seem to be increasingly souless and artless. And to pursue those careers, they will have to have immense self-regulation to resist avoiding learning, as my own reflection showed just how much software engineering expertise is required to use them productively, and at great personal cost.

The situation for non-programmers who might want to vibe code software is similarly fraught. One might be able to prompt their way into something functional enough to meet a personal need; I think that the harm in that might be less, even though it comes with substantial external risks (e.g., security threats, data loss, fragility), as it centers an individual’s requirements by definition. But it’s also hard to imagine a world full of fragile software being a good foundation for commerce, connection, and work, as vibe coding is being used to support.

At the end of this experiment, I leave both more knowledgeable of what LLMs are capable of, but also more knowledgeable if what concrete harms they can bring to an individual. If you are also a skeptic, I encourage you to also study what these tools are and what risks they pose. Perhaps you find that there are some narrow uses that help you adjust to this new LLM-based regime we live in; perhaps you’re better armed to dismantle it, because you know it’s weak spots. Tell me what you learn.