How Good is Claude Code?

LLMs for Coding

Since 2022, when ChatGPT released, LLMs have become increasingly used for generating code. Pretty much every software engineer I know uses LLMs to help them code. Much of the code I've written in the last 3-or-so years has been assisted by LLMs.

Over the past few years, companies have been training models specifically to be good at coding. And the models have been getting better. Since code is a verifiable task (We can check if the code the LLM generated runs properly) it's easy to use Reinforcement Learning to improve coding performance.

I've read lots of accounts (especially from non-technical people) where someone tries a coding LLM, and they're shocked at how capable it is. People have been using these coding agents to build all sorts of stuff, and I've seen plenty of sensational headlines like this:

In spite of all this, my experience has been much more grounded and measured. In my experience, code purely generated by LLMs is finicky and doesn't cover lots of important edge cases. Often, LLM-generated code doesn't work properly or breaks when I try to use it. This is especially true if the LLM is creating anything with a UI (LLMs don't have bodies and eyes, and even ones with vision modalities struggle with many of these tasks). While I use LLMs to help code and debug my games, they often fail and need considerable additional work from me.

I expressed a little bit of this in my State of AI post from the beginning of this year. Whenever I'd express these feelings to my coworkers, they'd say something like, "You're not using the right model" or "You have to give it more permissions." I don't think it's fair for me to make sweeping statements without fully exploring what I'm talking about, so I decided to fully commit to trying Claude Code

Trying Claude Code

I paid Anthropic $20 to try a month of Claude Code. If it was as much of a productivity boost as it claims to be, this would pay for itself many times over. I installed a plugin in VSCode (the IDE I use for coding) and gave Claude permissions to run whatever it wanted without asking me. I was off to the races.

The goal here was to see how easy it would be for Claude to do something that I would need to learn a new skill if I wanted to do. Could Claude do something for me that I couldn't do for myself?

I decided to make a video game. Now, I've already made a few games myself (and have a few more in the pipeline), so for this game I decided to do something different. I wanted the game to be completely playable without touching the keyboard. Instead, the game uses hand tracking. Players have to have their webcam on, and they use their hands to navigate and play the game.

Getting Started

As the most basic proof-of-concept, I asked Claude to make a page which showed the webcam with hand tracking, where the hand used finger guns to aim and shoot. We got an open-source hand-tracking model from Google, and Claude made a prototype. Unfortunately, the page was finicky and the finger guns didn't work. I did a bunch of iterating and testing, trying to use hand pose like a joystick, and hand position to aim. Things ended up working out pretty well, but I had to have Claude rebuild things several times after I tested it. Claude (who doesn't have wrists or hands), had some really iffy ideas about what's comfortable for aiming.

It took me a couple hours with Claude to get hand tracking working. Don't get me wrong, this would have taken me much longer without Claude's help. However, this experience was super far from the, "Claude can make anything by itself" hype I've been seeing online.

Claude also didn't think through everything required for the game. For example, if we want people (some of whom are left handed) of different sizes with different webcams to all be able to play this game, we'd need to set up calibration. I had to think of that myself - Claude didn't. Claude also needed me to verify simple things like 'are one or both axes inverted?' while testing.

Making a Game

With controls working, I was ready to make a simple game. I decided to build a simple carnival game. I was looking through free CCO-licensed assets at Kenney, and I found a 'shooting gallery' pack I could use.

After downloading the assets, I gave Claude access to the pictures and told it to make the game. It didn't really work. The game was laid out poorly, and the ducks would just fade in and fade out when hit. It wasn't like a shooting gallery at all. I had to specify where everything should go, the makeup of each layer, which sprites to use (claude chose some wrong ones), and how the ducks should move. I also needed to explain the background motion in great detail, have the ducks drift past in a sine-type wave, and troubleshoot overlapping and many other bugs.

I also had to suggest features like a reload, round timer, game difficulties, and pausing. The fact that I needed to specify basic features and tweak the implementation several times makes me really doubt things like, "Claude can make Minecraft by itself." But, Claude did get it to work (with a lot of direction from me.)

Claude's really good at writing code for me, but I needed to specify basically every aspect. On top of that, I needed to tweak almost everything Claude did. I probably could've just told Claude how to change things, but I found it much easier to go in and tweak code myself to make it work right. I had Claude build out a tutorial, force calibration, and save calibration and high score info in a browser cookie. We created a splash screen so the page didn't just start a game by itself.

I also found a great font, Jua, on Google fonts. Although it was designed for Korean, it has a Latin alphabet as well. It's perfectly playful and is great for the feel I wanted for the game. As a side note, I feel like fonts are often neglected by devs but really make a game feel polished.

Claude also has session limits (an amount of work it'll do in a day), and I hit my session limit while working on this game and had to finish it the next day. Not that that's an issue, but it's far from the "I have Claude working for me 24/7" I hear from AI bros.

I did some extra testing and found flow problems and issues with buttons, menus, etc. In fairness to Claude, it did catch some edge case issues (what happens when you pause the game and try to reload at the same time?) before they cropped up, though not all of them. After about 24 hours, I had a working game.

Once the game sees your hand, you flick your fingers, opening your hand (kind of like you're throwing a baseball) to toss a ball at moving ducks. When you run out of balls, holding your hand open with the palm toward the camera will 'reload'. There's not much to it, but it's kind of addicting.

Of course, any game developer knows not to deploy a game without playtesting. I'd like to thank my wife Madalyn and my brother Kyle for their assistance in making sure the game worked okay before I launched it.

Thoughts on the Process

I was able to make this shooting gallery hand-tracking game in less than 24 hours. I'm certain that I wouldn't have been able to make a game this quickly by myself without Claude's help. However, I'm equally certain that the only reason the game works is because I have a decade of software engineering experience, and lots of web and game development experience.

Am I proud of this app? Yes. I acknowledge that Claude was a proficient engineer. I was able to specify things I wanted and not have to do everything myself. However, this was more of an AI-assisted project than a purely AI-created one. I put several hours of work into it, as did the talented artists at Kenney who created the assets, and the Woowahan Brothers, who created the font.

While I liked having Claude work for me, I don't like the idea of having it run ad nauseum. I'm pro-AI, but I also am pro-Earth, and running Claude has an environmental impact. I think it's good to use coding LLMs as a productivity boost for specific tasks, but I also don't see Claude as a panacea for all our software engineering ills.

Working with Claude felt like having a junior engineer working for me. This engineer is very capable, but sometimes makes mistakes or doesn't take initiative. Overall, Claude Code (and other coding LLMs I've used) are super useful tools that let engineers do a lot more work, but my experience continues to be a far cry from the magical experience people have been gushing over online.

Of course, as I explained the results of this experiment to a colleague, he told me the reason my experience wasn't magical was because I wasn't using Fable, Anthropic's newest coding model. The hype train rolls on, with its refrain of, "You just need to try X!"

Ramifications

What does this mean for coding? I subscribe to Andrew Ng's (Cofounder of Google Brain) AI newsletter, and last week he said something in his newsletter that really resonated with me:

I've found that over the last few years, I've spent less time coding and more time thinking about problems and how to solve them. Then, a coding assistant has been able to do some or most of the implementation for me. This gives me more time to think hard, make creative and architectural decisions, and really squeeze the most out of my brain.

The Game

Of course, Claude and I worked hard on this game. I think the quality is pretty good, and it's pretty fun once you get the hang of it. You should try it out! You'll need to be on a PC with webcam access.

While I haven't become an expert yet, my high score is 28. Can you top that?