Guilt-Driven Development

A few weeks ago, the day before Midsummer, I went on my summer holiday. I merged my last pull requests and shut down my computer, looking forward to a few weeks of relaxing.

Come Midsummer morning, I checked my work emails 1 and I noticed that a failure had been reported earlier that day for one of the services I maintain.

Damn it! The failure was regarding an HTTP endpoint, and among the last few changes I made before shutting down was to migrate some of our endpoints to newer technology.

I absolutely hate this feeling. Someone innocent, who was not fortunate enough to be free on Midsummer, could not do their job. Or at least that was what I thought. I made some coffee and booted my computer again - I needed to take a closer look at this error before I would be able to relax enough to start our celebrations 2.

When investigating the error, it turned out to be a false alarm. The failure was real, but it was bot traffic that caused it, not "real" traffic, and the error had nothing to do with my changes. Phew!

Still, the request should not have caused an error to be reported (bot or not). That needs to be adjusted, but it is not urgent. I added some comments to the error and once again shut down my computer to celebrate Midsummer.

This got me thinking: what is the essence of that bad feeling, and what do I do to avoid it?

Guilt

Whenever I introduce an error in production I feel guilty - I have let someone down. Someone was trying to accomplish something using the software I have built (or been part of building) and they could not do it. Something that worked yesterday is no longer working.

Maybe the task the user tried to accomplish was critical to them? Could this failure stop them from doing their job? What about their income, or their customers? What is the extent of my failure?

You see, failure cascades across domains and organisations. I like to think of it like ripples on the water. Throw in a stone (an error) and it will ripple across the surface. How many rings? It depends on the shape of the stone (error). And like ripples on the water, the effect eventually fades out, but there might be an awful lot of rings.

Failures also scale, in the bad way. How many of your users are suffering from these - one, a hundred, thousands? That might be a lot of rings.

Many years ago, I was involved in a project that made a big-bang release3. Everything was running smoothly until it didn't. The project manager called me and said that the point-of-sale systems of a thousand shops were down. I asked of how many. A thousand, he said. Fortunately this was detected in the evening and not during prime time. My colleague and I stayed up all night reverting the change everyone thought was causing the outage (we were not sure, we had basically no automated tests). Later, it turned out that there was an infrastructure change made as well, which was most likely causing the problem, but no one knew for sure.

That was the worst day in my professional life.

Neglect

The world is full of crap software - we have all experienced this. Poorly designed "solutions", or unstable software we have to use.

Poor software is such a waste of money and/or a missed opportunity. But what really annoys me is bugs, when things just don't work. Or when you are forced to work around a system to perform your task!

Suggesting a workaround as a fix has almost been normalised these days. This is ridiculous, a workaround is NOT a fix!

Time. Time itself is infinite, but for each person and organisation it is finite, and very precious. This is true for all persons. Then why are we (as software makers) so disrespectful of others' time?

Value. Software is supposed to add value; when software fails to deliver what it is supposed to, the value is not zero, it is negative! The user needs to compensate for the malfunctioning software with, you guessed it, their time.

Guilt-Driven Development?

I do not remember where I first heard this term (I guess a lot of people might have independently coined it), but I like it.

You should not (and should not be forced to) deliver something that gives you guilt, guilt of not being done, not being robust, not being maintainable, etc.

That feeling of guilt I felt at Midsummer is something I'd rather avoid, and I do so by following my four T's. Thinking, Types, Tests and Telemetry.

Thinking

This might come as a surprise to you. Everybody thinks, right? Right? Well, no. I have both fallen into and seen others fall into the trap of immediately bringing out the keyboard and starting to chisel away at the solution.

For me, the absolute best way to increase the quality of a solution is to present it to someone else and get their feedback. Depending on the size of the thing I am working on, I either:

• Have a peer-to-peer meeting with someone in my team, communicating what the task is about and how I intend to solve it. This covers both how it will work from a user perspective (black box) and technically how it is implemented (white box).
• Write a solution description document containing the same things, and then have multiple people give feedback on it.

I have been consciously vague on who and what to review. Best is, of course, when both a domain expert and a fellow technician give feedback. But sometimes the task at hand is reasonably clear enough that one of them will do.

What should they review? Well, that is up to you. I do not believe in formality, strict processes, or tollgates. I most often include:

• Why is the change needed?
• How will I implement it?
• How will compatibility be handled?
• How will the user interaction work?

Thinking before any code is being ~~written~~ prompted has some benefits:

• As the well-known (but hard to attribute) quote says, "Writing is nature's way of letting you know how sloppy your thinking is.".
• Getting feedback on whether you got the "why" right.
• Getting feedback on whether this is a reasonable technical solution.

Turns out that whenever I skip this step, the risk of me misunderstanding or missing a technical detail is pretty high. And changing software down the road is almost always more expensive than doing it right the first time.

Types

I remember the first time I heard of Ruby's method_missing. I thought it was proper crazy4!

Throughout the years I have programmed in a lot of different languages. There was a time when I favoured dynamically typed languages such as Python and Clojure. For the last ten years or so I have been pretty set that statically typed languages are better suited for most teams and most problems.

But types are not only about the presence of types, it is how you use these types.

The language Elm (and possibly others) has this saying:

Make Impossible States Impossible

When you design your program using types, you should try to eliminate problems at a design level. This is the opposite of the saying in dynamically typed languages:

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.5

Probably is not good enough, and it is being evaluated at runtime. Most certainly, not all functions or call sites will put in the extra effort to ensure that the type is correct. This is easily forgotten. With a statically typed language and a compiler, it is always there, you do not have to do it manually.

The help of a compiler is only a minor part of the benefits, though. The real benefit comes from the design. By using types to represent things in the domain using Aggregates, Value Objects, Algebraic Data Types, etc., you can rule out a lot of errors and mistakes already at compile time, when you have all the time in the world to make it correct before the first user even comes close.

The amount of bells and whistles your type system offers is quite different between each programming language. But there are a few things that are almost universal.

Value objects - small, immutable types that represent a thing in the domain rather than a data type.

Type invariants - types that guarantee a rule or condition for a type. E.g. that the title of a Blog cannot be empty.

Data Transfer Objects - use differently shaped types suitable for other systems, such as HTTP, queue messages, etc.

Mapping - use many types and map between them where needed. E.g. let the validation map a DTO representing an HTTP API payload to a type with stronger guarantees (such as input used at the service layer).

I feel that a lot of people wrongfully consider some of these concepts as "enterprise software" and too verbose and cumbersome to use in "modern" software. This is unfortunate, and while I think some of the criticism of "enterprise software" is correct, the blame is not on the concept of types, but on how they were designed and used (e.g. annotations, reflection, auto-mapping). It would be like blaming Perl for all the bad scripts you have run!

Tests

I have written about tests before, and consider the time in my career before I used TDD my "dark ages."

Well, you do not have to use Test-Driven Development if you do not want to, but you have to have tests! I think that developing software without tests is the peak level of arrogance toward your users.

I always have unit-tests for any code I write6 as a bare minimum sanity check. All my code has some sort of service-level tests (which, in my way of working, are close to integration tests) as well, though not for all variants. And I add integration tests where suitable.

Implementing tests for all types of situations and input I can think of as part of the development process is a true game changer. It forces you to think about input validation, error handling, etc.

Learning to write good tests is hard, but I would say it is one of the best-returning investments I made in my career.

Telemetry

Once I have covered what I can think of ahead of time, I add logging or similar to any corner case or failure that I want to be made aware of.

Sometimes a state might be a theoretical one, or at least very unlikely to happen. You want to capture any such situation, if it occurs. Maybe your "this will never happen"7 reasoning proved wrong after a seemingly unrelated change?

Also, always capture failures from clients such as web browsers. Whenever something is not working, you want to know this as soon as possible so that you can fix it!

And like the Midsummer error, it is important to dial in what counts as an error and what does not. You do not (like me) want to be stressed out by noise!

Agentic disclaimer

Given the present time, I might as well preempt some feedback right here. Using AI and agentic coding does not make any difference here. The four T's are just as important as before, if not even more so!

Thinking - agents give you the illusion of competence. Do not outsource your thinking to agents. By all means, rely on agents to help you think. And you should still rely on another human to review your approach.

Types - using well-defined types and keeping impossible states impossible is really helpful in order to utilise agents to the fullest. When you have well-defined boundaries that the agent can lean on, it produces better code, in my opinion.

Tests - having the agent use existing tests as guardrails, and provide new tests for future sessions, is, in my opinion, a MUST in order to use agents for coding. There are far too many assumptions and illusions that do not hold true, tests make this a much smaller problem.

Telemetry - nothing to add, really. You want to capture all errors, regardless of how the code was produced.

Flawless?

So, if I use these fantastic ideas when developing software, how come I still had to boot up my computer on Midsummer?

Types and tests are still only your thinking, only more formalised. Sometimes my thinking is incorrect. Sometimes my tests are incorrect or insufficient. Other times my types are too weak. Everyone makes mistakes.

However, when an error happens, I always add a new test that reproduces the problem, then I implement a fix. Damn if I would let the same error happen twice!

Part of fix is to take a step back and see if the design is flawed (the thinking), or if the types can be adjusted to prevent this from happening again?

This was my attempt to describe how I try to avoid the guilt of disappointing my users. I would be super happy to hear what you do, where you think I can improve my process, or anything else that is on your mind!

Please e-mail me with your thoughts, and thank you for reading.