Consistency and the damage done

My daughter needed to set up a new bank account for her savings. As she is under-age she could not do it herself. Neither could I, and neither could my wife. We needed to physically visit the bank office, both of us, during their almost hostile opening-hours to open it. Maybe she could just keep her savings under the mattress?

Or, there was an alternative. One of us could sign a letter of approval, attested by two witnesses.

The bank needed consent to open the account. Us, both, visiting was not a requirement.

Looking at software, I think we make this kind of mistake too often. We introduce the need for consistency not considering the cost. Consistency is an alluring attribute, it is often elegant to us, full of technical details, and leaves us knowing that our data is in a consistent state, we have done a good job.

Consistency requires coordination, and coordination sucks.

Introducing order and control

Recently I was in a feature discussion around a feature that spanned two systems. These two systems represent two different stages in a product life-cycle, and the user can synchronise state between them to some extent.

Let's call them system X and Y. Where data naturally goes from X to Y.

What should happen if you delete a thing in one of these systems?

A) It should be deleted in both.

B) It should not be allowed to be deleted in X until it has been deleted in Y.

C) Allow deletion in any system, mark it as deleted in the other.

Option A is nice, keeps the state synchronised and balanced. But, it requires coordination in case of failure mid-flight. Also, in our situation it was the case that there might be changes made around the Thing in system Y that the operator of X was unaware of. So it should not be automatically deleted, option A decides too much.

That turned our eyes on option B. Have the operators in Y make the decision on whether the Thing should be deleted. And only when it is deleted there should it be possible to delete it in X. This introduces another type of coordination. Not between systems, but between people, possibly working in different parts of the organisation.

If you thought coordinating computer systems is tricky, people are much worse, and they are dead slow1 too!

Imagine you are an operator in X wanting to delete a Thing, you have the following options.

The human Turing-machine option

You are now the coordinator of this, and you have to:

• Find the person responsible for the Thing in system Y.
• Explain or persuade them why Thing should be deleted.
• Have them log in.
• Find and delete the right Thing.
• Synchronise state between X and Y.
• Delete the thing in X.

Phew, now you can carry on with whatever you were doing. But hey, at least the system is consistent!

The Kafka option

You are the modern version of Josef K. but instead of courts and proceedings you have to file a support case in an unnavigable system with endless options.

We have all been here, first we have to select the right combinations of departments, people and labels to even be allowed to draft the issue. Then, it is the bouncing back and forth. And by bouncing, it is not like a tennis ball swishing by, it is like when an astronaut tosses a tool to a fellow astronaut, and you see it drift in slow-motion through space.

If you are lucky2, the Thing will be deleted in Y – but you are likely not aware of it since the support case is closed without comments.

Option C

The last option is what we settled on. Allow these systems to independently delete their representation of the Thing whenever they feel like it. When data is synchronised the Thing is marked as deleted in the peer system.

Now, hold it. Is this not an anti-solution? Causing littering and still requiring people to coordinate?

Well, the system does not fully know the intention of why something is deleted, maybe there is some reason why it should remain in X or Y? Instead, the Thing is highlighted and the operator can make their decision on what to do next. Should it be deleted, or should it be restored?

So the system has not caused any harm by assuming that it should be deleted everywhere.

And better yet, the system has not blocked anyone from doing their job.

The whiteboard

When you are designing software it is so easy to try to introduce control and consistency when there is none.

We are drilled to keep our data consistent, adding locks and coordinating actions. And while that might solve our problems, the real world is not as structured and compliant as we want it to be, and our solution might in fact introduce new problems in that world, the real one.

At the whiteboard, there is no problem deleting Thing in Y and then in X. It is just to draw a few lines and add some sequence numbers.

And when we, as engineers, start to consider concurrency, failures, distributed state and all the other really hard stuff, we are still at the whiteboard.

Do not get me wrong. We absolutely have to think about those things! But we should not stop here!

Designing for correction

Let us take another view on the options. I would argue that options A and B would have been poorly designed solutions in our specific case. It is not to say that they are never justified, they are. In situations were you cannot correct things affordably and in a reasonable time you might have no choice, such as:

• Safety - When people can come to harm.
• Security and compliance - When something legally must be consistent.

Consistency is a cost both in implementation and maintenance, but also in user interaction. And in this case there was no real need for immediate consistency, and the solution was better off without it.

Not having immediate consistency is not free though, now the system and users must handle the highlighted things.

• Where should the highlighted things appear?
• Who will deal with them, and how?

Option C is trying to adapt the system to fit the real world, designing for correction and not prevention. Option A is discarding details about local changes made. And option B is trying to adapt the real world to fit the system, preventing something from happening.

Design for correction when possible and affordable, and only take on coordination (to achieve consistency) when absolutely needed.

Domain-Driven Design

Making complex systems fit the real world is not something new. A popular approach to help deal with this is known as Domain-Driven Design (DDD).

In DDD there are three things in particular I think are fitting to shape the solution for these types of problems.

Bounded context. An autonomous part of a system that has its own vocabulary. A Thing in X is not the same as a Thing in Y. They can represent the same physical object, but the characteristics of it are different. Each bounded context has its own domain model. The opposite is an Enterprise Data Model3, where you enforce the same model across an organisation.

Aggregate. The consistency boundary is limited to an aggregate (an identifiable entity). These are a natural fit for single-writer principles (and actors). Aggregates are part of the domain model and should4 model state transitions and be modelled to allow correction of state.

Events. Facts about an aggregate can be modelled as Domain Events. These can be published to other bounded contexts as Integration Events. Events are just facts, not instructions. Any consuming bounded context interprets them by their rules.

Eventual consistency

Most of the time eventual consistency is centred around performance and availability trade-offs for distributed systems. At least when you drop the term among your engineering friends. Distributed systems are very interesting and offer plenty of challenges and really cool technical things like CAP-theorem, quorum, etc.

But I feel that, at least we as engineers, do not consider it as much as an organisational model. The real world is, to a large extent, eventually consistent. It is also asynchronous by nature.

In our example, A and B were already eventually consistent, option C just acknowledged it in the system as well.

I think designing systems to be eventually consistent should be done when the domain they represent is eventually consistent, not just to reach more nines.

Conclusion

Designing systems is hard, distributed ones, even more so. Be careful about when you introduce the need for coordination and when you can let it be. Most often this should be a business decision and not a technical one. Still we as engineers often take this decision already at the whiteboard.

Represent failures and corrections in your model. That lets both you and your users fix things in the future.

The next time you feel the need to introduce consistency, approach the problem from the people and organisations using your software. What are their mental models, and how is that represented in the software? Can two people disagree for an hour, a day, or not?

The best thing? Designing for corrections often results in less code. Having a system that can be corrected also means that your modelling mistakes are not fatal, which as an engineer, is priceless.