Day to day application of the CAP theorem

When someone explains the CAP theorem, they do it in the context of distributed data stores. But, if you understand the CAP theorem’s essence, you can use it in your day-to-day application design.

Nowadays, most applications(even rudimentary ones) are distributed applications—they invoke external APIs and services. Hence, understanding and applying the CAP theorem to application design is crucial for building robust applications.

CAP theorem says that you have to choose either consistency or availability in the face of network failures; you cannot have both.

CAP theorem is the programmer’s version of you cannot have your cake and eat it too.

Let us take a hypothetical application that exposes an API to register a user. For storing the user data, the application uses two datastores—MongoDB and MySQL. The datastores and the application backend are on different servers. When the client invokes the user registration API, the backend makes a network call and stores the data in MySQL and MongoDB.

For the sake of this post, I will define application availability as the backend returning a success response. I will define application consistency as the user data being available in MySQL and MongoDB on a success API response from the backend.

The above definitions are not strictly in line with the definitions of availability and consistency, as defined by the CAP theorem. I am taking liberties to convey the idea.

Designing the application using the CAP theorem

The backend and the datastores are on different servers. Since network failures are a given, the datastores might become temporarily unavailable to the backend. How should the application behave during network glitches?

Let us work with a specific scenario.

The client invokes the API; the payload reaches the backend, and the backend is trying to store the data in MySQL and MongoDB. The backend successfully stores the data in MySQL, but the backend cannot access MongoDB due to a temporary network failure.

You can design your application in two ways to handle this scenario.

Design option one

The backend stores the data in MySQL and returns a success response to the client. A periodic reconciliation job runs that queries MySQL, compares the data with MongoDB, and inserts the missing records into MongoDB.

With the above design, your application favors availability over consistency. When there are network failures, the backend is available(does not throw an error), but the data is inconsistent(present in MySQL but not in MongoDB). With this design, the application is eventually consistent—once the reconciliation job runs, the application becomes consistent.

Design option two

The backend returns an error response to the client when there are network glitches(when backend cannot communicate with the datastores). You put the onus on the client to retry under such circumstances.

With this design, your application favors consistency over availability. The application does not return a success response when it cannot store the data in both MongoDB and MySQL. You do not need reconciliation jobs in this design.

Every application these days is a distributed application. Hence, you need to understand the CAP theorem and use the CAP theorem’s essence while designing applications and thinking about the tradeoffs.

PS—To simplify the explanation and make it accessible, I have washed down the CAP theorem and the application failure modes. I will write a follow-up post, expanding the idea, and make it rigorous.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Diagram created using the amazing https://www.diagrams.net/.

Small company advantage

Below tweet appeared on my timeline.

The tweet brought back a quote(paraphrased from memory; I do not know the author).

Every employee does two jobs. The job and the job to prove that they are doing the job.

Arduous performance reviews, stakeholder management, timeline-based project managementlong term project planningjustifying project delays, and creating visibility are typically the work that one does to prove that they are working.

In a big organization with multiple layers of management and divisions, it might be tough to avoid these. There is no reason why a small business should follow the same policies and work style. Small companies should be optimizing for output.

A big business CEO has to be a master capital allocator. A small business CEO has to be a master resource allocator. Small businesses are perennially short of resources. An employees’ time is one of the most scarce and valuable resources for a small business. A small business should do whatever it can to maximize employees’ productive time. That means cutting all the things that people do to prove that they are adding value.

A small business’s competitive advantage comes from creating an environment where employees can spend their time creating value instead of posturing.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Engin Akyurt from Pexels

Do not rage against the machine

In life, we have to do many pointless things; at least pointless for us.

We have to fill unnecessary forms.
We have to get documents signed by random people.
We have to write foolish things.
We have to take part in pointless activities.
We have to follow senseless social norms.
We have to run behind people, asking them to address issues they should be pro-actively solving.

It is frustrating to go through this. When this happens, you feel the entire world is teaming against you. The sleeping rebel within us wakes up and wants to rage against the unfair world.

Do not let this happen.

If only raging solves problems. Instead, raging aggravates it. You lose agency of the task at hand—your brain switches from problem-solving to complaining mode. And, we will know how that turns out.

We usually get to know when our brain starts flipping. But, we do not try to stop it; we fuel the rage with an internal monologue of how unfair all of this is. The way to prevent this is by taking ownership of the task. You anyways have to do it, why not own it and finish it off quickly? When you take ownership of something, your brain finds innovative ways to hack through the task.

Do not rage against the machine; be the mechanic who fixes the machine.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Image from Wikimedia.

Why the heck is she doing that?

Good Will Hunting is the story of Will(Matt Damon), a janitor at MIT, who is an undiscovered maths savant. Will is an orphan who spends his spare time drinking with his buddies and getting into petty scuffles. Professor Lambeau(Stellan Skarsgård) discovers Will’s mathematical acumen and promises Will a fulfilling life. Professor Lambeau tries to get Will interested in working with him on hard math problems. Will resists his efforts, and Professor Lambeau cannot grasp why. Professor Lambeau approaches his psychologist friend Dr. Sean Maguire(Robin Williams) to help him get to Will.

Below is the explosive scene where Sean(the psychologist) explains to Professor Lambeau(maths professor) that Will’s life experiences make him behave the way he does.

Perspective dictates our actions and decisions. We do not act and decide based only on the data at hand. Our life experience plays a significant role in our actions and decisions. Perspective is how we color the information we have with our life experiences.

Two people might have the same information, but their life stories will compel them to make different decisions.

Someone who has made risky choices in the past and succeeded will tend to make more dangerous decisions. Someone who has made risky choices in the past and failed will lean towards conservative options.

We encounter situations in our lives where we cannot comprehend others’ actions. We think—what this person should be doing is obvious; even then, why is this person not doing that? 

Whenever we see someone acting differently than what we expect, the questions to ask are:

  1. Does this person have some information that I do not?
  2. How does this person’s life experience differ from mine?

Even though the above might sound obvious, this kind of structural thinking does not come naturally. Many just seeth at others acting weirdly without ever pondering why.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Fade to back

In this post, I will walk you through the principles to keep in mind while designing processes.

The north star:

If you want someone to do something, make it easy for them to do it.

When someone wants to start the fitness journey, the standard advice is to start with an exercise regime that is easy to do.

If you do not want someone to do something, make it difficult for them to do it.

If someone wants to stop eating junk food, the standard advice is to clean their house of junk food.

Keep the above two principles in mind when you think of processes.

mohammad-alizade-XgeZu2jBaVI-unsplash

What a process should not do:

A process should not eliminate autonomy and introduce coupling between individuals or teams.

A process should not kill creativity by making it difficult to do things.

Good processes are those that are autonomous and become a habit. A process becomes a burden when you have to spend energy thinking about it and doing it. If it becomes an integral part of doing things, it no longer feels like a burden.

Example:

You want people to wear seat belts while driving; you create a process to enforce this. Let us go through some of the different ways to do this.

A way:

Everyone has to take approval before they get into their car. They go to someone and say I want to take my car out. That person accompanies them to their vehicle, makes sure they wear seat belt, and then hands over the car keys.

The above is an example of a bad process. It is restrictive and will slow you down. It eliminates autonomy—you are always dependent on someone else to do something. It kills creativity by introducing coupling—you are making it hard to do things.

Another way:

You emphasize the importance of seat belt. You create awareness about the dangers of not wearing seat belt. When someone gets into a car, an automated message prompts them to wear seat belt. If they do not wear seat belt, the message continues to beep. You put stickers on dashboards of cars, reminding people to wear seat belt. You mandate painting seat belts in glowing colors to avoid missing them while entering a vehicle. After doing all these, you have random checks on the road for people who do not wear seat belt.

The above has all the hallmarks of a good process. You are trying to convert wearing a seat belt into a habit.

Yet another way:

Device a mechanism so that the car does not start until one wears a seatbelt.

The above is the best way to institute a process. It is autonomous, easy to follow, habit building, and does not let anyone not do it.

A process becomes a burden when it is not part of the way of doing things. In the seat belt example, you start your car and drive some distance. You suddenly remember the dictum to wear seat belt. You look for a safe place to stop, wear your seat belt, and start again. When you go through this, the process mandating seat belt feels like a burden. Contrast this with starting your vehicle by wearing seat belt. If wearing the seat belt while starting a vehicle becomes a habit, it does not feel like a burden anymore. The act of wearing seat belt fades to back.

Best processes are the ones that fade to back.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by mohammad alizade on Unsplash

Uncomfortable and Confrontational

No one has an incentive to tell you hard-hitting uncomfortable truths about yourself that you are blind to. Giving hard feedback is an awkward confrontational situation—to the person giving feedback and the person receiving feedback. We avoid confrontation with people we know.

This behavior(avoiding face to face confrontation) of ours might have something to do with evolution. In the yesteryears, co-operation was essential for survival.

kristina-flour-BcjdbyKWquw-unsplash

If you are stupid or your idea is ridiculous, no one is going to tell it to your face; they might give you hints.

Giving hard feedback takes effort. We feel these things; it is tough to verbalize. It is tougher to convey it in a way that does not offend the other person. There is nothing to gain for the person giving the feedback by putting in the work. It is an asymmetric situation.

When you ask others for feedback, most of the time, the other person knows what you want to hear. We hardly have a neutral position on anything. When we bounce ideas with others, we are after confirmation, asking them to endorse the thought rather than poke holes in it.

This is also why we find it tough to say no or why you hardly hear no from others. I am not talking of abject strangers but someone familiar. Saying no is awkward. When we say no, we believe we have to justify it. Nos are feelings; we find it hard to verbalize them. When we say no, it is a confrontational situation that we want to avoid. Hence we sidestep the situation by staying silent or saying yes and not doing it later.

When you meet someone who gives you hard feedback or says no to your face, keep that person close to you and thank your stars.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Kristina Flour on Unsplash

Does code quality matter?

What role does code quality play in the outcome of a business?

I know it when I see it—said a judge on pornography. Quality code is the same—you know it when you see it. It is challenging to define what quality code is. It is tough to come up with quantifiable metrics on quality code. But, the instant you see quality code, you know it.

Another reason for the difficulty in verbalizing quality code is that it lies in the beholder’s eyes. One person’s high quality is another person’s low quality.

Whenever you talk of quality code, a question pops up—how important is code quality to the success of a business(startup)?

volodymyr-hryshchenko-L0oJ4Dlfyuo-unsplash

I have wrestled with this question for ages. After spending years meditating on this question under a Bodhi tree in the Silicon Valley of India—Bangalore, I have arrived at an enlightened(also flippant) answer—you cannot quantify this, it matters for sure.

Whenever you talk of quality code and business success, someone usually points out at a business that succeeded despite horrible code.

Businesses are messy. Code is only a part of the story. There are other things that matter to a business’s success. It would be specious to claim a business succeeded despite bad code.

Many businesses that succeed with lousy code are in markets so good and have their timing right that they would have hit the home run anyhow. With quality code under their belt, the journey to the podium would have been pleasant.

A parallel I can think of is the importance of good health and habits. Conventional wisdom says that healthy habits keep you disease-free and lead to a long life. I can always point to a person who smoked and drank her way to a ripe old age. Conventional wisdom says that good habits lead to success. I can always point to a successful person with terrible habits.

Does it mean that good health and habits are immaterial?

Another problem with code quality is that you see its benefits gradually. It is a compounding effect.

final

The human brain finds it difficult to grasp compound interest. Albert Einstein said—compound interest is the 8th wonder of the world. He who understands it earns it; he who doesn’t pays it.

Compounding is tail heavy. During the initial days, it does not make your eyes pop. As time goes on, compounding gains momentum, reaching a crescendo at the end—the same with quality code.

Good code compounds positively. Bad code compounds negatively. Bad code gradually drags your business down, making it slow and sluggish.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Volodymyr Hryshchenko on Unsplash

Communication Architecture

Organizations do not give attention to their internal communication architecture. Internal communication evolves organically. Deliberately designing the internal communication architecture makes a difference.

By internal communication architecture, I mean:

  1. How does information flow?
  2. How do team members communicate with each other?
  3. When do they communicate?
  4. What is the medium they use for communication?

A decoupled(push-based or broadcast) structure for communication works the best.

franck-v-tiNCpHudGrw-unsplash

Guiding principles for a decoupled communication structure:

  1. People should not be polling each other for information.
  2. There should be a specific place for information lookup.
  3. There should be pre-defined contracts for the above.

Let us go through an example. 

Imagine an organization with a development(dev) team and a quality assurance(qa) team. Dev team deploys a build for testing. After the deployment, the qa team starts testing.

One way for the qa team to know the deployment status is to poll the dev team periodically and ask whether they have deployed the build.

Another way is to create a contract for the dev team to send a Slack message in a channel once they deploy the build. 

The latter broadcasting style of information dissemination is decoupled. No one has to poll each other for information. As long as the dev team adheres to the contract, and the QA team knows the place to look for this information, it works.

A simple test to figure out your organization’s communication structure:

If a person asks you for information, and you redirect them to a person instead of telling them the steps to find the information, your organization practices the polling style of communication. 

Polling Based:

Hey, how can I get this report?

You can ask Shyam to generate it for you.

Broadcast based:

Hey, how can I get this report?

Add yourself to this email group, and you will receive it regularly.

Polling based communication has the following downsides:

  1. It is anxiety driving for the person who is seeking the information.
  2. It irritates the person who is supposed to give the information.
  3. It does not scale as the team grows.
  4. All the above lead to unnecessary confusion and aggravation.

Push based communication leads to automation. In the dev qa example, the dev team can automate the publishing of the Slack message.

For the push model to work, everyone needs to adhere to the established contract. If one does not do that, the system collapses. 

Communication forms the cornerstone of organizational culture. Internal communication can make or break organizations.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Franck V. on Unsplash

When Not to Abstract

Software developers love abstractions. It is satisfying to hide the perceived ugliness and complexity of the underlying system in a so-called easy to use, beautiful interface. Also, abstractions are considered a good design practice.

The abstractions that I am referring to are those along the lines of structural design patterns like adapter and facade. ORM is a classic example.

The flip side of abstractions is the complexity that it brings. Abstractions would have bitten every software developer at one point or the other. Cursing ORMs is a rite of passage for web developers.

Poorly written abstractions cause more problems than they solve. Not to mention the countless hours wasted in designing, writing, and maintaining them.

Knowing when not to abstract is as vital as knowing when to abstract. 

lucas-benjamin-wQLAGv4_OYs-unsplash

When not to abstract?

  1. The abstraction does not add value.
  2. The underlying thing that you are abstracting is not a standard and is rapidly evolving.
  3. You are prototyping.
  4. You are short on time.

Before writing an abstraction, always introspect—what is it I am solving with this abstraction? What is the value add of this abstraction? 

Abstractions work well when they are written over components adhering to well defined, comprehensive standards and specifications. For example, SQL is a well-defined standard. SQL might evolve, but slowly. Hence ORMs are ubiquitous and work well for a majority of the use cases. 

When you try to abstract a non-standard rapidly growing platform, it becomes a game of catch up and drastic design changes each time the underlying platform changes. Without knowing the direction of evolution of the underlying platform, it becomes a disruptive change each time to the users of your abstraction.

When you are prototyping, concentrate on proving the idea as quickly as possible. Do not waste time writing layers of abstraction, which you might throw away in the future if the concept does not work.

Abstractions that add value require deep focus and ample time to design. If you are short on time, you will do a lousy job with the abstraction. It will cause problems than solving something.

In all the above cases, instead of structural abstractions, concentrate on utility functions. Writing utility functions for simplifying recurring patterns will give you more bang for the buck.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Lucas Benjamin on Unsplash

Fighting FUD

FUD stands for fear, uncertainty, and doubt. FUD is the strategy of influencing perception by spreading false and dubious information. Fighting FUD takes energy leaving no steam for real work.

Marc Andreessen, a Silicon Valley venture capitalist, recently wrote a post saying: It’s Time to Build. The gist of the writing is—in the US, people are no longer innovating and building core infrastructure—health, banking, transportation, finance, education, etc. anymore. The article is a call to arms to get back to building ambitious foundational projects.

While Marc Andreessen’s writing is in the context of the US, it is true, albeit to a smaller extent, in India too.

2-people-doing-karate-during-sunset-62376

What has led us to this?

We were in a homestay in a green estate that had a stream running through the plantation. The owner of the property told us how the government allows him to build and operate a compact hydroelectric plant on the stream, but he does not want to. He said that as soon as he would construct the plant, environmentalists would raise a hue and cry, create a ruckus, and he would end up wasting all his energy on that front, leaving him little energy to do other things.

The above is anecdotal, but you see a parallel whenever our government announces big-ticket, ambitious projects. There is always a cacophony of protests. We live in an age where everyone has a strong opinion on everything, and if that person is creative with words, she can multiply her reach, thanks to technology. Nowadays, spreading FUD is just a click away. Plus-oneing the protest satisfies the modern age’s thirst for wokeness and fuels the FUD. In such a situation, a government that lives by the Damocles sword of public opinion gets into perception management mode leaving little energy to solving problems.

Organizations are not immune to FUD. US intelligence, as back as in 1944, published a manual with tactics to sabotage workplace productivity. The manual was a field guide to be used against Axis powers in the world war.

Some gems from this manual:

  • Never permit shortcuts to expedite decisions.
  • Talk frequently at great lengths.
  • Refer all matters to committees.
  • Bring up irrelevant issues.
  • Haggle over precise wordings.
  • Advocate caution.

The above is how one creates an environment of FUD. One can see such behaviors to varying degrees at workplaces. In a culture where this is extreme, fighting FUD and preception management replaces work. Once this culture takes hold, it is impossible to weed it out—productivity nosedives to zero.


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog


Photo by Snapwire from Pexels.