Murphy’s Law Of Software Abstractions

All software abstractions, sooner or later, leak.

When this happens, it hurts.

To drive a car, you need not know how it works internally. The mechanics of an automobile is well abstracted from the driver. Similarly, software libraries, tools, and frameworks promise abstraction to the engineers using them. They promise that one can use them effectively without delving into the internals.


Let me tell you a story of uWSGI, processes, and threads.

uWSGI is a container for running web applications. It is popular in the Python world. Global interpreter lock makes concurrency muddy in Python applications. A way around this is to spawn multiple processes. While starting uWSGI, one can configure the number of processes to spawn to service concurrent web requests. The master uWSGI process starts, initializes, and loads the Python application code, and then forks the configured number of child processes. One caveat while forking is that the child process does not inherit any of the threads created in the parent process. Since uWSGI loads and initializes the Python application and then forks, the child process will not inherit any threads created during initialization in the parent process.

This bit us hard.

We were initializing a Kafka library on application load. This library internally created background threads that aggregate and push messages to a Kafka broker. The child processes uWSGI forked did not have these threads. Hence, our payloads were not flowing to Kafka.

In 99% of the cases, an application developer need not bother about how uWSGI forks and creates child processes. uWSGI abstracts this well, and the application developer can go on with her regular day to day work willfully ignorant of these abstractions. The same goes for the Kafka library. One need not pry open the library to figure out how it aggregates the messages and sends them to Kafka. In this particular case, the abstractions leaked and bit us. We spent a couple of days debugging.

One of the principles of good software design is that it should be easy to reason about.

You can interpret the above in different ways. A short interpretation is that when you look at a piece of code, it should be easy to figure it out in one’s head. Whatever you need to reason about the code should be in it and apparent – no hidden surprises. Leaky abstractions break this principle.

The uWSGI incident is fresh in my mind, but I have seen this to be a recurring pattern.

This is the problem with abstractions and magic technologies. When the abstractions leak, they bite us in ways we do not expect, and at times we do not anticipate.

  1. Be skeptical of technologies that claim to be magic. Do not pay only lip service to KISS, follow it in spirit. Think twice before incorporating such technologies into your stack.
  2. Figuring out the anti-patterns beforehand is as crucial as figuring out the best practices. All technologies come with a list of dont’s – grok them.
  3. Even though it may sound like an oxymoron, invest time and effort in going beyond the abstraction and peel the layers. At the least, build a minimal mental model.
  4. If you are a library developer, documenting how things can go wrong is as important as highlighting the rosy use cases; this is the least you can do to help your fellow craftsmen. Map the minefields, so the one does not accidentally trip on them.

1. uWSGI is a fantastic piece of software. I am not trying to diss on uWSGI at all.

2. For our particular problem, we disabled the default forking behavior of uWSGI. We enabled, lazyload, which loads the Python code after the fork rather than before.

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Image by Sang Hyun Cho from Pixabay

Charlatans and Us

Charlatan – a person, falsely claiming to have special knowledge or skill.

“How do we hire amazing engineers fast?” is a question people ask me often.

When someone asks me the question, they usually expect a profound answer, which will cure all their hiring pains. Hiring, especially good people, is a long, involved, and arduous process. There are no deep secrets to this. But, this is not what people want to hear because they already know this. Instead, they expect a magic potion, a hack, which will wipe out all the hiring woes. My standard answer to the hiring question is along the lines of – “I do not have any tricks up my sleeve to help you with that.”


When someone is expecting a profound answer, and you do not have any, it is very tempting to come up with one. When the other person is seeking enlightenment, and all you have is mundaneness, you feel like an amateur and a buzz kill.

Charlatans start like this. People expect magical answers from them; they do not have any. Still, the expectation from others is so high that they start coming up with one, and then it becomes a self-fulfilling prophecy. Also, once you do this multiple times, you start drinking your own kool-aid. You do not even recognize that you are a charlatan. You genuinely start believing that you are a messiah.

Growth is a catch-22 problem. You need to endure pain to grow. You are not ready to experience pain unless you see the growth. But you do not see growth unless you suffer pain. Charlatans, with their quick and simple hacks, give us hope of disproportionate returns by investing little effort, hence the demand for charlatans.

The rich and the famous are often called charlatans. We hold the rich and the famous accountable for lofty morals and weave stories of their impeccable character. We forget that they are just like us – winging through life, taking shortcuts, not knowing what is happening or where they are heading. When the rich and the famous know of these high expectations, which a lot of them do not have(there is nothing wrong with this), they artificially try to mold themselves on these lines. In today’s age of social media, where virtue signaling is just a click away, it is getting easier to do this. When you fake it, it can only go so far. One day, the cloak falls, and the grandness built on flimsy appearances comes tumbling down. We then start calling these celebrities charlatans, but little we introspect on our role in them turning out to be charlatans.

As much as we like to blame charlatans for their deception, a significant part of the blame rests on us for creating them. It is our unholy expectations and quick reward-seeking nature that gives rise to charlatans.

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Image by Pexels from Pixabay

The Games We Play

Two members of a criminal gang are arrested and imprisoned. Each prisoner is in solitary confinement with no means of communicating with the other. The prosecutors lack sufficient evidence to convict the pair on the principal charge, but they have enough to convict both on a lesser charge. Simultaneously, the prosecutors offer each prisoner a bargain – betray the other by testifying that the other committed the crime, or cooperate with the other by remaining silent. The possible outcomes are:

  • If A and B betray each other, both of them serve two years in prison.
  • If A betrays B, but B remains silent, A will be set free, and B will serve three years in prison (and vice versa).
  • If A and B both remain silent, both will serve only one year in prison (on the lesser charge).

The prisoners cannot communicate and come up with an optimal strategy.

If A betrays, then it is in B’s best interest too to betray. B will end up serving two years instead of three if B remains silent. If A does not betray, then also it is in B’s best interest to betray. B will walk out scot-free by ratting out on A. Same line of thinking applies to A too.

If A and B think only of their self-interest, they end up betraying each other — A and B will spend two years in prison. Instead, if they cooperate by remaining silent, they get out in a year — cooperating results in a better outcome. Optimizing for their self-interest ends up harming them instead of helping.

Game theory refers to the above as prisoner’s dilemma. Game theory is the study of human cooperation and the incentives driving us to cooperate.


Prisoner’s dilemma in a nutshell models situations in which individuals selfishly act in their self-interest, thinking it will benefit them when, in reality, it ends up harming all including themselves.

If you keep your eyes open, you can see prisoner’s dilemma everywhere.

You see it on the roads every day. All are selfishly optimizing for themselves by not following traffic rules, thus leading to detrimental traffic conditions for all.

You see it in companies where individuals and teams selfishly optimize for their narrow goals, which ends up harming the company.

You see it in the treatment of public resources like buses, toilets, and parks. No one seems to care about the upkeep of shared public resources, whereas caring for these resources would lead to a better quality of life for all.

Use prisoner’s dilemma as a lens to understand why people do not collaborate even when collaboration would have resulted in a better outcome.

Prisoner’s dilemma gives us a model to think about:

  1. Cooperation between people who do not know each other.
  2. The incentives to cooperate when the benefit of collaboration is not apparent.

Show me the incentives, and I will show you the outcome.

– Charlie Munger

In the face of non-communication and unclear incentives to collaborate, what would have lead to A and B cooperating?

Imagine that the criminal community had a strict rule of never confessing to the police. Breaking this code meant certain death. In the presence of such a system, perhaps A and B would have remained silent, leading to implicit cooperation. The Italian mafia has such a code called Omertà.

Imagine that in the criminal community, confessing to the police meant that your reputation is tainted forever. You will never find work again. In the presence of such a convention, perhaps A and B would have remained silent, leading to implicit cooperation.

If A and B had to work together in the future on other projects, perhaps A and B would have remained silent, leading to implicit cooperation.

If A and B were the members of a cult that says betraying a fellow member leads to eternal damnation in the afterlife, perhaps A and B would have remained silent, leading to implicit cooperation.

Many of the social constructs like strong laws, fervor nationalism, religion, trust, reputation, and community are society’s answer to prisoner’s dilemma. We, humans, have collectively evolved these practices as a way to facilitate implicit cooperation, thus leading to a better quality of life.

Thinking through the lens of prisoner’s dilemma explains why:

Small teams are more successful than big ones.

Scrawny resource-starved startups trump multinational corporations with deep pockets.

Tightly knit small communities have a lower crime rate than big cities.

Small homogeneous nations are more successful than diverse big ones.

How do companies solve the problem of prisoner’s dilemma?
Organization values, emphasis on team building and bonding, rewards and recognition, processes, and rules are some of the obvious ones. Some companies create a religious cult-like atmosphere as an answer to the problem of prisoner’s dilemma.

Implicit collaboration between people is critical to the success of everything – teams, projects, companies, societies, and countries; use prisoner’s dilemma as a way to think and model this. Thinking in terms of prisoner’s dilemma helps us to devise constructs that incentivize collaboration.

This post is not a rigorous explanation of prisoner’s dilemma; I have taken poetic liberties with it. Wikipedia entry on prisoner’s dilemma has a thorough explanation; it is an engaging read too.

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Photo by Perry Grone on Unsplash.

Becoming a Guru Programmer

Are you in awe of the Jedi programmers who seem to produce bugless code? Are you bewildered by the Guru programmers who fight inefficient code with their hands tied and eyes closed? They are not superhumans; these are mere mortals who have a repertoire of bug patterns in their heads owing to their experience. They have also mastered behavioral traits that aid in detecting bugs and flushing out inefficient code.

One can avoid the majority of bugs by adopting two behavioral traits.

  1. Taking a step back and asking – What can go wrong?
  2. Asserting your assumptions.

Let us work with an example.

The below code accepts a list and returns the first element.

def get_first_elem(lst):
    return lst[0]

What are the implicit assumptions that you see?

  1. No one will pass a null list.
  2. No one will pass an empty list.

What can go wrong?

  1. If someone passes a null list, the code errors out.
  2. If someone passes an empty list, the code errors out.

The idea is not to code defensively but to be aware of the assumptions and error conditions. It might as well be that the function should error out when someone passes a null or empty list; when it happens, it should not be a surprise.

Adopt these behavioral traits whenever you read or write code; you will be miles ahead of the rest.

Some of the other common bug patterns follow.

Never let it leak


Not closing opened resources – be it file descriptors, database connections, HTTP connections, or socket connections. Programming languages have constructs to do this – finally block in Java and Python, defer in Go lang. Whenever you open a resource, close it; never let it leak.

Fence it


When establishing a connection to an external resource, be it a database or a remote server; configure appropriate timeouts. Being stuck for an undefined period establishing a connection is not a happy place to be in; fence the connection establishment time to a reasonable value. Also, timeouts come in various flavors – connection establishment timeout, socket timeout, HTTP server timeouts. Familiarise yourself with all that apply to your scenario.

Bound it


Keep an eye on runaway resource creation. There is a limit to the no of connections that an external system can handle; there is a limit to the no of files one can create on a file system. Be aware of these limits and put in checks and balances to bound the creation to acceptable values.

Reuse over recreate


If you are opening a connection or creating an object repeatedly, check whether it is possible to pool the resources instead of repeatedly re-creating. Create a pool of resources once and then reuse when needed. This principle applies to all sorts of connections – HTTP, socket, database.

Using a resource pool alleviates the boundless resource creation problem too.

Encoding it right


While working with text, take care of character encoding. Things work great until one beautiful day someone passes a foreign text to your code, and everything collapses like a house of cards. Familiarise yourself with character encoding and take care of it while coding.

Stand on the shoulders of giants


If I have seen further, it is by standing on the shoulders of Giants.

– Isaac Newton.

The three chief virtues of a programmer are: Laziness, Impatience and Hubris.

– Larry Wall.

Doing everything yourself and not delegating to established libraries, frameworks, and tools is the root cause of a large number of bugs. In all probability, someone would have faced the problem that you are facing and crafted a well-tested solution to it – shamelessly use it. There is a reason why great programmers claim laziness is a virtue – follow this in spirit and practice.

Grokking these bug patterns and adopting the behavioral traits will make you a coding Yoda, who does not want to be one?

An earlier post on similar lines on Software Security.

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Yoda image by Mario Eppinger from Pixabay

Leaky tap photo by Luis Quintero from Pexels

Fence photo by Min An from Pexels

Recycle image by 95C from Pixabay

Encode image by Gerd Altmann from Pixabay

Elephant image by Sasin Tipchai from Pixabay