The goal was “write once, publish everywhere.” I prefer writing on my own blog, but once I publish, I want to share that content across platforms like X and LinkedIn. I was curious if I could use AI to automate this—specifically by acting on my behalf, opening a browser, and posting. Thus, the project commenced.

You can find the end result here: github.com/abhirama/ghost-ride.

The utility leverages Claude Code’s browser navigation capabilities to post blog content directly to social media.

Some Reflections

The Joy of the “Pointless”

This is a small experimental project, but there is an innate joy in building things. Every time I work on a personal project like this, I feel immense satisfaction. It reminds me of the concept of “Flow”—that state of total immersion where the challenge perfectly matches your skill level.

Much of my evolution as a programmer has resulted from these throwaway experiments. It’s a reminder of Steve Jobs’ “Connecting the Dots” philosophy—you never quite know how the skills from a weekend project will pay off later.

The Speed of AI

AI is surprisingly slow when mimicking a human user. I spent the majority of my time testing the tool, which involved painfully waiting for Claude to open Chrome, navigate to the sites, and type out the content.

“Vibe Coding” Still Requires Direction

I “vibe coded” most of this. Even for a small hobby project, it still required significant prompting and instruction to get it right.

The Complexity of Polish

Creating software is becoming easy and will soon be commoditized. However, creating awesome software that handles every edge case remains difficult. It’s a classic case of the Ninety-Ninety rule: the first 90% was a breeze with AI, but that final 10% of polish is where the real engineering (and most of the time) lives. Regardless, AI makes it incredibly easy to start, prototype, and validate ideas.

The Overton Window is a model for understanding how acceptable positions in society change over time. While originally a political concept, one can view any social phenomenon through this lens, including our prevailing social mores.

The ‘rules’ we accept are not permanent laws of nature. This is a difficult concept for a young mind to grasp. A kid usually views the world in terms of black and white—Right or Wrong, Good or Bad. They crave certainty. But, as we age, we see the window move. We see that most things in this world are actually different shades of grey.

Even in something as deplorable as taking a life, there are shades of grey. Killing someone is morally wrong. But, if the state does it as the death penalty, then it is legally acceptable. If you do it in self-defence, it is debatable in court. If it occurs as collateral damage in war, it is justified.

In a podcast I was listening to, the guest referred to traffic speed limits. If 70 is the speed limit, do you really think 69 is fine but 71 is wrong? Someone somewhere decided that a line should be drawn, and they chose 70. It could just as well have been 69. If you apply mathematical induction to this thinking, you can take it to either extreme.

But, at the same time, these rules, however arbitrary, are essential for the functioning of a society. Mike Munger, in the EconTalk podcast, gives an example that drives home this point. Imagine if at every intersection, all the drivers had to get out, debate, and come up with rules on who goes first—how cumbersome would driving be?

Becoming an adult isn’t just realizing the world is grey; it’s realizing that we have to draw black-and-white lines over that grey to keep society functioning.

The bigger question is: does it matter?

Have you ever audited your binary’s assembly code? You don’t. You trust the compiler. AI-generated code will head in the same direction, though the journey will take time.

We do not yet blindly trust AI-generated code because we lack the necessary checks and balances. Our current ecosystem—languages, tools, workflows—evolved with humans in mind. When that human is replaced by an AI, we hit an impedance mismatch.

Take dynamic languages. They are often less safe than static ones, yet we use them because they are easy for humans to read and prototype. But if an AI crafts the software, who cares about readability? Why not use a statically typed language where unintended errors are mathematically impossible?

But types only prevent invalid code, not wrong code. To ensure the logic holds up, we will have to change our workflows too. Test-First Development will return to vogue. Let the AI generate exhaustive test cases, then write the code to pass them. Follow up with fuzzers for correctness.

Even sacred software engineering practices like code review will evolve. Today, a human reviewer acts as a poor man’s compiler: checking for syntax errors, logic bugs, and variable naming conventions. This is low-leverage work.

If you insist on mandatory line-by-line code reviews for AI-generated code, you merely shift the bottleneck from generation to verification. You are still reading the assembly.

We must shift the burden to reviewing test cases.

When you review a test, you are validating the business logic and the edge cases, not the plumbing. You stop asking “Did they use the right loop structure?” and start asking “Did they handle the null user state?”

This is a higher order of engineering. Let the AI handle the implementation details. If the test suite is exhaustive and the fuzzers are running, the underlying code—like assembly—becomes an implementation detail we can afford to ignore.

We are entering a new era of programming. Don’t be surprised if new programming languages emerge—ones with built-in formal verification, designed not for humans, but for machines.

Performance engineering techniques broadly fall into the following categories, and you can mix and match them wherever appropriate.

Can I be lazy?
Re-use what’s already done—this is what caching is all about. Compute something once and reuse it until it goes stale.

Can I bring it closer to the user?
When you cache something in the browser, you’re bringing it closer to the user. The same principle applies to CDNs: they sit farther than the browser but much closer than the server. It’s all about reducing distance between the payload and the end user; the shorter the distance, the faster the response.

Can I parallelise the work?
Execute units of work simultaneously—that’s what threading and concurrency are about.

Can I reorder the steps?
Break a unit of work into multiple steps and see whether some steps can run in parallel or out of sequence. Combine this with re-use and bring-it-closer techniques. Can you cache some of those steps?

Can I use idle time to get more done?
If a unit of computation spends time waiting for something, use that idle time to compute another task. Event based concurrency uses this idea.

Can I defer non‑critical work?
Defer execution to a later time and respond to the user immediately. This gives the impression of snappiness even if the actual work happens in the background.

Can I pre-compute?
If you know something will be needed later, calculate or prefetch it in advance so that when the user asks for it, you can serve it instantly.

Can I reduce the total amount of work?
Sometimes speed comes not from doing things faster but from doing less. Simplify algorithms, use efficient data structures, and avoid unnecessary computation to reduce total workload.

Can I shrink the payload?
Shrink what’s sent over the wire—compress assets, optimise images, eliminate dead code, and lazy-load what’s not immediately needed. Smaller payloads mean faster load times.

Can I optimise the critical path?
Identify which operations directly impact what the user sees first (like rendering above-the-fold content) and optimise those paths for speed.

Can I bundle multiple operations together?
Combine multiple small requests or operations into a single one to reduce round trips, latency, and overhead.

Can I make it feel faster?
Show a loading screen or skeleton UI to give the illusion that progress is happening, rather than displaying a blank page.

Can I stream results as they’re ready immediately?
Instead of waiting for the entire process to finish, send results as soon as partial units are ready—this is what streaming is all about.

Things to keep in mind while working on performance:

1. Do some quick napkin math to estimate the headroom for optimisation.
2. Account for real-world constraints. The real world is messy—slow networks, dropped packets, and noisy neighbours can all disrupt your ideal optimisation plan.
3. Once you form a performance engineering hypothesis, profile your application to validate it. Many optimisations sound good in theory but fail in practice.
4. Measure the outcome using both profiling tools and real-world monitoring to confirm that your changes actually improved performance.
5. Lastly, balance performance with maintainability. Often, optimisations and maintainability pull in opposite directions, so aim for a pragmatic middle ground.

Facebook began with PHP and later built HipHop and HHVM—eventually evolving PHP into Hack to add optional static typing and better performance. Dropbox started with Python and later introduced large-scale static type checking across millions of lines of code. Stripe launched with Ruby and created Sorbet, their own static type checker, to bring type safety to Ruby. The pattern is clear: wild beginnings giving way to domesticated systems.

When you’re just starting out, you don’t deliberate much about programming languages and their nuances. That’s not top of mind—solving a problem is. You pick a language you’re familiar with and start hacking. You optimize for speed and freedom. Once you hit product–market fit and your codebase explodes—with hundreds of developers committing daily—you start craving predictability. But by then, switching languages becomes a stop-the-world exercise. It’s too late. So you adapt. You reinforce the old language, patching and constraining it until it behaves safely enough for scale.

Early in my career, I consulted for a startup built on PHP. When we had to build a new product, I naïvely chose Java. It was an uphill task. The team, used to PHP’s minimal friction, kept stumbling over Java’s verbosity. Back then, you could deploy a PHP “Hello World” site with a single file. Doing the same in Java meant learning the language, a framework, a templating engine, and a server like Tomcat. It was more scalable and robust, yes—but it came with a mountain of overhead. Looking back, PHP was a better fit for that small team and unproven product.

That contrast has stayed with me. Startups are wild—fast, instinctive, and full of creative chaos. As they mature, they become domesticated—structured, cautious, and focused on preserving what they’ve built while still trying to move fast. The transformation is inevitable, not just in culture, but in code.

In 2005, when I started my career, the popular, easy-to-learn languages were mostly dynamic. If you wanted type safety, Java was the de facto choice. But Java wasn’t considered cool—it carried the enterprise tag. Today, the landscape looks very different. Languages like Go, Rust, and Kotlin are popular, accessible, and increasingly used by startups from day one. Perhaps the trend of retrofitting dynamic languages for safety will fade, replaced by languages that balance freedom and discipline from the start. Perhaps the next generation of startups won’t have to choose between the wild and the domesticated.

If you are a product builder, you can probably empathise with the above meme.

Keeping the snarky meme aside — why is this happening?

Missing the bus

Whenever something is hailed as a generational shift, businesses become terrified of missing the bus. Missing the bus is expensive and often fatal. Think back to the smartphone era — if you were a desktop‑first website and ignored mobile, you were toast as upstarts captured your users. AI is that kind of moment for companies today. No one wants to be the one who “did nothing” when the industry moved.

Communicating nuance

What execs really want is for teams to think deeply and identify where AI genuinely improves things. But that’s a nuanced message. And nuance doesn’t scale. The larger the organisation, the harder nuance is to communicate. Absolutes, on the other hand, scale beautifully. So instead of “Use AI where it actually helps,” the message becomes: “Mandate adding a generative AI feature” or “Do not start coding without prompts.” It’s easy to communicate, easy to measure — and so, that’s what sticks.

Repeat, repeat, repeat

Repetition is a core part of management — saying the same thing in multiple ways until it becomes muscle memory for the organisation. Especially if something is deemed crucial. AI is crucial right now because no one wants to miss the AI bus. Hence the repetition—in slides, town halls, emails, training, and awkward product features.

So now you know why there’s an AI cacophony in organisations.

I had picked up this book a while ago. At the time, I lacked the background to fully grasp its ideas. I stopped reading after getting about a third of the way through.

Recently, I returned to it with a sidekick—Gemini. Each time I stumbled on a tricky concept, I snapped a photo of the page and sent my questions across:

• Explain the intuition behind this formula.
• What does this formula convey?
• Explain the concept XYZ in more detail.
• What is the author really trying to express here?
• Am I interpreting this section correctly?
• Explain this concept as if I were five.

This time, the experience was completely different. I’m now three-quarters through the book and have grasped its concepts—thanks to the power of large language models(LLM).

Earlier, I had written that we live in the golden age of learning, fueled by blogs, podcasts, and videos. But with LLMs, we have something far greater: a tutor, personalized to our learning goals, that can adapt to our learning style and is capable of explaining anything under the sun.

The best part is that instead of the tutor dictating your learning style, you can shape it to match your own. You can start with a broad overview and dive deeper when needed. You can ask even the simplest questions without embarrassment, and repeat them as often as you need—your tutor will never snap, “How many times must I explain this?”

LLMs, truly, make this the Rhodium Age of learning.

We often think, if only I did this one thing or if only I had that, my problems would vanish. And while that might solve one challenge, another quickly takes its place. Struggles, it seems, are a constant companion of existence.

Outside famine, serious bodily harm, or death, most struggles are a matter of perspective—what feels overwhelming to one may appear trivial to another. Outlook shapes our sense of suffering.

Children remind us of this every day. They approach us with troubles that seem catastrophic to them. From our vantage point, we may dismiss it as unimportant, yet for them, it feels like the end of the world.

I once listened to a podcast featuring a man who hears the worries of the wealthy for a living. His value lies in listening without judgment. One story stayed with me—a rich client fretting over where to dock his luxury yacht. To us, that may sound absurd, a classic first-world problem. But to him, it was a genuine source of stress.

There’s a memorable line in Fight Club: “The things you own end up owning you.” It sounds profound, and it is—but the inverse is also true. The things you don’t own end up owning you too—through longing, comparison, or regret. As the Seneca quote alludes, anxiety will never be lacking—whether you possess something or not.

The book The Black Swan by Nassim Nicholas Taleb ends with this: I am sometimes taken aback by how people can have miserable days because they feel cheated by a bad meal, cold coffee, or a social rebuff. We are quick to forget that just being alive is an extraordinary piece of good luck, a remote event, a chance occurrence of monstrous proportions. Imagine a speck of dust next to a planet a billion times the size of earth; the speck represents the odds in favor of your being born. Don’t be like the ingrate who got a castle as a present and worried about the mildew in the bathroom. Stop looking the gift horse in the mouth—remember, you are a Black Swan.

An average person today enjoys more material comforts than a monarch from the past. The king had no painkillers, vaccines, on-demand entertainment, or the simple luxury of calling someone instantly. Yet, despite all our modern conveniences, we often romanticise the past, recalling how good those old days were.

There is a name for this phenomenon— nostalgia bias. Nostalgia bias is our tendency to remember the past more fondly than it truly was, filtering out the hardships and preserving only the warmth and simplicity of earlier times.

I once read an explanation for this. As social beings, we rarely make absolute comparisons. Instead, we measure ourselves against others. So even though we live better than an ancient monarch, we don’t compare ourselves to him—we compare ourselves to Meena next door or that friend on social media whose life looks far more exciting than our daily grind.

Extending this reflection from the personal to the societal level, Europe has embraced the idea of de-growth—intentionally slowing down economic activity to protect the planet. Some Indians have also adopted this luxury belief. But for a developing nation like India, growth and infrastructure are essential to lift millions out of poverty. Also, wherever technology has reduced the need for human intervention, corruption has largely disappeared. For instance, getting a passport once meant chasing signatures and paying bribes at multiple desks. Today, the process is digital: you upload your documents, book an appointment, and receive the passport the very next day.

It might feel comforting to long for the past, but the future is where true comfort lies.

A Different History

Protests start with hope. Crowds believe change is near. They think entrenched powers can be dismantled. However, most movements fail to create real change. Power often shifts from one corrupt group to another. Are the Arab world or Syria better off since their revolutions?

On a recent EconTalk podcast, Mike Munger cited Douglass C. North, a Nobel laureate. He argued that societies need a different history for real change. Institutions—like laws and traditions—come from long histories. This leads to path dependence: once societies head in a direction, they often repeat it, even with new leaders. Revolutions that only change rulers usually fail because old institutions and incentives stay. True change needs a shift in historical and institutional foundations, allowing new possibilities to emerge.

The Vocal Minority and the Silent Majority

In many protests, a vocal minority takes charge while a silent majority endures. In Nepal, around 200,000 to 300,000 people protested, mainly in Kathmandu. With a population of about 30 million, that’s only 0.7% to 1%—visible but small compared to the silent majority. In the 2022 elections, 61% of eligible voters participated, far outnumbering protesters.

This trend is also seen on social media. Studies show the loudest voices online come from a small, active group, not reflecting the wider public opinion.

On Media Reporting

Media often fails to capture the true pulse of the people. It glamorizes revolutions, portraying protesters as heroes. This pattern is common in protest coverage. I’m skeptical of this view, especially with how foreign media reports on protests in India, often distorting events.

Most protests arise from the human desire for change, but they rarely ask if that change is better than the status quo. Psychology calls this the grass is greener effect, where alternatives seem better just because they are different. There’s also status quo bias, where the present is judged harshly without fully assessing alternatives. True transformation needs more than energy in the streets. It requires patience and an understanding that fast change often leads to disappointment. Psychology describes this as present bias—the tendency to overvalue immediate outcomes while undervaluing long-term consequences. Careful reflection is vital to ensure that the new path leads to real progress, not just a quick reshuffle of old patterns. Alas, a mobocracy cannot engage in this critical thinking.

After downloading a few episodes, AntennaPod pleasantly surprised me with a popup along the lines of: “You seem to be downloading a lot; do you want the download button here?” I responded yes, and voilà — the download button was now just one tap away.

This sparked a thought that had been dormant in my mind for a long time: why don’t applications radically personalize their user interface and experience?

Apps know which features I use most, and which ones I rarely touch. So why not make frequently used features more accessible for me, and push the rarely used ones into the background?

They know my common workflows — why not personalize the UI to support those?

They know which buttons I never click — why not hide them?

Beyond the technical complexity, the biggest reason I can think of for why this isn’t done more often is that this kind of personalization would make the app behave differently for each user, which in turn complicates UI/UX consistency and product decision-making.

I wish more apps would start doing this.

An iterated prisoner’s dilemma is just the classic prisoner’s dilemma played out repeatedly.

In the classic Prisoner’s Dilemma, two players can either Cooperate or Defect. Mutual cooperation gives both moderate rewards, mutual defection punishes both, and if one defects while the other cooperates, the defector gets the highest payoff while the cooperator gets the lowest. Read more here.

In the iterated version, this interaction is repeated many times, allowing for strategies that adapt based on past actions, modeling real-world relationships more accurately.

In Axelrod’s tournaments, participants submitted various strategies for this repeated game, and these strategies competed against one another. The simple tit for tat strategy outperformed all the complex ones.

Here’s what made tit for tat so effective:

• It was nice—cooperation was its default.
• If the other party defected, it responded in kind (tit for tat). It wasn’t a pushover or a pacifist.
• It was forgiving. If the opponent cooperated after defecting previously, it let go of the past and resumed cooperation.
• It was predictable—its response was easy for the opponent to reason about.

India seems to be following this very strategy in response to the recent Pakistani state-sponsored Pahalgam terror attack. I’m no defence or geopolitical expert, but if you go by Occam’s Razor which says that the simplest explanation is often the right one—India’s response looks like a classic tit for tat approach.

We retaliated with Operation Sindhoor, taking down terror bases deep inside Pakistan (tit for tat). Every attempt by Pakistan to escalate further was met with a measured, proportional response (again, tit for tat). At the same time, we’ve consistently emphasized our willingness to cooperate and de-escalate, if the other side is willing. The Ministry of External Affairs reiterated this message in every press briefing and official communication. Not once did the Indian state indulge in false bravado or chest-thumping like Pakistan did. Instead, throughout, we remained dignified, calm, clear, and firm. We’ve given the ceasefire a real chance—demonstrating that we are, indeed, willing to forgive.

This is a welcome shift from the past, when Pakistani state sponsored terror attacks were met with nothing more than strong words and no visible negative repercussions.

Money can buy a Lamborghini—but not a torso free of sagging love handles. It can buy a Birkin bag with a mile-long waiting list—but not the insight to see through the marketing gimmick behind such manufactured exclusivity.

But now, one of these is starting to crumble.

Enter GLP-1 weight loss drugs. A weekly jab can sculpt a fit body. As more people adopt them, prices will drop, making them increasingly accessible.

That leaves us with the mind.

And for now, at least, a wise mind remains shortcut-proof. It still demands effort, introspection, and the slow burn of lived experience. Neuralink might be the key that unlocks that final frontier—but until then, the grind is non-negotiable.

I was watching an Instagram reel from Fauxy (an account known for juxtaposing the absurd with deadpan, news-like delivery). It reminded me of a meme: I quit my job; it was interfering with my workout.

That sparked an idea—to make my own Fauxy-like video based on that meme.

I turned to AI. I gave Gemini the context and asked for a one-minute script. Then I fed that script into Invideo. I had a complete video in just a couple of minutes!

https://youtu.be/htWr44h4TTs

AI shortens the distance between inspiration and action, lowering(eliminating?) barriers to acting on ideas.

The future is exciting!

It has been some time since Elon Musk’s dramatic takeover of Twitter and the aftermath. Musk laid off a good portion of the Twitter team. A refrain I commonly hear is—see, Twitter seems to be running fine with a lean team.

Everything is great

Just because you anecdotally see nothing wrong with Twitter does not mean everything is fine and dandy with the product.

It takes time to see the effects of laying offs. Things slowly start breaking and then compound—like how good things compound, bad ones too compound. Bugs start creeping up slowly and then become chronic. Executing simple tasks takes a bit more time and then a lot more. Security reliability and developer productivity start taking a back seat to user-facing features; they are essential for the long-term health of a product but are challenging to justify in the short term, especially in a resource-crunched environment lacking technical leadership.

Doing more with fewer people

Musk has brought efficiency into front and center, which was overlooked in an environment of excess.

If you are in a big tech company, having more reports is equated with accomplishing more and is a path to promotion and growth. If you are in a startup, the bigger the team size, the larger the funding and valuation. Resumes and LinkedIn profiles beam with pride about having built mammoth engineering teams. There is hardly any incentive to do more with fewer people.

When a startup I was with got acquired by a bigger company, the work we were doing was done by a team 5x our size in the acquiring company. It was not that we were magically more efficient; we took deliberate decisions keeping our small team front and center. We did not have a not-invented-here syndrome. We leveraged open-source, cloud, and hosted products wherever possible, enabling us to build more with less people. Observability and developer productivity were integral to everything we made; this allowed us to move fast and debug problems efficiently without throwing people at them.

I am not justifying red-lining teams or inviting the dreaded 996 Chinese work culture. But, if you make operating with a small team a cornerstone of your culture, you can be lean and mean while maintaining sanity.

Ending thoughts

It takes time to see the impact of losing institutional knowledge. There is no way for an external person to briefly look at a product’s facade and proclaim things are fine. Building a good product is more grinding backstage work than what one sees in the limelight.

You can build successful companies with razor-thin teams. It takes deliberation, focus, and explicit product decisions; operating with a small team should be in the company’s DNA and not an afterthought.

It is tough to pivot a company like Twitter to this; that is probably why Elon Musk is rebuilding the company from scratch with shock therapy.

Photo by Samrat Maharjan.

As usual, I opened Hacker News to get my daily fix on the tech world and the esoteric. I was surprised to see a post titled: A case for toe socks on the front page. There was a lively debate in the community on this post that had garnered three hundred-odd comments.

I was surprised that something as mundane as toe socks could spark so much interest.

The post had nothing out of the ordinary; the author crisply narrated their experience wearing toe socks and how it improved their lives. I have been wearing finger socks for some time and could have been the one writing the post, but I did not.

Many hold back on writing publicly because they think they have nothing new to say; what had to be said on the topic has already been said and done. If they have to write, it has to be something novel and revolutionary.

This post and the ensuing discussion debunk that myth.

Writing is a way to share something you have discovered with others. It may not be new to this world, but it is new to you, and you have a unique perspective. You never know who might read your post, get influenced, and try it out.

Even if you have made someone’s life marginally better, is it not something to relish?

Image by Peggy und Marco Lachmann-Anke from Pixabay

As you age, you become a skeptic. Everything feels like re-runs of the past where something or the other has gone wrong; déjà vu. You are steeped in confirmation bias; you would have had enough time to search for evidence that confirms your priors; you become inflexible. Life would have punched you hard many times over.

It is natural.

Being a skeptic is good; being a cynic is not.

Questioning ideas is good; rejecting every idea as bad is not.

Being wary of the shiny new thing is good; romanticizing the past and yearning for it is not.

Pattern matching experiences, identifying problems, and coming up with solutions is good; complaining is not.

Besides trying to maintain a child-like curiosity as you age, a perspective that helps you objectively evaluate is to ask—what has changed since the last time I saw this happen and fail?

Many present-day successes are failed ideas from the past in a different context. If you take tech, for example, today’s equivalents to dot com busts like pets.com, Webvan, etc., are thriving. Technological advances in the past twenty years have made this possible—ubiquitous mobile devices, a rich application ecosystem, and blazingly fast internet, which were absent at the beginning of the 21st century.

It is tough to see this without deliberately trying to adopt this perspective.

Photo by Pixabay.

Whenever we hear a life story packed with wisdom, our reaction oscillates between two extremes. Either we go—oh man, this is so true; I can completely relate to this. Or, we say—there is no way this can be true; there is this one instance in my life that negates this story.

We take the story literally. We either agree or disagree wholesomely; there is no middle ground. Check social media. People go bonkers +1ing a post or say go to hell; there are no nuanced positions.

The right way to assimilate life stories is not to take them literally but to understand the essence of what the person is saying. Think of different episodes in our lives where the wisdom imparted could have made a difference.

We should not dismiss insights with which we disagree. We should get curious, reflect, and introspect about what this person sees that we are missing.

Perspectives we agree with only re-confirm our position. Perspectives we disagree with are an opportunity to update our beliefs.

Photo by Jose Manuel Gonzalez Lupiañez Photography.

https://twitter.com/austen/status/1328374281588273152?lang=en

One way to get extraordinary outcomes is by being consistent with the mundane. It may sound easy, but it is not.

Everyone knows what it takes to keep one’s weight in check. But, if you see around, every other person is overweight.

Doing the mundane once in a while is easy, but consistently doing it is hard. And, if you can do the mundane consistently, that becomes your edge. In the land of the blind, the one-eyed man is king.

On the other end of the spectrum lies doing something that absolutely no one else is doing—the Elon Musk kind of thing.

Like all things, this is not an either-or but an and. One usually needs to do both. But, our brains gravitate towards the latter ignoring the former. Being consistent with the mundane is not appealing. It is boring. But, doing something that no one else is sounds cool and haute. It is appealing. We over-index on this.

The contrast is fascinating.

Photo by Alice Yamamura on Unsplash

You have to write a document. You have a rough idea. You do not have the exact structure. You procrastinate, waiting for the ideal form to emerge. The deadline arrives. You are still not done.

We have all experienced this. It could be a document, a report, or a presentation.

We believe we have to do things linearly, from start to finish, beginning to end. That is not the case. If you have the end hashed out, start with the last part. If you do not know how to start but have the rest of the content thought out, pen that down. You need not go from start to finish linearly; it can be all over the place, zigzagging.

Once you have something written down, the structure starts emerging. You get ideas. The brain fog clears.

Try this iterative, non-linear process the next time you are stuck.

Image by Lisa Yount from Pixabay

All startups are dysfunctional. To some extent. Some more. Some less. As a founder, one should always be striving to reduce this dysfunction. It is a journey, not a destination. There is no end to it.

Two young fish were swimming in a stream. They see an older fish coming towards them. The older fish passes them, greeting, how is the water, boys? The young fish look at each other and mutter, what the heck is water? 1

When people spend time at an organization, they internalize the dysfunctions. They figure out ways around them to get things done. These might be sub-optimal. These might be taking 10x the time. These might be leading to inter-department strife. These might be dragging down the entire company. But, from the surface, it does not appear so. Everything looks hunky-dory. We do not see the obvious.

https://twitter.com/shreyas/status/1316573473934696449?lang=en

But, not so to a new hire. To her, these dysfunctions are very apparent She comes with a fresh pair of eyes and perspective. She is not yet indoctrinated. She has not yet internalized the system. The organizational dysfunctions stand out starkly to her.

Use a new hire, especially a leadership new hire, as your surrogate eyes to unearth the dysfunctions within the organization. You might be pleasantly surprised by what you find.

Photo by Lisa Fotios from Pexels

1 https://www.newyorker.com/books/page-turner/this-is-water

There is an expectation in startups that once a leader joins them; they should have a big win in the first 90 days.

I believe this is a wrong expectation to have and to set.

A new leadership hire in a startup should be spending their first couple of months in learning and assimilating the org context, not in making an impact.

Chesterton’s Fence

Chesterton’s fence is the principle that reforms should not be made until the reasoning behind the existing state of affairs is understood.

Imagine there is a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.“1

The undue pressure to show an early impact forces people to take short-term decisions detrimental to the long-term health of the company. It is like the person trying to tear down the fence without understanding why it was put there.

A new leadership hire should be spending their first couple of months understanding why someone thought it was wise to build the fence. Once they have this context, they can plan to pull the fence down. Or, maybe, there is a good reason why the fence exists, and it is not prudent to tear it down.

Image from Pexels.

1 https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence

There is no one best way to learn. The best way to learn is what works for you.

For some, it is visual, and for others, it is auditory.
For some, it is long hours of focussed study, and for others, it is tiny bite-sized sessions.
For some, it is reading text, and for others, it is watching lectures.

The best learning medium should be asynchronous, personalized, reflective, and in your control, not the other way around.

And, internet and social media have made it possible for anyone to learn through a medium and form that is best suited to their style of learning.

In the yesteryears, learning was possible only by sitting in front of a guru and listening to her discourse. Then came the written word and books that democratized learning to a large extent. But, not everyone likes to read. Also, not everyone had access to books.

Thanks to the internet and social media, today, learning is accessible to all in a form that suits their learning style.

If you are a visual learner, you can learn through YouTube videos and infographics.
If you are not into books, you can listen to an audiobook or a podcast.
If long videos cannot hold your attention, you can consume bite-sized information through short videos(TikTok, YouTube, and Instagram Reels).
If you only want an overview of the topic, you can skip the book and read a blog on the same subject.
If you want to mix entertainment and knowledge, you can watch a documentary.
If conversations and asking questions is how you learn, you can follow discussion boards and forums like StackOverflow, Reddit, Hacker News, etc.

By following the right people and channels on social media, one can expose themselves to new thoughts and ideas.

Internet and social media have eliminated the resistance to learning and made it easy; this is significantly underappreciated. Habit-building literature says that if you want to build a habit, make it small, fun, and easy to do. Internet and social media have precisely done this to the acquisition of knowledge. Earlier learning was seen as tedious, and rightly so, one had to be determined and seek the path of wisdom. Not anymore. One can start small, build their interest gradually and passively, even serendipitously, and then go down the rabbit hole.

Social media and technology get a lot of flak. But, if you are smart and know how to use them right, you have hit a goldmine. We are all in a golden age of learning, and it is upon us to make the best use of it.

Photo by Pablo Heimplatz on Unsplash

In a startup, there is an emphasis on creators and builders. This stems from the fact that a startup attempts to challenge the status quo, and hence you need innovative thinkers and builders who can do more with less. This line of thinking is right in spirit but can derail your execution if taken literally.

A startup’s tech execution strategy should be based on composition rather than creating something from scratch. In the " build versus buy" spectrum, the default should be “buy”; probably, borrow is a better term.

Borrow components, compose them to solve problems. Think of this like assembling a pre-fabricated house rather than building one from scratch, brick by brick. In today’s world, there is no shortage of components to choose from. There is open-source on one end of the spectrum, and on the other, you have proprietary software and in between, SAAS and cloud providers. Pick and choose components from this spectrum based on your cost and risk appetite and compose your way to success.

A critical skill a tech leader in a startup must have is the ability to recognize opportunities for composition, zero in on the right components, and orchestrate the composition.

Image by chenspec from Pixabay

W hat is a store of value?

A store of value is an asset that does not lose its value over time.

Take, for example, gold. Pure gold does not lose its shimmer as time goes by—it retains its gloss and quality. Contrast this with iron that rusts with time and becomes unusable.

Do not confuse a store of value with an income-yielding asset like a bank fixed deposit that gives you returns periodically.

Other examples of stores of value are land, antiques, vintage wine, art, and collectibles(comic books, sneakers, vinyl records, etc.).

There are two defining characteristics of stores of value:

1. They are desirable—there is a demand for them.
2. They are available in limited quantity—you cannot produce them on demand in large amounts.

If an asset is not desirable, no one would want to have it. If you can produce something en masse, it will not be valuable—at least not as an investment.

The price of a store of value is set by collective belief. As more and more people buy into this belief, the price of the asset increases over time. The limited availability of the asset aids this price increase.

Historically, stores of value have been the sole domain of the affluent(maybe excluding gold). Investing in stores of value is a tedious and prickly affair that requires knowledge, knowing the right people, and having the right connections. Also, most forms of stores of value need significant investment to own them.

How many people do you know who invest in art?

In other words, only the rich could afford to do this.

Enter bitcoin.

Bitcoin has upended the exclusivity of the store of value and has made it accessible to all. Anyone with an internet connection can invest in bitcoin. You do not need exclusive access and large sums of money to invest in bitcoin.

When I say bitcoin, I am referring to all forms of cryptocurrencies.

You can love bitcoin or hate it, but you cannot ignore it. Bitcoin has democratized access to store of value.

Photo by Moose Photos from Pexels

You have a database table. At the end of the day, you want to copy all the records for the day from this table to another table. Let us assume that you use the second table for generating reports.

You want to do this quick and dirty. So, you write a batch job to do this. You script your job. You schedule(probably using a CRON) the job to run a couple of minutes after midnight—say at 12:05 AM every day. You do this so that the job can copy all the records of the previous day.

The job needs a start and an end timestamp to copy the records from the source table to the destination table. The job can pick up the timestamp in two ways.

One way

The batch jobs has to copy the records of the previous day. Hence, it selects the timestamps as the start of the previous day and the end of the previous day.

Robust way

The batch job checks the timestamp of the last record in the destination table. It chooses this timestamp as the start and the current time as the end timestamp.

Why is the second approach better than the first?

The second approach is auto-healing in the face of failures.

In the second approach, if the script fails to run one day, the next day when it runs, it will automatically copy all the missing records. In the first approach, you will have to intervene and copy the missing records manually.

All or nothing

What happens if the job dies mid-way while executing?

Your destination table will have half-baked data. You can prevent this by using database transactions in your script to either copy all the records or none. This keeps your destination table consistent.

Make it idempotent

Make the copy operation idempotent by deduping the records in the destination table and deleting the duplicates. If the script is idempotent, you can blindly re-try in case of failures without worrying about duplicate records in the destination table.

Conclusion

The above scenario is an example. This pattern keeps popping up when you write batch jobs; adapt it to your context.

The two modes of failure with batch jobs are:

1. The job misses a schedule.
2. The job dies mid-way while running.

If you follow the above principles, your jobs will become resilient in the face of failures.

Image by Free-Photos from Pixabay

As an organization grows, so do the checks and balances to do things.

When you were a five-member company, provisioning a server was a click away. Not so when you are a 50 member company. There are processes to follow and permissions to take.

Maker-checker

A variation of maker-checker is the most common model followed in organizations to make sure that everyone adheres to the process and follows the laid out best practices.

In the maker-checker model, you have a maker who wants to effect a change and a checker who checks and approves the change. The checker is the gatekeeper and the protector of the best practices.

The most common example of a maker-checker model is the code review process. A maker wants to make changes to the code and proposes the change(in a pull request). The checker reviews the change and approves.

If there is only one principle that companies have to imbibe, that would be—make it easy for your employees to do the right thing. A manual maker-checker model does the opposite. It introduces friction and back and forth between the maker and the checker and makes it difficult for people to do the right thing.

Companies land up in this shit storm because they are not deliberate when introducing manual maker-checker steps.

What is the solution?

Automation to the rescue

The way to think about automating maker-checker steps is—how do I build guardrails into my system to only allow changes that follow the best practices?

Engineering tooling has evolved to a stage where many erstwhile manual maker-checker steps can be automated.

• Instead of asking your engineers to follow a certain style guide while coding, automate that with a linter and a pre-commit hook.
• Instead of asking your engineers not to create open S3 buckets, automate that with a CloudCustodian security policy.
• Instead of having an elaborate onboarding process for every tool your organization uses, automate that with an SSO solution.

The advantage of automation is that the engineers get feedback immediately and can fix their missteps when they are in the flow without having to wait for another human to check and get back to them in their own sweet time.

A manual maker-checker step is as good as the human behind it. If the human screws up(which we are renowned for), your system goes for a toss. Automation does not have this problem. Also, automation scales seamlessly as your organization grows. A system that rests on humans does not.

Conclusion

Whenever you have an urge to introduce a manual maker checker step to an existing process, think hard of the ways to automate it. Your engineers will thank you, and you will have a special place in heaven for doing this.

Image by Michal Jarmoluk from Pixabay

There is no I in team

Even though the above is cliched, it is packed with wisdom.

When you are an individual contributor, it is about you. Your effectiveness is measured by your contributions. Not so when you are a manager. It is no longer about you. Your effectiveness is measured by your team’s output.

This is a radical shift, and you need to internalize this if you want to become an effective manager. You need to shift the focus from yourself to the team and do whatever it takes to make your team effective.

Do not search for yourself in your team

Every member of your team is going to be different with their unique strengths and weakness. Do not seek your reflection in them. You are not seeking friendship; you are after proficient team members. Develop a sense of what makes your team members tick and play to their strengths. If their weakness is debilitating, work on a plan to mitigate that.

Not all glamour and glory

A manager is a representative and spokesperson for their team. Hence, they are in the spotlight. This might give an impression that it is all about glamour and glory. It is not so. There is hard, sometimes tedious work—writing performance reviews, stakeholder management, etc.—going on behind the scenes. If you want to be an effective manager, you have to ace these. If you are addicted to intellectual stimulation, many management tasks might seem dry.

As a manager, you do a lot of behind-the-scenes work. Your team members will not be aware of this. You do not get credit for this. You have to get used to this asymmetry and opaqueness.

Also, from time to time, you will have to give tough feedback to your team. It is not easy. It goes against our nature—we humans shy away from conflict. Giving feedback(both positive and negative) is part and parcel of being a manager. You have to master this art.

Photo by Gelgas Airlangga from Pexels

The software world is moving away from specialist roles to generalist roles. When I started working, the term “Full Stack Engineer” was not to be heard. Today, everyone knows the “Full Stack Engineer.”

What gives to this trend?

1. Better abstractions.
2. Economies of scale.
3. Market forces.

Better abstractions

Software abstractions are getting better day by day. You can do more with less.

In the yesteryears, if you had to deploy server-side software, you needed someone to:

1. Buy a physical server machine.
2. Place the server in a data center and hook it up to the outside world(networking).
3. Set up and upkeep the server.
4. Deploy your software on the server and maintain it.

Each bullet point in the above is a specialized role. If you wanted to build software in this era, you had to employ these specialists in your organization.

Today, all developers have to do in the Serverless or Platform As A Service(PAAS) world is—write their code and click deploy. The abstractions magically take care of the rest.

You can do away with the specialists and only employ generalist coders to build software now.

Economies of scale

Even in a Serverless world, someone has to buy the server, place it in a data center, and do the gamut of work. Since the demand for software is high— software is eating the world—only a handful of companies(and people) can specialize in this and expose the service to others in the guise of Serverless and PAAS.

The economies of scale of software dictate that not every company that builds software has to hire these specialists. The number of companies building generic software outstrips the specialist companies—hence the buzz about generalist engineers.

Market forces

Time to market is the mantra in today’s world. If you do not do it quickly, someone else will and eat your lunch. Companies want to be agile. In such a scenario, siloed roles within organizations dampen the pace.

Instead of having separate people(or teams) build software, test it, deploy it, and run it—which slows the pace—one person(or team) is tasked with these roles. Also, thanks to the abstractions, specialized knowledge required to be effective in these roles has reduced; a generic skillset suffices.

Take, for example, the role of a database administrator. Today’s databases have abstracted the specialist knowledge required to run, tune and maintain them so that a generalist engineer can do these tasks. You do not need a specialist.

The quality and maturity of tools(harking back to the point of abstraction) and frameworks make sure that the default settings(without extensive tuning and customization) give you a long runway for growth—with generalist teams. This also implies that a small set of specialists can service large generalist teams.

This trend will accelerate, as evidenced by the rise of N o-code tools that eliminate the need for generalist engineers too. We are not far from a future where a layperson can build and deploy sophisticated software without programming knowledge.

Image by pramit marattha from Pixabay

What is your edge?

Conventional wisdom says there are qualities that make one successful—hard work, persistence, curiosity, attention to detail, yadda, yadda.

All these are good. No doubt, one has to imbibe these. But, these are table-stakes. If you know them, so does every other person.

If you do what everyone else does, at best, you can be average.

Then, how can these be an edge?

Your true edge is your instinct(or intuition). Your instinct is what makes you unique. No other person in the world can have the same instinct as you; it is like your DNA, unique.

Instinct is what tells you when you should follow the rules and when you should bend them.

Learn the rules like a pro, so you can break them like an artist.

Pablo Picasso

Learn the form to break the form.

A martial arts quote

Your intuition becomes better when you have rich life experiences. The more you expose yourself to unique situations, more the opportunities you have to hone your instincts.

Photo by Oleg Magni from Pexels

When you read the history of humankind, you can see that our advancements have come from exploring(and conquering and inhabiting) novel lands and exploiting (the newly found) natural resources. This is illustrated in books like Sapiens and Guns, Germs, and Steel. Until the 19th century, the pace of these explorations was slow and gradual—there were still lands to be found and resources to be exploited. In the last couple of centuries, we went on an overdrive and exhausted all lands and resources. As of today, there is hardly a virgin land or a natural resource that humankind has not stepped on.

Where do we go from here?

Space.

Space is literally the next frontier for human development and progress.

Elon Musk and Jeff Bezos see this and want to be at the forefront of this race.

Imagine the possibilities that will open up when:

• We mine asteroids and planets for minerals and natural resources.
• Set up colonies on planets and branch off from Earth.
• Dump our ever-growing lethal waste and garbage on remote planets and make Earth beautiful again.

Arthur C. Clarke stated—any sufficiently advanced technology is indistinguishable from magic.

The possibilities are endless, and we are only getting started.

Photo by Miriam Espacio from Pexels

Bitcoin price has been going up like there is no tomorrow. Financial experts have opined that Bitcoin has no intrinsic value—it is a fad.

A group of nerds in a Reddit forum got together and drove up a lumbering video game retailer’s stock price by buying the stock in a frenzy. They did it partly in jest and partly as socking it to the man. Institutional investors were betting on the retailer’s stock price going down and got caught in the mêlée.

Telsa stock seems to be on the same trajectory as Elon Musk’s rockets. At one point, Tesla’s value was more than the combined worth of all the prominent car manufacturers in the world put together. Many believe Tesla stock is overvalued.

What is the lesson in all of this?

The right price for anything is what the market is willing to pay for it. The question that you need to concern yourself with is—are you comfortable paying that price?

Long ago, I had the snobbish view on Bangalore’s real estate prices—it is way too overvalued. Now, I say, I am not comfortable paying that price. Do you see the subtle difference? Who am I to decide what should be the right price of real estate in Bangalore? Someone is willing to pay; that is why it commands that price. All I can do is decide whether I am comfortable paying that price or not.

In Sapiens, Yuval Harari talks about how societal constructs are about the stories we collectively believe. Concepts like money, corporation, human rights, and religion are tales we all choose to believe. Since all of us(a majority at least) believe in these stories, they exist.

The same can be said of value too. If a majority of the people believe that a commodity commands a value X, then it does. What you can do is—you can independently come up with a value based on rational reasoning(as per you) or a story that only you believe at this point in time. You can hope that, over time, people will come to the same rational conclusion as you, and the value would converge to the price you believe is the right one. A key point to stress here is that this assumption hinges on the fact that people will think rationally, and eventually, they too will believe in the story that you believe in right now. If that does not happen (or your thesis on the value was wrong to start with), you have to be comfortable not owning that commodity. If you still want to own it, you might have to pay a higher price(as price appreciates over time as more people buy into the story)—that is the cost of your conviction.

There is a saying in the stock market—markets can remain irrational for longer than you can remain solvent.

Photo by Charles Deluvio on Unsplash

A simple yet powerful framework for life is:

1. Do many small experiments.
2. Double down on the wins.
3. Ruthlessly cull the failures.
4. Rinse and repeat.

As long as your experiments do not entirely wipe you out, you are on the right path.

Borrowing the above philosophy to product development.

1. Try many small product experiments.
2. Measure the outcome.
3. Double down on the wins.
4. Cull the failures.
5. Rinse and repeat.

A way of product development

You have an idea for a feature. You wear the user’s hat, chisel the feature, add embellishments and create the complete end-to-end implementation plan. Since the feature is in your head, you do not know how your users will perceive the feature—you have to assume things. You develop the feature and release it. All this time, you are building in a vacuum—without real-world data. Since the feature is end-to-end, you spend significant time, energy, and resources on developing it. It might turn out that your assumptions were faulty, and your users do not see the value in the feature. All the time and effort gone for a waste.

Since you invested vital time into the feature, you spend more time trying to fix it. Nothing comes out of it; finally, you let it go, and the useless feature lingers on in the product. Instead of analyzing what went wrong and drawing lessons from it, you search for proof to justify why the feature was essential and how it was not your fault that it did not click.

“The way” of product development

You have an idea for a feature. You create the smallest possible experiment(MVP) to validate it with real-world users. You build the MVP, release it, and collect data. If you see a positive outcome, you double down and enhance it. If not, you nix the feature and move on to your next idea.

Since it was a quick experiment that did not require considerable energy, time, and resources, you are happy to call it quits and draw lessons from the failed experiment.

With this methodology, you are not building castles in thin air. You are continuously iterating using real-world user feedback. As long as the cost of doing these experiments is negligible, you can carry out many of these, collect data and chart the course of action based on the data.

A word of caution

This is not an excuse to be sloppy or try random things. With great power comes great responsibility.

Photo by Michael Marais on Unsplash

You want to write regularly. You sit down to write. You are blank, bereft of ideas.

We all have ideas, but the problem is—the ideas do not surface when we want them to. Start an idea-list. Whenever something crosses your mind, note it down. Google Keep does the job for me.

You want to write regularly. You have nothing unique to write.

There are no stark new ideas in the world. Everything is an improvement over existing ideas. Saying something that has already been said but from your perspective makes it unique.

You want to write regularly. You do not have the time to write.

Designate half an hour a week on a particular day for writing. Sit down at the designated time. Pick a topic from your idea-list and start typing. It does not matter whether it is structured, whether it makes sense, whether it is polished. The goal is to type, not to be a perfectionist. Once you have exhausted what you have to say, start from the beginning. You will find numerous ways to refine your writing, polish it. Do this a couple of times.

You have the draft. The draft is not good enough for publishing.

The draft is never going to be good enough. Designate a day in a week to publish your writing. On this day, no matter what, you hit the publish button. This rule acts as a forcing function to tame the perfectionist in you.

Writing is not easy. People who do it as a profession complain. The above tactics make it easy to start(with the idea-list) and create forcing functions(rules of designated day to write and post) to refine your writing and publish.

If you liked this, you will love my other post on writing.

Photo by Lum3n from Pexels

Lagom is a Swedish word that translates in spirit to “just about right.” I came across this word while listening to Tim Ferriss’s conversation with Daniel Ek(Spotify founder).

The design phase of an application is a stressful time. Developers have to grapple with unknowns and make assumptions. While designing applications, a recurring thought pattern is—what architectural details should I obsess over, and what can I relegate to the future?

During the design phase, the aim should be to get the architecture “just about right.” Chasing perfection is like trying to find the mythical elixir that turns metal into gold.

What does it mean to get the design “just about right”?

In decision making, there is the concept of reversible decisions and irreversible decisions. Jeff Bezos popularised this idea with his analogy of doors. Reversible decisions are like two-way doors—you can go through, and if things do not work out, you can come back. Irreversible decisions are like one-way doors—there is no coming back. Do not deliberate too much while going through a two-way door; one can accumulate experience and information, and if things do not work out, one can come back. Do spend quality time before deciding to go through a one-way door—it can be a dead-end if things do not work out.

Think of your application design in terms of one-way and two-way doors— there are things that you can change later once you have enough data, and then there are things that are tough to change.

Example of tough to change things—the programming language and the framework. Examples of things that can be changed later—design patterns, application server, and caches.

When designing an application, aim to get the architecture “just about right” using the rubric of reversible and irreversible decisions.

If you liked this post, you will enjoy my post on how to build scalable applications titled " The three pillars of scalability."

Photo by Oleg Magni from Pexels

What is the sound of one hand clapping?

The above is a Zen kōan.

Kōans are open-ended, short, pithy statements(questions or riddles) that one reflects upon. Kōans do not have an accepted answer. The idea behind kōans is that the act of contemplating(journey) on the kōan is more important than the answer(destination).

A parallel to kōans is the Socratic method.

The Socratic method is a teaching style involving a dialogue—asking and answering questions—between a student and a teacher to stimulate critical thinking.

Giving feedback and coaching is a crucial part of being a leader. The best way to do this is to encourage your team to think by asking pertinent questions and motivating them to arrive at answers independently.

Plant a seed in their mind by asking the right question and let them go down the rabbit hole in search of answers. The process of doing this will enrich them more than you serving the answers on a plate.

This method works great with parenting too.

Photo by Austin Ban on Unsplash

I love listening to podcasts. I started three years ago, never looked back. Thanks to podcasts, reading books has taken a backseat. Authors appear as podcast guests when they launch their books. Podcasts seem to be on their way to becoming the primary marketing platform for book launches. During the show, authors highlight the big ideas from their books. One can get a concise book summary by listening to these conversations.

I listen to podcasts when I walk, run, cycle, drive, and sometimes while I do household chores. This is what I love about podcasts—I can mix them with physical activity and mundane tasks. As you progress in life, time becomes valuable, and podcasts help me optimize my time while satisfying my craving for knowledge and wisdom.

I subscribe to more than 90 podcasts. Not all episodes of a particular podcast are worth listening to; I have to pick and choose. Some of the memorable shows that I have heard are from podcasts under the radar with non-descript people.

Below is the system I use to manage the 90 odd podcasts with minimal time investment.

• I subscribe to podcasts through Google Podcasts. I have tried other apps in the past; I have settled on the Google one. Google Podcasts shows a red dot(like a notification) on a podcast whenever a new episode is released.
• I keep a queue(akin to a playlist) of episodes to listen to from various podcasts. Every Saturday(or Sunday), I replenish the queue. I limit the queue size to ten.
• When replenishing the queue, I go through new episodes and add them. I stop when my queue has ten episodes. For example, if my queue already has eight episodes, I add only two.
• When I listen to podcasts, I have a rule to avoid the sunk cost fallacy. Ten minutes into a show, if I find it uninteresting, I stop and start the next one.

The above system helps me listen to a large and diverse set of shows without wasting time managing them.

Image by 🎄Merry Christmas 🎄 from Pixabay

During pre-historic times, when going to the office was a thing, there used to be a traffic policeman in my commute route. The archetype of a police person is someone who is grim, officious, with a no-nonsense attitude. This policeman was the opposite. He was well-groomed, neatly dressed, fit, had a big smile on his face, and used to greet the passersby. I could not help but think—wow, must be a great policeman. Mind you, I had zero data(that matters) to evaluate his performance; all I saw this person do was manage a trickle of traffic with a smile.

I underwent a routine health check. Once the reports were in, I had a consultation with the doctor. The doctor ran through the report, explained it eloquently, answered my questions patiently—he did all this with a pleasant smile on his face. I could not help but think—wow, must be a great doctor. Mind you, I had zero data(that matters) to evaluate his performance. He did not do any investigative or diagnostic work; all he did was explain the report to me. I could have probably done the same through Google.

It is easy to mistake friendliness and excellent communication skills for being good at the job. Charm can mask inefficiencies and give others the impression of you being great at what you do.

When you evaluate someone, discount the charisma. Do not mistake affability for effectiveness.

The policeman and the doctor might be highly competent at their job.

I am not saying you should be a jerk. Having a pleasant personality will do wonders for your life. If it does not come to you naturally, work on it.

Photo by chaitanya pillala on Unsplash

By now, you must have read many radical takes on the future of work. Whenever I read these extreme predictions, two things come to mind—regression to the mean and Amara’s Law.

A layman’s translation of the statistical term regression to the mean is—what goes up comes down. Regression to the mean is not directly translatable to the remote work predictions. Still, a loose way to think about it is that remote work is being overhyped, and the majority of the companies will revert to working from the office once the pandemic is over.

Amara’s Law states that—we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run. You can apply this law to any recent, top of the zeitgeist phenomenon. In this case, replace technology with remote work.

One thing is for sure; remote work has become mainstream. Before the pandemic, remote work was the domain of heretics and hippies. These people were very vocal about it; few paid heed to their preaching. Now, they get to say—I told you so, you did not listen to me; it took a catastrophe to change your mind on something so obvious.

From now on, when anyone starts a company, not setting up an office and letting people work from wherever they want will be a valid, socially acceptable option. In the pre-covid world, it would have been diabolical to think along these lines.

Many companies will announce(already announced) permanent remote work(or remote first) and when problems crop up, think of reverting to the office. When things do not go as expected(which is bound to happen from time to time), leaders will search for a scapegoat, and remote work will be the easiest to blame.

Companies that revert to working from the office will face resistance from employees after some time. Initially, people might like going back to the office. It might feel like a good break after a long period of remote work. But, once you start spending a significant chunk of your day commuting to work, people will yearn for the days when they could work remotely from the cozy comfort of their homes, avoiding the mind numbing commute. Young people who socialize at work and can shift residences based on their office location might not have this problem, but people with families will reminisce their remote work period.

It will rain bonanza on developers who will have many choices to pick when it comes to where and how to work. If the work is remote, what prevents a foreign company(or a young foreign startup) from starting an overnight Indian(or any other country where there is a critical mass of talent) office where they can hire the same(or better) talent? All that is needed is a group of people working in the same timezone. Many do not even agree that the timezone matters. I predict a war for talent in India, and Indian companies will have to compete with increased pay scales(at least for the top percentile of the skilled folks) that foreign companies can afford.

At the cost of stating the obvious, companies opting to go remote will be able to tap a diverse talent pool, which is excellent for companies and employees. Remote work will help in alleviating the stress on tech-centric cities.

People fresh to work will be at a significant disadvantage when compared to tenured folks. It is tough to remotely re-create the mentoring that one would receive when physically sitting next to each other; freshers need this the most. Tools might be able to bridge this gap; it will be interesting to see.

People will self-select their employment. Some folks rely on the environment(office) to motivate them to get things done; they feel more productive in an office. They cherish the human connection that stems from working with their co-workers in the same physical space. These people will find it tough to work remotely for long periods. Hence, they will opt for companies that have a robust physical office culture. Remote work favors those who are intrinsically motivated and can self-manage. Such people will naturally gravitate towards remote work.

Remote companies will have to invest in processes, tools, and written communication. While working in the office, you can get by without these, but it is impossible to do it in a remote setting; it becomes dysfunctional. Even though it is easy to institute these, entropy is the way of the world—people will veer off this path unless someone is continually reminding them. If you invest in processes, tools, and written communication, you can eliminate middle management— bullshit jobs might reduce as a side effect.

Some believe that physically working together is essential to do great work; that is where the magic happens. Some are vocal about remote work and believe that it is a significant productivity booster; letting people work on their own time and terms is the future. Ultimately, the stories that the leader believes in will shape how a company operates.

Image by InstagramFOTOGRAFIN from Pixabay

A simple way to think about over-engineering is—are you wasting your today for an unknown tomorrow? “Unknown” is the keyword here.

Are you building for a million users—at a time—when a couple of hundred users use your application?

While building features, are you trying to accommodate fictitious(assumed without any basis) future needs?

Are you using technologies that you do not need today, with the assumption you might need them in the distant future?

What leads to over-engineering?

Will this scale?

Is a question software developers keep getting asked. It is a question software developers lose their sleep over.

To mitigate the scaling question, developers tend to over-engineer. From day one, they try to take care of that momentous day when their application will have a million users(they lose sight of the fact that, right now, there are only a couple of thousand users using their application).

Let me tell you a dirty secret— scaling is a commodity today; the cloud has made it so. While designing your application, as long as you follow the three pillars of scalability—statelessness, idempotency, and coding to interfaces—scaling is a breeze.

https://twitter.com/abhyrama/status/1329877264234913794

Lure of abstraction

Software developers love abstraction—it is satisfying to hide the perceived ugliness and complexity of the underlying system in a so-called easy to use, beautiful interface.

There is a thin line between abstraction and over-engineering. Read more about this here.

Future needs

When developing features, it is intellectually satisfying to feel that what you are creating today will cater to every future use case(irrespective of whether it is needed or not); in pursuing this euphoric feeling, people over-engineer. Instead, build to a rigid scope, assuming that you will need some enhancements and changes in the future.

Buzz word driven development

Using technologies that are in vogue today without thinking whether they add value or not. Because someone wrote a blog post about a shiny new technology does not imply it will add value to your application stack. Follow the pragmatic school of thought— use boring technologies.

A majority of the bugs that I have seen in my professional career have resulted from over-engineering—bugs arising from using components and technologies that one does not need or understand.

What is not over-engineering?

Things you do to increase your time to market is not over-engineering. They might have a one-time cost(when working on them) but will pay off due to the subsequent lower time to market. Automation and engineering tooling that focus on developer productivity fall in this category. It is easy to dismiss these as over-engineering, but they are not.

Anything that makes it easy to ship features faster is not over-engineering(how and what you should do is a nuanced question).

Like any subjective matter, over-engineering is contextual—depends on the team, the stage at which the company is in, the engineering culture, the kind of market the company is in, and one’s skills. One man’s over-engineering is another man’s under-engineering.

Comic from XKCD

Your company has planned a massive marketing campaign. Millions of households are going to view the company’s advertisement during a prime time television show. An enormous number of users are expected to try out your company’s application. You, the engineer, has been given a couple of weeks to scale the tech stack to handle this massive traffic surge.

What do you do?

I am assuming that you are using a public cloud. If not, good luck(use this opportunity to make a case to your decision-makers to move to the cloud).

Preliminary

Get a rough sense of the traffic surge—take help from your marketing team for a guesstimate. Use this number as the anchor in all the subsequent steps.

Horizontal scaling is the mantra

All cloud providers have some form of autoscaling—on-demand addition of servers based on specific conditions. Utilize this to the hilt. Figure out the thresholds for autoscaling. There will be a lag between the autoscaling condition being met and the addition of the server—plan for margin of safety. Lean on the side of over-provisioning than under-provisioning when the entire world’s eye is on you.

If you have sticky sessions or local server persistence that span multiple requests, now is the time to refactor your application to be stateless—on a war footing.

Make scaling someone else’s problem

CDN is a one-word answer to all scaling problems. Wherever possible, offload traffic to a CDN. One successful result from the CDN is one less hit to your backend.

Like CDN, many other frameworks help you scale seamlessly—Firebase, DynamoDB, etc. Leverage these to the maximum.

Cache, cache, cache, and cache some more

When one thinks of a cache, Redis or Memcache come to mind—do not restrict yourself narrowly to these.

Visualize your application in layers and see what can be cached(sometimes implicitly) at each of those layers—client, proxy servers, load balancer, web server, application server, middleware, database, and the operating system.

HTTP has many ways of caching—cache headers and ETags. Utilize all these to the hilt. One less hit to your backend is one less scaling woe.

If you are not using connection and object pooling in your application, now is the time to look into those— creating and closing connections(and objects) can get expensive, especially under heavy load.

Know your limits

If you are doing this exercise for the first time, your stack—server, libraries, VM, OS, and dependent components like databases, caches, etc.—are probably running with defaults that came out of the box.

Identify all the parameters(configurations) that matter and tune them—connection limits, file descriptor(IO) limits, thread counts, timeouts, pool, and memory settings.

Trace the critical paths in your application and gauge third-party dependencies(external services from other vendors) on these paths. Talk to these third parties and make sure they are ready to handle the extra load coming from you—do not let them be the weakest link in the chain.

Talk to your cloud provider(s) and ask them to increase the limits. All cloud providers have soft limits on resources. Check with your account manager and request a limit increase.

The chink in your scaling armor

The database is going to be the chink in your scaling armor. Use the traffic guesstimate you did to make sure that your database will hold up. Know the limits of your database(previous section) and tune for performance. Simultaneously, do not underestimate modern-day databases and servers. They can handle a ridiculous amount of load as long as you are not doing something stupid(if you are a hyper-growth startup, you probably are doing multiple stupid things; time to fix those).

If you believe your database will not be able to service the load, cache, cache, and then cache some more. If you have write-heavy traffic, then pray to God. Or, use a queue( async processing) and/or figure out a way to delay and spread the writes(write-back cache).

The ever seeing eye

Have observability in place. Murphy’s law states that whatever can go wrong will go wrong. When this happens during stressful times like a high decibel marketing campaign, and you are running blind to what is happening, cortisol levels go through the roof and tear you to pieces. Observability is the calming friend you lean on during these pressing times.

Conclusion

Have a checklist of all these, and make sure you tick them off as and when you close them. Having a list is critical as these can be stressful times, and losing track of what needs to be done, what has been done, and what is pending is easy.

You might be wondering why I did not ask you to do a performance test.

Doing a performance test and figuring out all the bottlenecks would be the theoretical(and systematic) way to go about this. The problem with performance testing is that carrying out one is challenging and requires expertise and experience. Also, mimicking real-world traffic and user behavior through a performance test is tough. If you know how to do that, you might as well know all the scaling tricks that I mentioned here(you are not the audience for this post).

If it is not obvious, I am borrowing the term Blitzscaling coined by Reid Hoffman.

Image by Johnny Gutierrez from Pixabay

Ghost Ride

GhostRide is a browser automation tool that uses Claude’s Computer Use capability to automatically create social media drafts from blog posts. One command extracts the blog content, captures a screenshot, and creates drafts on Twitter, LinkedIn, and Facebook.

https://github.com/abhirama/ghost-ride

Kwery

Kwery was my attempt at a single-person lifestyle business. The business did not work out; I open-sourced the code.

https://github.com/kwery/kwery

External Calls

Accompanying Jupyter Notebook for my blog on external calls.

https://github.com/abhirama/external-calls-blog

Selfie

Selfie is a Jenkins plugin that lets you configure recurring jobs born out of a practical need at work.

https://github.com/abhirama/selfie

SmushIt

SmushIt is a command-line tool which losslessly compresses all the images in the passed directory using yahoo!‘s smush.it image compression utility.

https://github.com/abhirama/smushit

Coursera Download

Coursera Download is a Python script to download Coursera lecture assets.

https://github.com/abhirama/coursera-download

XML Splitter

XML Splitter is a Java utility to split a large XML file into well-formed smaller XML files.

https://github.com/abhirama/xml-splitter

PHP Police

PHP Police is a small specific PHP linter.

https://github.com/abhirama/php-police

Tab Jumper

Tab Jumper is a Google Chrome plugin that lets you name tabs and then switch to them based on their names.

Example WordPress Plugin

An example Wordpress plugin that I wrote for a blog post.

https://github.com/abhirama/wordpress-example-plugin

n-puzzle

n-puzzle is a Facebook game of n-puzzle written during the days when Facebook games were all the rage.

https://github.com/abhirama/n-puzzle

Geek Nav SoundCloud

Geek Nav SoundCloud is a Google Chrome extension that lets one navigate SoundCloud using one’s keyboard.

https://github.com/abhirama/geek-nav-soundcloud

Robot Fruit Hunt

Robot Fruit Hunt is a solution to the initial version of https://github.com/scribd/robot-fruit-hunt.

https://github.com/abhirama/robot-fruit-hunt-uber

There are charismatic people in your team. They know how to talk glibly and build a reputation in the organization. They excel at the art of self-promotion(intentionally or not). Senior leadership knows them. When you, their manager, makes a case for their raise or promotion, it smoothly goes through.

Then, there are those in your team who silently excel at work. They are fantastic; everyone on the team knows. They do not self-promote—people external to the team, and senior leadership have no clue about these quiet folks. When you, their manager, makes a case for their raise or promotion, it is not smooth—data is asked, and questions are raised; you need to put in the effort and persevere to make them get what they deserve.

As a manager, you must make sure that your team’s silent but deserving members get their due as much as those who excel at building visibility in the organization.

Promoting the right leaders

If you are an extrovert and know your way around people, everyone assumes that you are a natural leader. There are going to be people in your team who fit this bill. They usually self-select themselves as leaders in the team.

Then, there are those introverts who would make great leaders but do not self-select. As a manager, you must make sure that you promote the right people in your team for leadership roles—not only those who self-select.

Be the bad cop

If you want to make everyone happy, don’t be a leader; sell ice cream.

Steve Jobs

When you are a manager for long enough, there will be the occasional bad apple in your team—people who exhibit behavior that bring down the team’s efficiency. No one in the team will confront these people and ask them to fix their behavior—everyone avoids confrontation. As a manager, you must fish out the bad apples and ask them to do the right thing—be the team’s bad cop.

Image by Clker-Free-Vector-Images from Pixabay

Skeptical: not easily convinced; having doubts or reservations.

You cannot go through life cocksure of your ideas all the time. You cannot go through life skeptical of your ideas all the time.

How does one resolve this dichotomy?

Being skeptical of your ideas makes you open to change. It forces you to re-evaluate your beliefs. It pushes you to seek disconfirming evidence. It enables you to update your views. In short, it makes you a better decision-maker.

At the same time, you cannot wade through life doubting yourself. You have to execute with conviction. Your execution will suffer if you do not believe in yourself and keep wavering.

How do you execute with conviction while at the same time being skeptical?

I am going to answer this question indirectly by discussing a couple of ideas.

Strong opinions weakly held—is a concept introduced by Paul Saffo. The principle’s gist is to allow your intuition to guide you to a conclusion and then look for information that doesn’t fit or indicators that point in an entirely different direction.

A simple way to think about this is—execute with conviction but keep your mind open to disconfirming evidence and when you find them, update your beliefs; easier said than done.

A great approach to build this kind of mindset is to understand deeply all sides of an idea—not just the side with which you agree.

“I never allow myself to hold an opinion on anything that I don’t know the other side’s argument better than they do.”

Charlie Munger

When you believe in something, digesting evidence not supporting your position becomes tough, even when the evidence appears right in front of you. Acclimatizing yourself with all sides(opposite to the one you hold to be true) of an idea helps you recognize and digest disconfirming evidence when it surfaces.

Another way to look at this is not to tie your identity to your beliefs. When you mesh your identity with your beliefs, it becomes tough to be malleable and update your views. The only thing you should be married to is your improvement—if you believe that an idea you held previously is no longer valid, throw it away and update your view.

Even though society does not look favorably on people who change their minds and calls them turncoat, I believe it is a superpower(of course, it does not mean you have to be flippant).

If someone is doing something that is not in line with your beliefs, we look at them with disdain and ignore them. Instead, get curious and keep checking their progress. It is an excellent opportunity for you to learn at someone else’s cost and update your mental model.

You are not going to be right all the time; you are going to be wrong. As long as you are right most of the time(and the cost of being wrong is low), you are on the right path.

Image by ElisaRiva from Pixabay.

When you mention the word process, it invokes dread in software engineers. I, too, used to be in that group. But, years of experience has turned me over to the other camp—the one that appreciates the right process.

Software engineering is a team effort. For a group of people to succeed, the essential requirement is that they must trust each other; they should not be second-guessing one another. Also, they should have an innate, almost instinctual understanding of each other’s behavioral patterns.

Processes create the set up for the above.

Have you seen an F1 pit crew change tires during a race?

https://youtu.be/aHSUp7msCIE

Do you believe they would be able to achieve this rhythmic precision if:

1. One of the crew arrived late to the pit?
2. The driver stopped the car a couple of meters ahead or behind?
3. One of the crew forgot to keep the tires ready?

The pit crew can achieve this feat because:

• They blindly trust each other’s actions.
• Each one knows what is precisely expected out of them.

Process is the way to set behavioral expectations and make sure everyone adheres to it; this develops mutual trust and reliability in a team.

Where people go wrong with process is in introducing process for the sake of process; the classic case of missing the forest for the trees.

When introducing a process, you should have clear and tactical answers to the below questions:

1. Why is the process needed—the end goal of the process.
2. If we did not have the process, what would we miss?

A lot many processes in companies are vestigials of the past. No one knows why they exist. But at the same time, no one wants to put in the effort to eliminate them.

Removing processes that do not add value or are well past their prime is as important as introducing new processes.

Another area where companies go wrong with processes is introducing them because someone else is doing it. Context is critical for a process to add value. If you do not have the environment and the right set up to make a process work, it is a nuisance; it is better not to have them.

Processes are tricky because one needs excellent judgment to evaluate them, and excellent judgment is a rare commodity.

Nowadays, most applications(even rudimentary ones) are distributed applications—they invoke external APIs and services. Hence, understanding and applying the CAP theorem to application design is crucial for building robust applications.

CAP theorem says that you have to choose either consistency or availability in the face of network failures; you cannot have both.

CAP theorem is the programmer’s version of you cannot have your cake and eat it too.

Let us take a hypothetical application that exposes an API to register a user. For storing the user data, the application uses two datastores—MongoDB and MySQL. The datastores and the application backend are on different servers. When the client invokes the user registration API, the backend makes a network call and stores the data in MySQL and MongoDB.

For the sake of this post, I will define application availability as the backend returning a success response. I will define application consistency as the user data being available in MySQL and MongoDB on a success API response from the backend.

The above definitions are not strictly in line with the definitions of availability and consistency, as defined by the CAP theorem. I am taking liberties to convey the idea.

Designing the application using the CAP theorem

The backend and the datastores are on different servers. Since network failures are a given, the datastores might become temporarily unavailable to the backend. How should the application behave during network glitches?

Let us work with a specific scenario.

The client invokes the API; the payload reaches the backend, and the backend is trying to store the data in MySQL and MongoDB. The backend successfully stores the data in MySQL, but the backend cannot access MongoDB due to a temporary network failure.

You can design your application in two ways to handle this scenario.

Design option one

The backend stores the data in MySQL and returns a success response to the client. A periodic reconciliation job runs that queries MySQL, compares the data with MongoDB, and inserts the missing records into MongoDB.

With the above design, your application favors availability over consistency. When there are network failures, the backend is available(does not throw an error), but the data is inconsistent(present in MySQL but not in MongoDB). With this design, the application is eventually consistent—once the reconciliation job runs, the application becomes consistent.

Design option two

The backend returns an error response to the client when there are network glitches(when backend cannot communicate with the datastores). You put the onus on the client to retry under such circumstances.

With this design, your application favors consistency over availability. The application does not return a success response when it cannot store the data in both MongoDB and MySQL. You do not need reconciliation jobs in this design.

Every application these days is a distributed application. Hence, you need to understand the CAP theorem and use the CAP theorem’s essence while designing applications and thinking about the tradeoffs.

PS—To simplify the explanation and make it accessible, I have washed down the CAP theorem and the application failure modes. I will write a follow-up post, expanding the idea, and make it rigorous.

Diagram created using the amazing https://www.diagrams.net/.

https://twitter.com/therealfitz/status/1301278933497909248

The tweet brought back a quote(paraphrased from memory; I do not know the author).

Every employee does two jobs. The job and the job to prove that they are doing the job.

Arduous performance reviews, stakeholder management, timeline-based project management, long term project planning, justifying project delays, and creating visibility are typically the work that one does to prove that they are working.

In a big organization with multiple layers of management and divisions, it might be tough to avoid these. There is no reason why a small business should follow the same policies and work style. Small companies should be optimizing for output.

A big business CEO has to be a master capital allocator. A small business CEO has to be a master resource allocator. Small businesses are perennially short of resources. An employees’ time is one of the most scarce and valuable resources for a small business. A small business should do whatever it can to maximize employees’ productive time. That means cutting all the things that people do to prove that they are adding value.

A small business’s competitive advantage comes from creating an environment where employees can spend their time creating value instead of posturing.

Photo by Engin Akyurt from Pexels

We have to fill unnecessary forms.
We have to get documents signed by random people.
We have to write foolish things.
We have to take part in pointless activities.
We have to follow senseless social norms.
We have to run behind people, asking them to address issues they should be pro-actively solving.

It is frustrating to go through this. When this happens, you feel the entire world is teaming against you. The sleeping rebel within us wakes up and wants to rage against the unfair world.

Do not let this happen.

If only raging solves problems. Instead, raging aggravates it. You lose agency of the task at hand—your brain switches from problem-solving to complaining mode. And, we will know how that turns out.

We usually get to know when our brain starts flipping. But, we do not try to stop it; we fuel the rage with an internal monologue of how unfair all of this is. The way to prevent this is by taking ownership of the task. You anyways have to do it, why not own it and finish it off quickly? When you take ownership of something, your brain finds innovative ways to hack through the task.

Do not rage against the machine; be the mechanic who fixes the machine.

Image from Wikimedia.

Below is the explosive scene where Sean(the psychologist) explains to Professor Lambeau(maths professor) that Will’s life experiences make him behave the way he does.

Perspective dictates our actions and decisions. We do not act and decide based only on the data at hand. Our life experience plays a significant role in our actions and decisions. Perspective is how we color the information we have with our life experiences.

Two people might have the same information, but their life stories will compel them to make different decisions.

Someone who has made risky choices in the past and succeeded will tend to make more dangerous decisions. Someone who has made risky choices in the past and failed will lean towards conservative options.

We encounter situations in our lives where we cannot comprehend others’ actions. We think—what this person should be doing is obvious; even then, why is this person not doing that?

Whenever we see someone acting differently than what we expect, the questions to ask are:

1. Does this person have some information that I do not?
2. How does this person’s life experience differ from mine?

Even though the above might sound obvious, this kind of structural thinking does not come naturally. Many just seeth at others acting weirdly without ever pondering why.

The north star:

If you want someone to do something, make it easy for them to do it.

When someone wants to start the fitness journey, the standard advice is to start with an exercise regime that is easy to do.

If you do not want someone to do something, make it difficult for them to do it.

If someone wants to stop eating junk food, the standard advice is to clean their house of junk food.

Keep the above two principles in mind when you think of processes.

What a process should not do:

A process should not eliminate autonomy and introduce coupling between individuals or teams.

A process should not kill creativity by making it difficult to do things.

Good processes are those that are autonomous and become a habit. A process becomes a burden when you have to spend energy thinking about it and doing it. If it becomes an integral part of doing things, it no longer feels like a burden.

Example:

You want people to wear seat belts while driving; you create a process to enforce this. Let us go through some of the different ways to do this.

A way:

Everyone has to take approval before they get into their car. They go to someone and say I want to take my car out. That person accompanies them to their vehicle, makes sure they wear seat belt, and then hands over the car keys.

The above is an example of a bad process. It is restrictive and will slow you down. It eliminates autonomy—you are always dependent on someone else to do something. It kills creativity by introducing coupling—you are making it hard to do things.

Another way:

You emphasize the importance of seat belt. You create awareness about the dangers of not wearing seat belt. When someone gets into a car, an automated message prompts them to wear seat belt. If they do not wear seat belt, the message continues to beep. You put stickers on dashboards of cars, reminding people to wear seat belt. You mandate painting seat belts in glowing colors to avoid missing them while entering a vehicle. After doing all these, you have random checks on the road for people who do not wear seat belt.

The above has all the hallmarks of a good process. You are trying to convert wearing a seat belt into a habit.

Yet another way:

Device a mechanism so that the car does not start until one wears a seatbelt.

The above is the best way to institute a process. It is autonomous, easy to follow, habit building, and does not let anyone not do it.

A process becomes a burden when it is not part of the way of doing things. In the seat belt example, you start your car and drive some distance. You suddenly remember the dictum to wear seat belt. You look for a safe place to stop, wear your seat belt, and start again. When you go through this, the process mandating seat belt feels like a burden. Contrast this with starting your vehicle by wearing seat belt. If wearing the seat belt while starting a vehicle becomes a habit, it does not feel like a burden anymore. The act of wearing seat belt fades to back.

Best processes are the ones that fade to back.

Photo by mohammad alizade on Unsplash

This behavior(avoiding face to face confrontation) of ours might have something to do with evolution. In the yesteryears, co-operation was essential for survival.

If you are stupid or your idea is ridiculous, no one is going to tell it to your face; they might give you hints.

Giving hard feedback takes effort. We feel these things; it is tough to verbalize. It is tougher to convey it in a way that does not offend the other person. There is nothing to gain for the person giving the feedback by putting in the work. It is an asymmetric situation.

When you ask others for feedback, most of the time, the other person knows what you want to hear. We hardly have a neutral position on anything. When we bounce ideas with others, we are after confirmation, asking them to endorse the thought rather than poke holes in it.

This is also why we find it tough to say no or why you hardly hear no from others. I am not talking of abject strangers but someone familiar. Saying no is awkward. When we say no, we believe we have to justify it. Nos are feelings; we find it hard to verbalize them. When we say no, it is a confrontational situation that we want to avoid. Hence we sidestep the situation by staying silent or saying yes and not doing it later.

When you meet someone who gives you hard feedback or says no to your face, keep that person close to you and thank your stars.

Photo by Kristina Flour on Unsplash

I know it when I see it—said a judge on pornography. Quality code is the same—you know it when you see it. It is challenging to define what quality code is. It is tough to come up with quantifiable metrics on quality code. But, the instant you see quality code, you know it.

Another reason for the difficulty in verbalizing quality code is that it lies in the beholder’s eyes. One person’s high quality is another person’s low quality.

Whenever you talk of quality code, a question pops up—how important is code quality to the success of a business(startup)?

I have wrestled with this question for ages. After spending years meditating on this question under a Bodhi tree in the Silicon Valley of India—Bangalore, I have arrived at an enlightened(also flippant) answer—you cannot quantify this, it matters for sure.

Whenever you talk of quality code and business success, someone usually points out at a business that succeeded despite horrible code.

Businesses are messy. Code is only a part of the story. There are other things that matter to a business’s success. It would be specious to claim a business succeeded despite bad code.

Many businesses that succeed with lousy code are in markets so good and have their timing right that they would have hit the home run anyhow. With quality code under their belt, the journey to the podium would have been pleasant.

A parallel I can think of is the importance of good health and habits. Conventional wisdom says that healthy habits keep you disease-free and lead to a long life. I can always point to a person who smoked and drank her way to a ripe old age. Conventional wisdom says that good habits lead to success. I can always point to a successful person with terrible habits.

Does it mean that good health and habits are immaterial?

Another problem with code quality is that you see its benefits gradually. It is a compounding effect.

The human brain finds it difficult to grasp compound interest. Albert Einstein said—compound interest is the 8th wonder of the world. He who understands it earns it; he who doesn’t pays it.

Compounding is tail heavy. During the initial days, it does not make your eyes pop. As time goes on, compounding gains momentum, reaching a crescendo at the end—the same with quality code.

Good code compounds positively. Bad code compounds negatively. Bad code gradually drags your business down, making it slow and sluggish.

Photo by Volodymyr Hryshchenko on Unsplash

1. How does information flow?
2. How do team members communicate with each other?
3. When do they communicate?
4. What is the medium they use for communication?

A decoupled(push-based or broadcast) structure for communication works the best.

Guiding principles for a decoupled communication structure:

1. People should not be polling each other for information.
2. There should be a specific place for information lookup.
3. There should be pre-defined contracts for the above.

Let us go through an example. Imagine an organization with a development(dev) team and a quality assurance(qa) team. Dev team deploys a build for testing. After the deployment, the qa team starts testing.One way for the qa team to know the deployment status is to poll the dev team periodically and ask whether they have deployed the build.Another way is to create a contract for the dev team to send a Slack message in a channel once they deploy the build. The latter broadcasting style of information dissemination is decoupled. No one has to poll each other for information. As long as the dev team adheres to the contract, and the QA team knows the place to look for this information, it works.

A simple test to figure out your organization’s communication structure:

If a person asks you for information, and you redirect them to a person instead of telling them the steps to find the information, your organization practices the polling style of communication.

Polling Based:

Hey, how can I get this report?You can ask Shyam to generate it for you.

Broadcast based:

Hey, how can I get this report?Add yourself to this email group, and you will receive it regularly.Polling based communication has the following downsides:

1. It is anxiety driving for the person who is seeking the information.
2. It irritates the person who is supposed to give the information.
3. It does not scale as the team grows.
4. All the above lead to unnecessary confusion and aggravation.

Push based communication leads to automation. In the dev qa example, the dev team can automate the publishing of the Slack message.For the push model to work, everyone needs to adhere to the established contract. If one does not do that, the system collapses. Communication forms the cornerstone of organizational culture. Internal communication can make or break organizations.

Photo by Franck V. on Unsplash

When not to abstract?

1. The abstraction does not add value.
2. The underlying thing that you are abstracting is not a standard and is rapidly evolving.
3. You are prototyping.
4. You are short on time.

Before writing an abstraction, always introspect—what is it I am solving with this abstraction? What is the value add of this abstraction? Abstractions work well when they are written over components adhering to well defined, comprehensive standards and specifications. For example, SQL is a well-defined standard. SQL might evolve, but slowly. Hence ORMs are ubiquitous and work well for a majority of the use cases. When you try to abstract a non-standard rapidly growing platform, it becomes a game of catch up and drastic design changes each time the underlying platform changes. Without knowing the direction of evolution of the underlying platform, it becomes a disruptive change each time to the users of your abstraction.When you are prototyping, concentrate on proving the idea as quickly as possible. Do not waste time writing layers of abstraction, which you might throw away in the future if the concept does not work.Abstractions that add value require deep focus and ample time to design. If you are short on time, you will do a lousy job with the abstraction. It will cause problems than solving something.In all the above cases, instead of structural abstractions, concentrate on utility functions. Writing utility functions for simplifying recurring patterns will give you more bang for the buck.

Photo by Lucas Benjamin on Unsplash

What has led us to this?

We were in a homestay in a green estate that had a stream running through the plantation. The owner of the property told us how the government allows him to build and operate a compact hydroelectric plant on the stream, but he does not want to. He said that as soon as he would construct the plant, environmentalists would raise a hue and cry, create a ruckus, and he would end up wasting all his energy on that front, leaving him little energy to do other things.The above is anecdotal, but you see a parallel whenever our government announces big-ticket, ambitious projects. There is always a cacophony of protests. We live in an age where everyone has a strong opinion on everything, and if that person is creative with words, she can multiply her reach, thanks to technology. Nowadays, spreading FUD is just a click away. Plus-oneing the protest satisfies the modern age’s thirst for wokeness and fuels the FUD. In such a situation, a government that lives by the Damocles sword of public opinion gets into perception management mode leaving little energy to solving problems.Organizations are not immune to FUD. US intelligence, as back as in 1944, published a manual with tactics to sabotage workplace productivity. The manual was a field guide to be used against Axis powers in the world war.Some gems from this manual:

• Never permit shortcuts to expedite decisions.
• Talk frequently at great lengths.
• Refer all matters to committees.
• Bring up irrelevant issues.
• Haggle over precise wordings.
• Advocate caution.

The above is how one creates an environment of FUD. One can see such behaviors to varying degrees at workplaces. In a culture where this is extreme, fighting FUD and preception management replaces work. Once this culture takes hold, it is impossible to weed it out—productivity nosedives to zero.

Photo by Snapwire from Pexels.

If you keep the above three in mind, your application can scale a long way with your users. Of course, I am not implying these are the only three things to keep in mind while designing scalable applications.

Statelessness:

If an application does not store persistent state locally, one can scale it by adding servers.

Let us take the example of an application that requires users to sign in. Once a user signs in, the application has to remember that this particular user has logged in. You have the option of storing the logged-in state of the users in the application servers’ memory. When a subsequent request comes, the application looks up in its memory and acts accordingly.

If you are following the above scheme, you are storing the persistent state locally—in servers’ memory. The upside of this approach is its simplicity. The downside is that you cannot elastically scale the application by dynamically adding and removing application servers based on the load.

To figure out whether your application is stateless or not, ask the question: If the next request landed on a different instance of the server, will my operation fail? If the answer is yes, the application is not stateless.

Idempotency:

An operation is said to be idempotent if it produces the same result when executed multiple times.

Example:

a, b = 1, 2

a + b is idempotent—irrespective of how many times you execute this, the result is always 3.

a++ is not—each time you execute this, you get a different result.

If your application is idempotent, you can retry failed requests.

Applications can fail momentarily, especially under load. When this happens, ideally, you should retry the failed request. But you can do this only if the application is idempotent. With idempotency, you do not have the unintended side effect of retrying a request.

You are trying to create a user. You hit the user creation API. For some reason, you do not get a response; this could be due to anything—a temporary network glitch, an application error, or something else. The bottom line is that you are not sure whether the user is created or not. If the application is not idempotent, you cannot retry the request. One might end up creating multiple users with the same identity. Not so, if the application is idempotent. One can retry with abandon.

Coding to interfaces:

Coding to interfaces lets us swap components.

You are using a cache in your application. Instead of using the cache provider’s API directly, you hide it behind an interface of your own. In the future, when you have a deluge of users, if you find the cache lacking, you can swap it with a performant cache without incurring tons of maintainability. You can do this only if you decouple your application from the specific cache provider’s API and abstract it out.

Conclusion:

It is tough to foresee scalability problems. Following the above generic principles will help you to develop adaptable applications that you can cheaply scale while buying time to create sophisticated scaling strategies specific to your needs.

One thing the Coronavirus crisis has made vivid is that no one knows anything for sure. The fact that no one knows anything bubbles up every time there is a crisis. This time though, due to the severity of the mess, it is stark; in your face.

Experts used to say that eating fat is unhealthy. Now, not so much. Not long back, scientists used to believe that the adult human brain is static—once we enter adulthood, our intelligence stops improving. Today, everyone talks about brain plasticity—how the brain keeps growing with the right input even in adulthood and adapts well into old age. The scientific community is staring at a replication crisis—researchers are not able to consistently reproduce experimental results. Marshmallow experiment—one of the most cited psychology experiments, is under d oubt.

When I started putting my thoughts in public, I was hesitant. I always had a voice in the back of my mind asking: How do you know you are right? I face the same when someone comes to me for advice. I am guarded with my advice.

How do I reconcile with this?

I have benefitted immensely from the thoughts of others. I am thankful to all these people who take the pain to put their ideas in front of everyone, especially in the current environment where trolling is given. Today, it is fashionable to call anyone and everyone a virtue signaller. Thankfully, I have not gone through the trolling experience as I have a tiny audience.

What is wisdom?

Wisdom is knowledge synthesized with life experiences. By studying, observing, and paying attention, one gains knowledge. One accumulates life experiences by doing. When you mesh the two together and contemplate, you gain wisdom. If no one broadcasted their thoughts, the world would be a sad place.

Strong opinions, Weakly held

An excellent framework for better thinking is: Strong opinions, weakly held. The idea is not to be married to your views. In the face of disconfirming evidence, update your beliefs. Interestingly, even this maxim is under debate.

There is no way for you to be a hundred percent sure of anything; this applies when you give and receive advice. The best you can do is color your knowledge with your life experiences and share it with others in the hope that the other person takes something positive out of it—a small way for you to give back to the society.

Always be skeptical.

The above question plagues the minds of software developers.

Popular programming culture made microservices the de facto way to build software. Now, many are second-guessing their choice.

https://twitter.com/kelseyhightower/status/940259898331238402

Here is a post from Segment on why they consolidated their microservices into a monolith.

https://twitter.com/GergelyOrosz/status/1247132806041546754

Microservices is archetypical of software trends following the Hype cycle. In the Hype cycle, microservices has passed the “Peak of Inflated Expectations.”

A framework for making new technology choices

Before adopting any new technology, you have to:

1. Clearly define the problem you are trying to solve with the novel technology.
2. Understand how the new technology solves the problem.
3. Build perspective by studying the evolution of the technology.
4. List the supporting structures needed to make the new technology work.

[ Click to Tweet (can edit before sending): https://ctt.ac/zcw9I ]

The above may sound meh, but taking the pain to define them to the T is the key to the success of new technology adoption.

Clearly define the problem you are trying to solve

Nailing down the problem is the first step. You would be surprised by the number of people who try to solve a problem without defining it formally.

When the problem that you are trying to solve is vague, it becomes tough to find a solution to it. How many times has it happened to you that you describe a problem to someone, and in the process of doing so, you get closer to the solution?

Clearly define the problem that you are trying to solve with microservices. Is it a performance problem with the application? Are you trying to increase the productivity of the team with microservices?

When you do this, sometimes you find non-disruptive ways to solve the problem. Better communication between teams might be the solution, not microservices.

Understand how the new technology solves the problem

Understanding how the new technology solves the problem will help you to evaluate it objectively. Defining the problem, as stated in the first step of the framework, is essential for this.

There are two broad reasons for microservices adoption—technical and logistical.

Technical

The application has grown in complexity and has workloads vying for different types of resources. You are not doing justice to any of these workloads by packing them in a monolith. For example, some workloads might be CPU intensive, some IO heavy, and the others hungry for memory. By extracting each of these workloads into a microservice, you have the freedom to host them in different servers conducive to their demands.

The application has grown in complexity and has workloads better solved in different programming languages. Breaking the monolith into microservices gives you the ability to code them in the programming language of your choice.

Logistical

The application has evolved as well as your company. Different teams are responsible for different areas of the application. You want these teams to be independent. If you break the monolith into microservices that mimic the team structure, you will achieve this independence. These teams can work independently without stepping on each other’s toes, thus being more productive.

Build perspective by studying the evolution of the technology

When you try to dig up the history, keep in mind that you are not going after the rigorous academic definition of the term, but the cultural context of its evolution. The common definition of a term may not match with its formal description. For example, when people say microservices, they are usually referring to Services Oriented Architecture(SOA) and not microservices in particular.

Microservices exploded due to big companies like Amazon and Netflix evangelizing(maybe unintentionally) them. These companies have thousands of employees and divisions. Once you understand this and build a perspective, you will naturally ask, is this applicable to me? If you are a small startup that can count your tech team with one hand, in all probability, the answer is no. It is tough to build this perspective without studying the evolution of the technology.

Supporting structures needed to make the new technology work

Whenever you introduce a new technology, you might have to make some changes to the way you work. Some of these changes might be inconsequential, and others extensive.

For microservices to be successful, you will have to invest in tooling. You will have to have a robust monitoring system because, with microservices, you are treading into distributed computing where failure is a given. I will stop here as this requires a post in itself.

In many circumstances, these changes might be far-reaching negating the benefits of the new technology. Be keenly aware of this trade-off.

Summary

Doing this might sound time-consuming, but it pays off by preventing unmitigated disasters down the line, once you are in the middle of adopting the new technology. Many new technology choices bomb because someone did not do the above painstakingly enough.

Hype Cycle image By Jeremykemp at English Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=10547051.

Photo by Miguel Á. Padriñán from Pexels

What is the stereotype of a leader?

A charismatic extrovert who can spellbind an audience with her talk.

Leadership is not about how charismatic you are or how good you are at public speaking. Popular culture has narrowly defined leadership to be so.

In the book, The Little Book of Talent, the author Daniel Coyle writes: Most great teachers/coaches/mentors do not give long-winded speeches. They do not give sermons or long lectures. Instead, they give short, unmistakably clear directions; they guide you to a target.

What is the stereotype of a developer?

This twitter thread does an excellent job of it.

Being a good developer is not about which editor you use or how socially awkward you are. These are urban legends devoid of any real substance.

Leaders and developers come in all shapes and sizes. Take a step back and think of all the great people you have worked with. Do they stick to the stereotypes associated with their role? Can you pigeonhole them into a mold?

The movie, Money Ball, is the best illustration of this line of thinking. The plot of the film revolves around the real-life story of a manager who assembles a successful baseball team analytically by ignoring the mythical stereotypes associated with what makes one a successful baseball player. This approach of building the team was not a cakewalk for him; he met with resistance from all for his radically different line of thinking.

Peter Thiel talks of startup hiring as finding the talent which the market has mispriced(I am paraphrasing this from memory).

If you stick to stereotypes while hiring and promoting, you are:

1. Artificially restricting the available talent pool.
2. Pursuing the same set of people that everyone else is.
3. Going after qualities that you do not need.

We are sympathetic to underdogs, but we do not bet on them. Doing so is the not so secret strategy for building a great team.

In this post, I will go through some of the basics one should keep in mind while calling external services. I will use the Python Requests library to demonstrate this with external HTTP calls. The concepts remain almost the same irrespective of the programming language, library, or the kind of external service. This post is not a Python Requests tutorial.

I have created a Jupyter Notebook so that you can read and run the code interactively. Click here, then click on the file WildWildWorldOfExternalCalls.ipynb to launch the Jupyter Notebook. If you are not familiar with executing code in a Jupyter Notebook, read about it here. You can find the Notebook source here.

Let us call api.github.com using Requests.

https://gist.github.com/abhirama/5d4102dd8a158515565ac09d05b8decb

External calls happen in two stages. First, the library asks for a socket connection from the server and waits for the server to respond. Then, it asks for the payload and waits for the server to respond. In both of these interactions, the server might choose not to respond. If you do not handle this scenario, you will be stuck indefinitely, waiting on the external service.

Timeouts to the rescue. Most libraries have a default timeout, but it may not be what you want

https://gist.github.com/abhirama/83ab61a2824d1c38d670a011e04e337e

The first element in the timeout tuple is the time we are willing to wait to establish a socket connection with the server. The second is the time we are willing to wait for the server to respond once we make a request.

Let us see the socket timeout in action by connecting to github.com on a random port. Since the port is not open(hopefully), github.com will not accept the connection resulting in a socket timeout.

https://gist.github.com/abhirama/09b2b7ad09cbe14ad5b58202da0268af

The output.

Time spent waiting for socket connection - 3.42826354 Seconds Time spent waiting for socket connection - 6.4075264999999995 Seconds

As you can see from the output, Requests waited till the configured socket timeout to establish a connection and then errored out.

Let us move onto the read timeout.

We will use httpbin service, which lets us configure read timeouts.

https://gist.github.com/abhirama/fee259e76d875e528aee32755ef76f1e

The output.

Timed out after 6.941002429 Seconds

In the above, we are asking httpbin to delay the response by 9 seconds. Our read timeout is 6 seconds. As you can see from the output, Requests timed out after 6 seconds, the configured read timeout.

Let us change the read timeout to 11 seconds. We no longer get a ReadTimeout exception.

https://gist.github.com/abhirama/2e605b4f22bcd07a23d1d03a6eda0753

A common misconception about the read timeout is that it is the maximum time the code spends in receiving/processing the response. That is not the case. Read timeout is the time between the client sending the request and waiting for the first byte of the response from the external service. After that, if the server keeps on responding for hours, our code will be stuck reading the response.

Let me illustrate this.

https://gist.github.com/abhirama/8297a9079057c3106698f9f68e9b398c

The output.

Time spent waiting for the response - 28.210101459 Seconds

We are asking httpbin to send data for 30 seconds by passing the duration parameter. Requests read timeout is 15 seconds. As evident from the output, the code spends much more than 15 seconds on the response.

If you want to bound the processing time to 15 seconds, you will have to use a thread/process and stop the execution after 15 seconds.

https://gist.github.com/abhirama/bc5ea44164b4ca6f3f77edd5ff5d0640

The output.

Time spent waiting for the response - 20.012269603 Seconds

Even though we receive the HTTP response for 30 seconds, our code terminates after 20 seconds.

In many real-world scenarios, we might be calling an external service multiple times in a short duration. In such a situation, it does not make sense for us to open the socket connection each time. We should be opening the socket connection once and then re-using it subsequently.

https://gist.github.com/abhirama/28103fd63717b429758db820afe6c1fd

The output.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496

As you can see from the output, Requests started a new connection each time; this is inefficient and non-performant.

We can prevent this by using HTTP Keep-Alive as below. Using Requests Session enables this.

https://gist.github.com/abhirama/9f466cb8bf71cbb86707b044478facf8

The output.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496

Now, Requests established the socket connection only once and re-used it subsequently.

In a real-world scenario, where multiple threads call external services simultaneously, one should use a pool.

https://gist.github.com/abhirama/d7a69964183a166b6c58e50fe1e0bc7d

The output.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): api.github.com:443 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496 DEBUG:urllib3.connectionpool:https://api.github.com:443 “GET / HTTP/1.1” 200 496

As we have created a pool of size two, Requests created only two connections and re-used them, even though we made four external calls.

Pools also help you to play nice with external services as external services have an upper limit to the number of connections a client can open. If you breach this threshold, external services start refusing connections.

When calling an external service, you might get an error. Sometimes, these errors might be transient. Hence, it makes sense to re-try. The re-tries should happen with an exponential back-off.

Exponential back-off is a technique in which clients re-try failed requests with increasing delays between the re-tries. Exponential back-off ensures that the external services do not get overwhelmed, another instance of playing nice with external services.

https://gist.github.com/abhirama/aeb85a6660745f5b843315e2e5c588a3

The output.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org:443 DEBUG:urllib3.connectionpool:https://httpbin.org:443 “GET /status/500 HTTP/1.1” 500 0 DEBUG:urllib3.util.retry:Incremented Retry for (url=’/status/500’): Retry(total=2, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Retry: /status/500 DEBUG:urllib3.connectionpool:https://httpbin.org:443 “GET /status/500 HTTP/1.1” 500 0 DEBUG:urllib3.util.retry:Incremented Retry for (url=’/status/500’): Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Retry: /status/500 DEBUG:urllib3.connectionpool:https://httpbin.org:443 “GET /status/500 HTTP/1.1” 500 0 DEBUG:urllib3.util.retry:Incremented Retry for (url=’/status/500’): Retry(total=0, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Retry: /status/500 DEBUG:urllib3.connectionpool:https://httpbin.org:443 “GET /status/500 HTTP/1.1” 500 0

In the above, we are asking httpbin to respond with an HTTP 500 status code. We configured Requests to re-try thrice, and from the output, we can see that Requests did just that.

Client libraries do a fantastic job of abstracting all the flakiness from external calls and lull us into a false sense of security. But, all abstractions leak at one time or the other. These defenses will help you to tide over these leaks.

No post on external services can be complete without talking about the Circuit Breaker design pattern. Circuit Breaker design pattern helps one to build a mental model of many of the things we talked about and gives a common vocabulary to discuss them. All programming languages have libraries to implement Circuit Breakers. I believe Netflix popularised the term Circuit Breaker with its library Hystrix.

Image by RENE RAUSCHENBERGER from Pixabay

It is not easy to quantify the second-order effects of centralization, sometimes impossible.

The CEO, due to the broad nature of her work, sees the duplication everywhere. She also witnesses the expenses as a result of this; it is in her face. She wants to eliminate this and bring efficiency and cost-saving to the organization. Hence, she champions centralization. The rank and file are hands-on; they have to deal with the management policies to do their work. They experience the second-order effects of centralization day in and out. They instinctually develop anti-centralization spidey sense. Unline the rank and file; the CEO does not have the ringside view of the second-order side effects of centralization. The rank and file do not see the duplications the CEO sees because they do not have the same expansive look like that of the CEO.Centralization efforts have a quantifiable impact. If not entirely measurable, you can do some mental gymnastics to get an idea. The downsides of centralization are unquantifiable. The unquantifiable plays a crucial role in success, sometimes much more than the quantifiable. Morgan Housel calls this the McNamara Fallacy. McNamara Fallacy: A belief that rational decisions can be made with quantitative measures alone, when in fact the things you can’t measure are often the most consequential. Named after Defense Secretary McNamara, who tried to quantify every aspect of the Vietnam War. Let us flip the earlier scenario. Imagine that the centralized procurement team does bring in efficiency and reduce cost, albeit at a minor loss of productivity. The software procurement expense as a whole is never on the mind of the rank and file; the rank and file do not look at it as closely as the CEO; it is not always on their face. Hence, the rank and file still view centralization as a bane, even when it brings in advantages.The consensus is that a decentralized way of working trumps a centralized approach; this applies to the military too. Jocko Willink, a prolific US Navy Seal, champions decentralized command. There are valid cases for centralization, especially when the talent required to do something is in short supply, and there are legitimate gains to be had from economies of scale. But, when you centralize, think hard of the unquantifiable second-order effects of the decision.

Larry Wall, the creator of Perl, says - Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris.

What no one tells you is that this laziness does not come for free; one has to work hard to imbibe this trait.

In practical terms, what does being lazy translate to?

1. Doing as little as possible, never more than needed.
2. Instead of doing things yourself, delegating to well-established tools, libraries, and frameworks.

Let us work with some concrete examples.

You want to parse a CSV file.

You think: let me load the file, parse it line by line, and split each line on a comma. You roll up your sleeves and code this. You feel smug having solved the problem yourself without anyone’s help.

Trouble starts when the CSV you parse has a header. Now you add an if condition to detect the first line. Later, someone uploads a CSV separated by a tab instead of a comma. You add another if condition to accommodate this. Another person uploads a CSV which has quoted fields. You start doubting yourself and ask how many such “unknown unknows” are there when it comes to parsing a CSV?

Unknown unknowns are risks that come from situations that are so unexpected that they would not be considered.

CSV parsing might have a lot of “unknown unknowns” for you - a person who is not well versed with the intricacies of CSV format. But there are experts out there who know the CSV format and have written libraries to handle all the edge cases and surprises that it might throw. You hedge your “unknown unknown” risk by delegating the CSV parsing to one of these libraries.

In short, be lazy, do as little as possible, and delegate to well-established libraries.

“Fools say that they learn by experience. I prefer to profit by others experience.”

― Otto von Bismarck

Let us consider another scenario.

You want to store a counter in a database. One approach is: when you want to increment the count, you get the current count from the database, add one to it and store the new count back in the database.

Do you see the problem with this approach?

What if many threads are doing this in parallel? You will end up with a wrong count. A better approach is to delegate the task of incrementing the count to the database by leveraging SQL’s arithmetic operators. This approach makes the counter increment atomic. Many threads trying to increment the count is no longer a concern.

By doing less yourself and delegating the task of incrementing the counter to the database, you have saved yourself from bugs.

Why is this hard work?

This sort of thinking does not come easy; you have to work hard to identify where what and to whom you can delegate the work.

Dunning-Kruger effect might have a role to play in this. We believe we are the experts and best suited to do things.

In the field of psychology, the Dunning–Kruger effect is a cognitive bias in which people assess their cognitive ability as greater than it is. It is related to the cognitive bias of illusory superiority and comes from the inability of people to recognize their lack of ability.

While coding, most of the time, you are solving a problem that someone else has already solved, probably in a different context. Be aware of your biases and always question: Is this something I have to code myself, or can I offload this to an already written, well established and well-tested library or framework?

“Learn from the mistakes of others. You can’t live long enough to make them all yourself.”

― Eleanor Roosevelt

Image by Clker-Free-Vector-Images from Pixabay

All of us have pondered over this question. Luminaries have devoted their lives in the pursuit of an answer to this question. Philosophers have written voluminous texts trying to answer this question.

I am no Yogi, but that does not disqualify me from trying to answer this profound question. Beware, my answer might leave you with a feeling of meh.

During a holiday, a group of us friends played a weird game of football. We were randomly dribbling the ball, passing, and tackling each other - no teams, rules, goals, and referees. This pointless pursuit of the ball was fun.

What is the difference between kids and adults?

Kids involve themselves in pointless pursuits. They are always engaged in one activity or the other. These consume them. We, the self-critical adults, try to see a point in everything. Few things consume us.

Give a cardboard box to a kid. She can keep herself occupied with the box for hours—an adult dreads at the thought of this.

When a child is young, she loves to draw irrespective of whether she is good at drawing or not. As she grows older, she pursues drawing only if she finds herself good at it. Enter adulthood, she becomes self-critical and continues her hobby only if she sees a point in it.

As an adult, try to remember the last time you were engaged in and consumed by a pointless activity.

A child actively indulges in role-play, creating stories in her head and acting it out. An adult passively watches role play in tv-series and movies. A child plays a variety of games. An adult passively enjoys sports watching others play.

As we age, we move from an active to a passive life. We try to seek a point in everything.

A child has no time to search for meaning. She is busy indulging herself in everything. The activity is the end; it is not a means to an end. I believe the same goes for life.

The point of life is not to search for meaning but to indulge in it. It is a pointless existence, and there is a joy to be had in understanding this. It is liberating.

Photo by Emily Morter on Unsplash

When this happens, it hurts.

To drive a car, you need not know how it works internally. The mechanics of an automobile is well abstracted from the driver. Similarly, software libraries, tools, and frameworks promise abstraction to the engineers using them. They promise that one can use them effectively without delving into the internals.

Let me tell you a story of uWSGI, processes, and threads.

uWSGI is a container for running web applications. It is popular in the Python world. Global interpreter lock makes concurrency muddy in Python applications. A way around this is to spawn multiple processes. While starting uWSGI, one can configure the number of processes to spawn to service concurrent web requests. The master uWSGI process starts, initializes, and loads the Python application code, and then forks the configured number of child processes. One caveat while forking is that the child process does not inherit any of the threads created in the parent process. Since uWSGI loads and initializes the Python application and then forks, the child process will not inherit any threads created during initialization in the parent process.

This bit us hard.

We were initializing a Kafka library on application load. This library internally created background threads that aggregate and push messages to a Kafka broker. The child processes uWSGI forked did not have these threads. Hence, our payloads were not flowing to Kafka.

In 99% of the cases, an application developer need not bother about how uWSGI forks and creates child processes. uWSGI abstracts this well, and the application developer can go on with her regular day to day work willfully ignorant of these abstractions. The same goes for the Kafka library. One need not pry open the library to figure out how it aggregates the messages and sends them to Kafka. In this particular case, the abstractions leaked and bit us. We spent a couple of days debugging.

One of the principles of good software design is that it should be easy to reason about.

You can interpret the above in different ways. A short interpretation is that when you look at a piece of code, it should be easy to figure it out in one’s head. Whatever you need to reason about the code should be in it and apparent - no hidden surprises. Leaky abstractions break this principle.

The uWSGI incident is fresh in my mind, but I have seen this to be a recurring pattern.

https://twitter.com/abhyrama/status/1207334951605063681

This is the problem with abstractions and magic technologies. When the abstractions leak, they bite us in ways we do not expect, and at times we do not anticipate.

1. Be skeptical of technologies that claim to be magic. Do not pay only lip service to KISS, follow it in spirit. Think twice before incorporating such technologies into your stack.
2. Figuring out the anti-patterns beforehand is as crucial as figuring out the best practices. All technologies come with a list of dont’s - grok them.
3. Even though it may sound like an oxymoron, invest time and effort in going beyond the abstraction and peel the layers. At the least, build a minimal mental model.
4. If you are a library developer, documenting how things can go wrong is as important as highlighting the rosy use cases; this is the least you can do to help your fellow craftsmen. Map the minefields, so the one does not accidentally trip on them.

https://twitter.com/abhyrama/status/1186322830410956801

1. uWSGI is a fantastic piece of software. I am not trying to diss on uWSGI at all. 2. For our particular problem, we disabled the default forking behavior of uWSGI. We enabled, lazyload, which loads the Python code after the fork rather than before.

Image by Sang Hyun Cho from Pixabay

“How do we hire amazing engineers fast?” is a question people ask me often.

When someone asks me the question, they usually expect a profound answer, which will cure all their hiring pains. Hiring, especially good people, is a long, involved, and arduous process. There are no deep secrets to this. But, this is not what people want to hear because they already know this. Instead, they expect a magic potion, a hack, which will wipe out all the hiring woes. My standard answer to the hiring question is along the lines of - “I do not have any tricks up my sleeve to help you with that.”

When someone is expecting a profound answer, and you do not have any, it is very tempting to come up with one. When the other person is seeking enlightenment, and all you have is mundaneness, you feel like an amateur and a buzz kill.

Charlatans start like this. People expect magical answers from them; they do not have any. Still, the expectation from others is so high that they start coming up with one, and then it becomes a self-fulfilling prophecy. Also, once you do this multiple times, you start drinking your own kool-aid. You do not even recognize that you are a charlatan. You genuinely start believing that you are a messiah.

Growth is a catch-22 problem. You need to endure pain to grow. You are not ready to experience pain unless you see the growth. But you do not see growth unless you suffer pain. Charlatans, with their quick and simple hacks, give us hope of disproportionate returns by investing little effort, hence the demand for charlatans.

The rich and the famous are often called charlatans. We hold the rich and the famous accountable for lofty morals and weave stories of their impeccable character. We forget that they are just like us - winging through life, taking shortcuts, not knowing what is happening or where they are heading. When the rich and the famous know of these high expectations, which a lot of them do not have(there is nothing wrong with this), they artificially try to mold themselves on these lines. In today’s age of social media, where virtue signaling is just a click away, it is getting easier to do this. When you fake it, it can only go so far. One day, the cloak falls, and the grandness built on flimsy appearances comes tumbling down. We then start calling these celebrities charlatans, but little we introspect on our role in them turning out to be charlatans.

As much as we like to blame charlatans for their deception, a significant part of the blame rests on us for creating them. It is our unholy expectations and quick reward-seeking nature that gives rise to charlatans.

Image by Pexels from Pixabay

• If both betray each other, they each serve two years.

• If one betrays while the other stays silent, the betrayer goes free while the silent one serves three years.

• If both stay silent, they each serve one year on the lesser charge.

Since they cannot communicate, they cannot coordinate an optimal strategy.

Let’s consider prisoner B’s perspective.

• If A betrays, B is better off betraying too—two years in prison instead of three.
• If A stays silent, B is still better off betraying—going free instead of serving one year.

The same logic applies to A.

If both act purely in self-interest, they betray each other and end up with two years in prison each. Ironically, had they cooperated and remained silent, they would’ve only served one year. Optimizing for self-interest leads to a worse outcome for both.

This is the classic Prisoner’s Dilemma, a concept from game theory—the study of decision-making and strategic interaction, particularly around cooperation and incentives.

At its core, the Prisoner’s Dilemma models situations where individuals, acting selfishly, harm both themselves and others, despite a better outcome being possible through cooperation.

Look around, and you’ll see the Prisoner’s Dilemma playing out everywhere.

On the roads: people flout traffic rules to save time, leading to chaos and longer delays for everyone.
In organizations: teams and individuals chase narrow goals, often at the expense of overall company performance.
In public spaces: no one maintains shared resources like parks, buses, or toilets—yet if everyone did, we’d all benefit.

https://twitter.com/abhyrama/status/1158569767621419008

Use the Prisoner’s Dilemma as a lens to understand why people fail to collaborate, even when it would produce better outcomes.

It also helps us think about:

• Cooperation between strangers.

• The incentives required when the benefits of collaboration aren’t obvious.

Show me the incentives, and I will show you the outcome.

- Charlie Munger

In the absence of communication and clear incentives, what might lead A and B to cooperate?

• Suppose the criminal underworld enforces a strict code: never confess to the police. Breaking it means death. In such a world, both would likely stay silent. The Italian mafia has such a code called Omertà.

• Suppose confessing taints your reputation permanently. You’d never find work again. Fear of social exclusion might lead both to remain silent.

• If A and B knew they had to collaborate again in the future, they might stay silent to preserve trust.

• If they belonged to a cult that believed betrayal leads to eternal damnation, they’d likely cooperate out of shared belief.

Social constructs like laws, nationalism, religion, trust, reputation, and community are humanity’s evolved response to the Prisoner’s Dilemma. These mechanisms encourage implicit cooperation and improve collective well-being.

Thinking through this lens sheds light on why:

• Small teams often outperform large ones.

• Scrappy startups can beat large corporations.

• Close-knit communities have lower crime rates.

• Homogeneous small nations can be more cohesive than diverse, sprawling ones.

How do organizations solve the Prisoner’s Dilemma?

They emphasize values, team bonding, recognition, rules, and shared goals. Some even build cult-like cultures to foster loyalty and collaboration.

Implicit collaboration is essential—for teams, companies, societies, and nations. Use the Prisoner’s Dilemma as a mental model to understand and design for cooperation. It helps us build systems that incentivize the greater good.

Photo by Perry Grone on Unsplash.

One can avoid the majority of bugs by adopting two behavioral traits.

1. Taking a step back and asking - What can go wrong?
2. Asserting your assumptions.

https://twitter.com/abhyrama/status/1204776249635160066

Let us work with an example.

The below code accepts a list and returns the first element.

[code language=“python”]

def get_first_elem(lst):     return lst[0]

[/code]

What are the implicit assumptions that you see?

1. No one will pass a null list.
2. No one will pass an empty list.

What can go wrong?

1. If someone passes a null list, the code errors out.
2. If someone passes an empty list, the code errors out.

The idea is not to code defensively but to be aware of the assumptions and error conditions. It might as well be that the function should error out when someone passes a null or empty list; when it happens, it should not be a surprise.

Adopt these behavioral traits whenever you read or write code; you will be miles ahead of the rest.

Some of the other common bug patterns follow.

Never let it leak

Not closing opened resources - be it file descriptors, database connections, HTTP connections, or socket connections. Programming languages have constructs to do this - finally block in Java and Python, defer in Go lang. Whenever you open a resource, close it; never let it leak.

Fence it

When establishing a connection to an external resource, be it a database or a remote server; configure appropriate timeouts. Being stuck for an undefined period establishing a connection is not a happy place to be in; fence the connection establishment time to a reasonable value. Also, timeouts come in various flavors - connection establishment timeout, socket timeout, HTTP server timeouts. Familiarise yourself with all that apply to your scenario.

Bound it

Keep an eye on runaway resource creation. There is a limit to the no of connections that an external system can handle; there is a limit to the no of files one can create on a file system. Be aware of these limits and put in checks and balances to bound the creation to acceptable values.

Reuse over recreate

If you are opening a connection or creating an object repeatedly, check whether it is possible to pool the resources instead of repeatedly re-creating. Create a pool of resources once and then reuse when needed. This principle applies to all sorts of connections - HTTP, socket, database.

Using a resource pool alleviates the boundless resource creation problem too.

Encoding it right

While working with text, take care of character encoding. Things work great until one beautiful day someone passes a foreign text to your code, and everything collapses like a house of cards. Familiarise yourself with character encoding and take care of it while coding.

Stand on the shoulders of giants

If I have seen further, it is by standing on the shoulders of Giants.

- Isaac Newton.

The three chief virtues of a programmer are: Laziness, Impatience and Hubris.

- Larry Wall.

Doing everything yourself and not delegating to established libraries, frameworks, and tools is the root cause of a large number of bugs. In all probability, someone would have faced the problem that you are facing and crafted a well-tested solution to it - shamelessly use it. There is a reason why great programmers claim laziness is a virtue - follow this in spirit and practice.

https://twitter.com/abhyrama/status/1224543603931791360

Grokking these bug patterns and adopting the behavioral traits will make you a coding Yoda, who does not want to be one?

An earlier post on similar lines on Software Security.

Yoda image by Mario Eppinger from Pixabay

Leaky tap photo by Luis Quintero from Pexels

Fence photo by Min An from Pexels

Recycle image by 95C from Pixabay

Encode image by Gerd Altmann from Pixabay

Elephant image by Sasin Tipchai from Pixabay

All organizations have Platform Engineering teams in one form or the other; these are centralized engineering teams providing building blocks for other engineering groups within the company. The customers for these teams are the internal engineers, not the end-users of the product.

For Platform Engineering teams to be effective, do not strongly couple them with the other engineering groups. Even though these are centralized engineering teams, they should operate in a decentralized manner. Platform Engineering teams should follow the mantra of Loosely coupled but strongly aligned with the other engineering teams. To achieve this, view Platform Engineering teams as enablers, not as doers.

https://twitter.com/abhyrama/status/1154761074991742976

The build engineering team creates tools and resources so that any team in the organization can deploy their builds to production, i.e., the Build engineering team enables you to deploy builds using the tools they create; they do not deploy the build for you.

The performance engineering team creates tools and frameworks for you to identify performance bottlenecks, i.e., the performance engineering team enables you to identify performance bottlenecks using the tools they build; they do not identify performance bottlenecks for you.

The security engineering team creates tools and libraries for you to identify security loopholes, i.e., the security engineering team enables you to identify security holes using the tools they build; they do not identify security loopholes for you.

This is how organizations should mold and communicate the role of Platform Engineering teams. Positioning the Platform Engineering teams as doers instead of enablers makes them the bottleneck for other teams to get their work done.

Other teams rely on Platform Engineering teams for their success. You do not want them second-guessing whether the Platform Engineering teams are doing their job or not. They need to trust the Platform Engineering teams with their critical workloads. Without trust, everything breaks, leading to unnecessary back and forth, sapping the energy of all parties involved.

To build this culture of trust, Platform Engineering teams should focus on observability, performance, and stability.

Lack of observability creates anxiety. If the build engineering team does not give a dashboard where one can see the progress of builds, history of builds, and create alerts on build failure - teams would be anxious about their builds. They would bug the Build Engineering team on this, causing stress all around. Observability is a must for creating a culture of zero stress and anxiety.

Documentation is a subset of observability. Once the Platform Engineering team releases a tool or an API, others should be able to use them without intervention from the Platform Engineering team. To achieve this, Platform Engineering teams should focus on clear and concise documentation.

Since all teams use the tools and APIs of Platform Engineering teams, performance improvements have a multiplicative effect.

The build tool not working, brings the engineering org to a standstill. Login not working negatively affects all parts of the application. Hence, a keen focus on stability is a must. These are the foundations on which other engineering teams build their features.

Due to the nature of the work of Platform Engineering teams, consulting, guiding, and proliferation of best practices becomes part and parcel of the day to day responsibilities; this is understated, but Platform Engineering teams spend a significant chunk of their time on this. Look for ways to institutionalize these so that the Platform Engineering teams are engaged in their core work and not spending time on this. As discussed earlier, focussing on observability, performance, and stability goes a long way towards this.

Feature prioritization can become a challenge for Platform Engineering teams as everyone comes to them with feature requests. A simple yardstick to use is - If we do not release this functionality, is there a way for the requesting team to go about their work, albeit in a roundabout manner? If the answer is yes, then it is not a burning problem. If not, you need to figure out a way to get the feature out as soon as possible.

Platform Engineering teams should adopt a broad perspective when developing features. Do not develop features for a particular team. Think of how the feature relates to other teams and design it so that everyone in the organization can leverage the feature.

If you want to build a culture of speed and rapid iteration, viewing Platform Engineering teams as enablers and not as doers is critical.

Image by Susbany from Pixabay

What does this have to do with coding? How do you decide whether a code is good or not? There are many parameters, but a definitive indicator is the number of bugs - lesser the bugs, better the code.

What is the definition of a bug? A bug is a problem with the code, which makes it not work correctly.

Why do bugs occur? The person who authored the code did not anticipate that particular condition, and the code does not know how to handle that situation.

How does one prevent bugs? Anticipate all that can go wrong and take care of them while coding.

To do this, one needs to take a bleak look at things - exhaustively think of all that can go wrong and take care of these. To put it shortly, you need to wear a pessimist’s cap.

Then, do pessimists make better coders?

There is another way to look at this.

To create something, you need to be an optimist; coding is about creating something new. To write code devoid of bugs, you need to take care of edge cases and boundary conditions and account for them.

Is a better coder someone who can balance optimism and pessimism?

Photo by Christophe Hautier on Unsplash

Companies nowadays do not have a separate quality assurance(QA) team. People coming from the old world find this difficult to digest. In the last decade, the way of developing, deploying, and monitoring applications has changed.

Gone are the days of the waterfall model of development. You no longer take months to develop a feature, followed by a long testing cycle and finally hand it over to the production team for deployment. Release cycles have shrunk from months to hours. People who build the application are responsible for deploying, running, and monitoring it.

In this brave new world, base your quality assurance strategy on:

1. What is the cost of a bug in production?
2. How quickly can one detect a bug in production?
3. How quickly can one fix a production bug?
4. If a bug manifests in production, how does one limit the damage?

The cost of bugs is not uniform; bugs in different areas of the application cost differently. The line of business also dictates the price of bugs.

For an aerospace or medical devices company, a bug means the difference between life and death literally, not so much for a social network or an e-commerce website. If an e-commerce website does not load, it significantly hits the company’s bottom line. For a social network, it might make an insignificant dent in advertising revenue. For an e-commerce company, the cost of a bug in the recently purchased items section on the home page is not the same as checkout from the shopping cart not working. While your shopping cart checkout has to be bulletproof, you can be lenient with the recently purchased items section on the home page.

Vary quality control based on the criticality of the functionality. Be tight where it needs to be and loose where you have room to wiggle. You need not have a uniform quality assurance strategy for the entire application.

Testing dents your speed and time to production irrespective of whether you do it manually or automate it; this is the cost you pay for quality - you need to make peace with this. The stricter you get, the costlier it becomes. Automated testing is not free; it too has a cost - more things to maintain and manage. In some cases, you might end up writing twice the code due to automated testing. I am not arguing against automated testing but asking you to - factor in and mentally accept the cost - before you take the plunge.

There is a tendency to equate testing with quality assurance. Testing is a subset of quality assurance; testing is NOT quality assurance. Quality assurance is much more than JUST testing and comprises of a variety of things.

Today, experimentation, incremental development, and speed are the essence. Try a small idea, see whether it sticks, and then rapidly iterate and expand. In such an environment, following a design philosophy that gives your room for error and factors in bugs and things not working is key.

Quick detection of production bugs rests on the observability built into the application. Speedy recovery from production bugs is a function of deployment practices. Reducing the impact of production bugs follows how you roll out features. All these are a result of tooling, development practices, and engineering culture. Today, these are as important as vanilla testing - manual or automated.

The ease with which you can set up a development environment for the application has a direct bearing on the quality of the product. In a world of micro-services and external dependencies, setting up a development environment can get complicated with a lot of moving parts. If you make it difficult for developers to create a robust development environment, the quality of the product takes a hit.

Static code analysis and enforcing best practices through tooling like pre-commit hooks improve the quality of the code, which directly improves the quality of the application. It is paramount when you use languages that are promiscuous with typing; this is one of those things where the cost is low, resistance is nil, and the rewards are high. Be always on the lookout for tools and processes where you get a better quality product with zero resistance and impediment to speed.

Adopting the asynchronous and event-based architecture and design patterns gives room for error and recovery. In today’s fast-paced environment where you do not have the luxury of time to test every minute aspect of the product, this is a boon.

Do not have a tunnel vision when it comes to quality - do not restrict it to only testing; adopt the “NOT - Not Only Testing” strategy.

Comic from XKCD.

Photo by Austin Neill on Unsplash

Approximately 65,000 people attended the event this year, and I was one of them thanks to my employer Goibibo. It was a 5-day affair. I have been using AWS since 2012, have seen the platform grow from a couple of services to what it is today, but this was my first time at re:Invent.

Reflections on re:Invent follow.

re:Invent has the right mixture of fun and seriousness - multiple parties, tech talks, after hours with food and drinks, fun games, product launches, certification booths, and networking events.

The event takes place in multiple hotels(Casinos) on the Las Vegas strip. AWS ties up with the hotels on the strip, which results in discounted rates for the stay. This sweet deal closes as re:Invent nears, so plan your attendance. Staying close to the venues is highly recommended as the day starts early and ends late.

AWS has free shuttles running from these designated hotels to the conference venues. Another reason to stay in the chosen hotels. These shuttles also run between the various conference sites, but it does take time to move from one to the other. Casinos are like a maze; it is easy to get lost. Getting in and out of the casinos to the shuttle parking lot itself takes some practice. By the time you have perfected this art, the conference ends. AWS stations many people in the casinos to guide and help with directions and information, but still, it eats up quite a bit of time.

Umpteen number of tech talks run in parallel in multiple venues at all the hotels. Talks cover a wide variety of subjects: battle stories -victories, defeats, and lessons from AWS users; the internals of AWS services; best practices, and architecture sessions. As AI/ML are the current darlings of the tech world, there was a separate track dedicated to AI/ML.

AWS was heavily promoting serverless; there was a lot of emphasis on serverless throughout the event. Some of the vintage AWS products like SQS, SNS, and DynamoDB have been re-branded under the serverless moniker.

AWS publishes the schedule for the talks in advance. There is a re:Invent app which has all the details of the event. In the app, you can reserve seats for the talks. If you want assured seating, you need to reserve. Sessions do fill up fast, and the reservation closes as re:Invent nears. All talks have a line for the people who have booked in advance and a walk-in queue. Advance bookers get preferential treatment. The walk-in line is at the mercy of seats being empty. There are repeats of the talks on subsequent days. Sessions are also live-streamed; there is a dedicated area for viewing this on the big screen.

If you are serious about attending as many talks as possible, cluster your talks in one venue and then move to the other. Commuting from one place to the other takes time.

Another highlight of re:Invent is the keynotes from AWS stalwarts like Andy Jassy, Verner Vogels, etc. No walk-ins are allowed for the keynotes; you need to reserve your place in advance. Keynote seating fills up fast and closes as re:Invent gets closer.

There is an expo area where all vendor companies set up stalls and market their products with demos, brochures, and goodies. Expo is a fantastic opportunity to get a glimpse of all the tools available in the market today. The landscape this year was broadly divided into - APMs, logging solutions, and security products; data lake, data visualization, ETL, data warehouses, and data governance tools; SRE and DevOps tools. Even though AI/ML was the buzzword in all of these, surprisingly, there were not too many products catering to AI/ML specifically.

Vendors host multiple after-parties at various bars, restaurants, and nightclubs. One such event was the Sumo slam - Sumo wrestling match organized by Sumo Logic in association with other companies. The venue for Sumo Slam was Omnia, a prominent night club in Vegas.

There are multiple hands-on sessions where you can hone your AWS skills and also earn AWS certifications. There were deep racer league matches too, which I was keen to attend but could not.

The primary venue has a festive atmosphere throughout the conference days - DJs spinning out EDM and live bands performing. There was a mechanical bull ride to test your cowboy skills.

re:Invent started with a kick-off party on Sunday night and ended with a grand closing party on Thursday night. The kick-off party had a product launch - AWS DeepComposer, multiple music and dance shows, food, and drinks, marching bands, roller skaters, impromptu dance troops, and a chicken wing eating contest.

The closing party was massive and held in the Las Vegas festival grounds. It had multiple tents for different kinds of music. There was a venue for dodgeball and another site for other fun games. The EDM stage with the pulsating light show was fantastic. Thankfully, there were plenty of vegetarian options and a good selection of drinks. The highlight was a drone show organized by Intel. After witnessing the show, I believe we are not far from a future where an army of synchronized pulsating drones will replace fireworks.

It was an enjoyable week of broadening my horizon while having loads of fun. Amazon has done a fantastic job in organizing the event.

Am I trying to say that they do not add any value? No, the opposite. They are doing an excellent service by codifying common sense into pithy one-liners, simple rules, and principles.

All these people are generalizing lessons learned from specific instances into a broader set of rules and guidelines which apply to more expansive areas of life.

What do I mean by the above? Let me explain with an example.

Let us say that a company advertises telling - Deposit your money with us for two years for a guaranteed return of 15%. The bond yield rate is 8%. You see this scheme as attractive and invest. After a year, the company owner goes absconding, taking your money with her.

How do you generalize the lesson from this misadventure?

Be skeptical of any scheme(gold, real estate, etc.) that promises GUARANTEED returns over and above the current bond yield rate.

I believe generalizing lessons learned from specific situations to a much broader arena of life is a superpower which everyone should develop. Get into the habit of doing this for everything. If you followed a particular process that led you to success, try to make this process generic so that you can apply it elsewhere too.

Generalizing wisdom helps in pattern matching as well as molding the way you think. You begin to see patterns in your thoughts and actions where you can apply the principles created out of past experiences.

This habit sharpens decision making. You start seeing patterns in decision-making scenarios and can base your decision on a previously created rule.

https://twitter.com/abhyrama/status/1195932687371915265

You may not be able to create rules out of everything - creating a step by step guide on how to balance a bicycle is next to impossible. Wherever it is possible, do it; train your instincts with the rest and let your intuition take care of it.

Photo by Mpho Mojapelo on Unsplash

All posts on daily standups have a fundamental problem. They shy away from tackling the elephant in the room. I am also guilty of this in my take on daily standups. At some level, meetings are a forced collaboration attempt. In an ideal Utopian world, where everyone excels in collaboration and communication, we would not need meetings. Sadly that world does not exist.

https://twitter.com/abhyrama/status/1181261908445298689

Now that we are done away with addressing the uncomfortable truth, let us go deep into why daily standups are essential.

Timely and efficient communication and collaboration can make or break teams. Humans display a diversity when it comes to communication, some excel at it, and some are bad at it. Sometimes, you might not genuinely know you need to communicate or that you are a blocker to someone’s work.

A primary reason for project failures is unmet dependencies and someone not planning for them. When you get people together and create a platform for them to discuss and collaborate, blockers and dependencies which would have gone unsaid otherwise surface.

How many times has it happened that someone raises a red flag on the day a project is supposed to go live? A daily standup ensures a constant feedback loop wherein this does not come as a last-minute surprise.

Daily standups ensure that everyone in the team knows what their counterparts are working on; this prevents people from becoming islands and ensures everyone knows the big picture.

As an organization, how do you develop this habit?

One of the paradoxes in life is that rules set you free, help build good habits, and reduce cognitive overload. Giants in the field of behavioral psychology - Dan Ariely, Daniel Kahneman, B. F. Skinner; all support this. The general prescription to start a good habit or break bad habits is to create a strict set of rules.

Another trick to aid good habits is to design your environment to support the practice; remove obstacles that prevent you from getting into the said habit. Putting it succinctly, make it easy to start and sustain a habit. Charles Duhigg and James Clear have written books on this.

Scheduling daily standups at a specific time with a well-understood format does both the above; it makes it easy and creates an environment for teams to cultivate the habit of collaboration and communication.

People who rally against daily standups tend to be:

• Great at communication and pro-actively do it.
• Individual contributors who excel at their work.

These people operate on individual bits of information and do not see the entire picture. From their narrow perspective, they are correct, but modern workplaces are not only about individual brilliance but more to do with teamwork. An automobile will not function unless all the parts work in tandem; the same goes for a team.

The majority of people do not know how to run efficient meetings; as a result, people have developed an aversion to meetings. This general distaste towards meetings has given the daily standup a lousy reputation.

https://twitter.com/abhyrama/status/1161287211578515456

https://twitter.com/abhyrama/status/509608231711158272

Paul Graham talks about the maker’s schedule and manager’s schedule and how it is paramount that makers get a long uninterrupted chunk of time to create things. To avoid the context switch, schedule daily standup at the start of the day before everyone gets immersed in their work.

Most workplaces are chaotic. Daily standup gives you the means to bring order to the chaos.

Photo by Paulo Carrolo on Unsplash

When a behemoth does something well regularly, they create an impression that that is the new normal. Customers start expecting the same experience from everyone in the field. For example, Amazon keeps upping the ante in e-commerce. If you are a small boutique e-commerce firm, and if you are not close to the Amazon experience, you might be leaving a lot on the table.

Most of the new age enterprise SAAS tools have user experience on par with consumer applications. Earlier, enterprise tools used to be leaps and bounds behind their consumer counterparts. After using these new-age tools, products from some of the established behemoths look and feel clunky. Using them feels like being teleported to an earlier era. If you are an entrenched behemoth, you can get this wake-up call only if you regularly scan your competition, be it big or small.

https://twitter.com/abhyrama/status/1169305135522009089

Eyeing competition also matters when it comes to feature selection. For example, in Slack, you can edit a message after sending. I have hardly seen anyone modifying messages in Slack post sending. Usually, one sends a new message suffixing an * indicating it is an edit of a previous message. Why? We have been conditioned by popular chat applications not to alter chat messages once we hit the send button. None of the popular consumer chat applications have this feature. If you are bucking the trend - first, you should know of this; second, you should figure out how to educate your users to use the nonintuitive feature.

Incumbents also set standards when it comes to UI patterns. If Facebook shows error messages with a red background, you can be sure that most of the world’s population associates a pop-up with a red background as an error. It makes sense for you too to follow this.

I am not advocating aping the competition blindly.

Keeping track of the competition is essential to: 1. Know what is the new normal. 2. Know what might become the new normal. 3. Gauge how far off you are from the status quo.

Image by Sasin Tipchai from Pixabay

Let alone achieving these, we find it tough to start.

Below are mind tricks which help.

Timeout

Whenever you have an urge to do something you have resolved not to, do not give in instantly. Have a timeout after which you decide.

You have decided to go on a diet. An enticing scoop of ice cream is in front of you. Instead of immediately succumbing to your impulse, count till sixty. After that, decide whether you still want to gulp that scoop. In most cases, the urge would have died by then.

Reframe

Reframe unpleasant events in an empathetic manner; this nudges us from a feeling of angst to compassion.

We all go through bad experiences in life. Someone cuts us in traffic; a sales representative is rude to us. Whenever you have frustration creeping in due to things outside your control, reframe the situation.

The person who cut us in traffic is in a personal emergency and is trying to get somewhere. The salesperson, who was rude to us, is going through a terrible phase in her life.

Reframing a situation put us in a humane mode which drives away antagonistic thoughts.

Detach

Detach yourself from a frame of mind you do not wish to be in and observe your thoughts as a third party.

You want to meditate. You are not able to calm your monkey brain. Do not self-berate. Observe your fleeting thoughts as a third person. Do not curse your inability to control your mind. Watch the rise and ebb of ideas — the simple act of detaching and observing helps to calm the mind.

None of these tricks are my creation; I have summarized them in my own words. I have seen these repeated in various forms by experts in the field of psychology, wellness, and spiritualism.

Photo by Faye Cornish on Unsplash

Big O indicates how an algorithm behaves when the size of the input changes. The behavior might be either the computation time or the storage required for computation. What Big O tells is how the storage or computation time scales with the change in input.

The above is an intuitive definition of Big O.

A pivotal point to keep in mind is that Big O talks of change; it does not talk about specifics, i.e., Big O does not tell this algorithm will execute in 1 ms or 1 Mb of space is required for execution.

Let us understand Big O with a non-programmatic example.

There are lots of tennis balls piled up in a court. You have to devise an algorithm to move them to the clubhouse.

One approach is to pick a ball. Take it to the clubhouse and drop it there. Do the same with another ball. Repeat until there are no more balls left.

If there are n balls in the court, you have to go back and forth between the court and the clubhouse n times, O(n) is the Big O of this approach. As the number of balls increases, the work required to clean the court increases. If there are too many balls, with this algorithm, you might end up doing the task the whole day.

If you boil it down, what O(n) tells us is that the task increases linearly with the input, in this case, the number of balls in the court.

Another option is to collect all the balls in a bag in one go1. Take the bag to the clubhouse; this is a constant time algorithm. Since one is not repeatedly going back and forth for each ball; the task does not scale with the number of balls.

When trying to deduce the Big O of an algorithm, figure out the effect of input on storage space or computation.

Why do we need Big O?

Big O has no significance when the input is small. For small input sizes, the efficiency of an algorithm does not make a difference — the effectiveness of an algorithm matters when the input size is large.

Coming back to our example, let us assume we are dim-witted and have institutionalized the back and forth algorithm. We have not bothered to figure out the Big O of the approach. Days pass by; there are at most a couple of balls in the court, and cleaning goes on nicely. One beautiful morning, there are ten thousand balls in the court, and we are in for an unpleasant surprise - we spend hours cleaning the court.

If we had bothered to figure out the Big O of our cleaning algorithm, we could have avoided this surprise.

When someone talks of Big O, they are usually talking about the worst-case complexity.

If the majority of the members of our tennis club are well behaved and do not leave behind balls in the court, then we would not have a tough time cleaning the court. But while calculating the Big O of the algorithm, we assume the worst case, i.e., no one cleans the court after their game, everyone leaves all the balls behind.

While calculating Big O, we do not take into account any of the constant time work done as part of the algorithm. In the first approach of cleaning the balls, if the person takes a break when she is midway through, Big O of the algorithm does not change; it is still O(n).

Why?

Reinforcing the critical point, Big O is the relationship between the algorithm and the input; Big O only cares how the efficiency of the algorithm varies with the change in input. The break the person takes does not vary with the input; if there are two balls in the court or a thousand, the person takes a break when she is halfway through. The break time or the number of breaks is not dependent on the number of balls in the court.

While calculating Big O, ignore any constant time work, i.e., work that does not scale with the input.

In recent times, technical interviews have put the spotlight on Big O. Big O is one of the notations to represent the efficiency of an algorithm. There are many others too.

1 The astute reader might observe that the time required to bag the balls increases with the number of balls in the court. We ignore this and assume that it is constant.

Image by Pete Linforth from Pixabay

There are three rules for creating a successful product. Unfortunately, no one knows what they are.1

If the success of your product depends on changing a deeply ingrained habit, it is going to be challenging. Your product should be attractive enough for people to overcome the inertia associated with behavior change.

For example, Uber changed a deeply ingrained habit of how one hails a cab. Initially, it was cool; hence, people did it. Now it is convenient. Before launching such a product, list down the motivations for someone to use your product.

Sometimes, the most crowded markets are ripe for disruption. Example—Dropbox. Dropbox was a late entrant in the file-sharing and storage market, but it worked flawlessly and conquered the market.

Competition may not always be harmful, especially when you are trying to create a new category. Category awareness is crucial, and if a big guy does it for you, you can piggyback on it.

If a startup had launched a smart speaker like Alexa or Google Mini, they would have two challenges. First would be to educate users on what a smart speaker is. The second would be to convince people to buy their product. Competition brings awareness to a category so that you can concentrate on selling the product instead of educating consumers.

Think about product ownership and usage asymmetry. An online payment solution for schools is the perfect example. Parents want this but the want is not strong enough that they use this as criteria for picking schools. Any product that exhibits asymmetry like this needs to have powerful incentives for both sides.

Every time a customer reaches out to you; it is an opportunity for you to make your product better. Product enhancements should stem from customer service requests.

Users will find unique ways to use your product, which you would not have thought. Go with the flow.

https://twitter.com/abhyrama/status/1157858725601742848

More features are not always better. Be ruthless in culling features. New feature addition is a tug of war between simplicity and complexity. Irrespective of how small a feature is, it makes your product more complex and the complexity compounds over the long run. Even though adding a new feature is appealing, think twice before doing this. On the other hand, culling features is counter-intuitive. Be on the lookout for nixing features and simplifying the product.

Do not get attached to a feature based on the amount of effort you put, the technology used, or the uniqueness of the idea. Usage is the only benchmark for a feature’s success.

Customers do not always know what they want. Be careful while actioning on user feedback. Look around your house to see the plethora of unused stuff you brought thinking you need them.

In a consumer study, testers alternatively played French and German music in a supermarket selling French and German wines. Frech music resulted in more French wine sales while German music did the reverse. When quizzed, buyers were clueless about the music influencing their purchase.

Mix your product insight and intuition with customer feedback before acting on them.

You need something, does not mean the entire world is craving for it. There is no sure shot way to assess this but be aware of this.

Another corollary of the above.

You spot a problem does not mean others are looking for a solution to the problem. People are happy to live with minor inconvenience than change their habits.

Do not look at product features from your point of view. You might have a refined sense of UI, but your customers may not. Always assume a customer-centric viewpoint.

Assume no one reads anything. Figure out ways to make instructions implicit in product flows. My car displays a message on the dashboard when the service is due. It does not rely on me keeping a tab on this. If at all, you have to provide instructions, figure out what will make a user read it. The manual that comes with the Dyson vacuum cleaner has infographics familiarising the user with the product.

Treat customers differently based on their lineage. Someone new to the product needs more hand-holding than one who is used to the product.

New features may not pick up on their own. Figure out ways to incentivize a user to try out a new feature.

Do not be drawn to complexity. A simple feature trumps an overly complex one.

Figure out all the metrics to track before launching a feature. If you do this post-launch, it becomes a shifting goalpost where you are trying to prove the success of the feature rather than figure out whether it met the intended goal or not.

https://twitter.com/abhyrama/status/1155347961649385472

1 Quote Source

Photo by Kelly Sikkema on Unsplash

David Epstein makes a case for generalists. He also talks about Kind and Wicked learning environments.

A kind learning environment is one where all the information needed to make a decision is available. In a kind learning environment, you get timely and accurate feedback.

A game of chess is a kind learning environment. You have at your disposal the complete information needed to make a move. Once done, you get to know whether the move was right or not.

A wicked learning environment lacks all the information needed to make a decision. The feedback that you get is hazy and inaccurate. Luck plays a role in the outcome.

A game of poker is a wicked learning environment. You do not know the card your opponent has up her sleeves. Even if you play your cards right, you can end up a loser if lady luck frowns on you.

Working at a startup is more like a game of poker than chess. You are operating in a dynamic resource-constrained environment. You have too many things to do and not enough people to execute. You have to play the role of jack of all trades and master of none or one. You need to step out of your comfort zone. The business strategy is still evolving; you tweak it as you go - nothing is crystal clear or black and white. There is no one to guide you step by step. You need to make decisions with half baked information. Someone said - all startups are train wrecks inside.

Operating in such an environment is a fantastic learning experience. It is like packing a lifetime’s worth of education into a couple of years. It also forces you to look deep into yourself and check your biases and decision-making process.

Herminia Ibarra, an organizational behavioral specialist, says - “First act, and then think.” The reason being - “We learn who we are in practice, not in theory.” Startups give you a platform to do this.

Startups need generalists - people who can move up and down the technology stack as well as carry out many functions as and when required. In a startup, you get to work on the entirety of a product rather than a tiny weeny bit. You get a ringside view into what it takes to build an organization and run the day to day operations.

Being a generalist forces you to adopt the spiral method of learning. You learn enough to get your job done. You go back to it as and when needed and broaden your expertise. As a generalist, you get comfortable with not knowing everything and taking calls with the available information. You become comfortable with being uncomfortable.

Being able to don many roles lets you develop a broad mental model and good judgment - it gives you intellectual range. Also, it engenders curiosity - you want to learn more and more about a variety of subjects. When you are aware of many fields, you can borrow ideas from one and apply it to another. Charlie Munger calls it a latticework of mental models.

Please do not consider this as a case against working at a big corporation. The title of the post is - “Why work at a startup?” not “Why not work at a big corporation?”. All experience is valuable. In the future, I will write a counter post on why working at a big corporation is valuable.

If you like this, you should also read my old post on " What a startup is not?"

Photo by Samuel Scrimshaw on Unsplash.

The most cited analogy for this practice is the act of burying your gold under a tree versus locking it in a safe. Hiding your gold under a tree is security by obscurity. You are relying on the secrecy of the act for the safety of your asset. If someone uncovers this act, you are busted. Also, this gives you a false sense of security.

Recently my mesh router went kaput. I had to put in place a temporary alternative as internet connectivity is more important than oxygen these days. I had an old router given by an internet service provider. I tried to get it working with my current internet provider, but I could not. The router was locked to the provider. It would not work with any other internet providers. I tried to install OpenWRT on the router, but the hardware specs of the router were too low to host OpenWRT.

The other option was to change the custom firmware of the router to the router manufacturer’s default firmware. The manufacturer’s website had the steps to upgrade the firmware. I tried to follow the same. Strangely, the firmware upgrade section was missing in the router’s configuration interface. A bit of searching on the internet revealed that the internet company had removed this functionality from the interface, but if you directly go to the endpoint of the upgrade, it does work. They had just hidden this functionality in the interface. I did the update, and voila, unlocked the router.

This is textbook security by obscurity in practice. The internet company relied on only removing the upgrade link from the interface instead of eliminating the entire functionality. Anyone who discovers this is free to unlock the router. The right way to do it would have been to disable the feature as a whole. Not just obscure it in the interface.

Whenever you want to secure something, do not only rely only on obscurity. You can add obscurity as an additional security measure. For example, it is a common practice to change the ssh port on servers from the default port. Server admins do this in addition to a lot of other security lockdowns.

Photo by Tom Roberts on Unsplash

In this post, we will look at distributed tracing in the context of debugging web application errors using logs in a microservices environment. We will also figure out the way to implement distributed tracing in the least disruptive manner. The aim is to leverage components that are already part of your application stack without introducing anything new for the sake of distributed tracing.

There are many closed as well as open-source products to get distributed tracing off the ground. There is nothing wrong in using these; in a lot many cases, the investment may not be worth it.

The idea of distributed tracing is straightforward. Every request that lands on your application should have a trace id - a random unique alphanumeric string. The trace id is usually called a request id. From there onwards, the request id should be part of whichever code path the request invokes within the application as well as any external calls it makes. The request id should also be part of the logs generated in all these paths. If the code path invokes external services, those calls should have the request id in the header. The application serving the external calls should follow the same pattern as discussed here.

There you go, the simplest possible distributed tracing implementation.

One decision to make is where to generate the request id?

If you are serving a user-initiated action, generate the request id at the entry point of the request. Post that, all the subsequent code paths and dependent services will use the same request id.

The best place to generate the request id is the web server fronting the application server. All web servers have a way to add a custom header to an incoming request.

Below is how you would generate a request id in Nginx:

[code]

location / {   proxy_pass http://upstream;   proxy_set_header X-Request-Id $request_id; }

[/code]

Nginx doc - http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_id

We should generate a request id only if the request does not already have one. This sort of conditional generation can be achieved using the below configuration:

[code]

map $http_x_request_id $reqid {   default $http_x_request_id;   "" $request_id; }

location @proxy_to_app {   proxy_set_header X-Request-ID $reqid;   proxy_pass http://backend; }

[/code]

Helpful links:

https://stackoverflow.com/questions/17748735/setting-a-trace-id-in-nginx-load-balancer https://stackoverflow.com/questions/13583501/nginx-how-to-add-header-if-it-is-not-set/44761645

Also, make the request id part of web server access log.

Now that we have guaranteed all incoming requests have an id, we need to log the request id along with the application logs.

Most logging/application frameworks have a way to pass a thread-local custom context between layers. You can inject the request id in this context.

If you are on Java using Logback, MDC can help you to achieve this - https://logback.qos.ch/manual/mdc.html If you are on Python and Django - https://github.com/dabapps/django-log-request-id If you are on Go and Gin - https://github.com/atarantini/ginrequestid

Now that the access log, application log as well as any external calls made has request id, you can trace a request’s entire journey throughout the application and debug errors and pinpoint the cause.

Even if you are not trying to tie up micro service calls together, even within an application, having a request id helps in debugging issues faster - you can trace a request as it traverses through the different layers of an application.

Now that the strategy and tactic are clear, some enhancements; this is the right time to talk about span ids.

Let us say you have service A calling service B. Service A generates and sends a trace id as part of the call to service B. In addition to logging A’s trace id, service B should create a trace id of its own and log that too; this is the concept of a span id - an id owned by each service.

You can expand the span concept to a unit of work within a single service. For example, if your application has three different logical units, you can generate a span id for each of them too; depends on your use case and what is it that you are trying to achieve.

Having distributed tracing in place is a huge productivity boost. It is a low investment high ROI activity.

We will try to design the experience based on a couple of simple, timeless principles: 1. Nudging people towards proper behavior - Libertarian Paternalism. 2. Not relying on one’s will power to do the right thing - Ulysses Contract. 3. Designing the environment to influence productive behavior.

Here we go; this is more of a listicle than structured prose.

If you want someone to do something, make it easy for them to do it - following this principle, it should be effortless to book a meeting room. The booking interface should list all the meeting rooms and the available free slots. It should also inform all the facilities a meeting room has like video conferencing, the capacity of the room, etc.

Outside every meeting room, there should be a display of the schedule for the day.

One of the often irritations is you landing up for a meeting in a room that you booked well in advance to find a paper sticking on the door saying the room is blocked for some critical visitor; this sort of overriding should be restricted to only a very few rooms.

How many times has it happened to you that you get up during a meeting to write something on the whiteboard to find the marker and the duster missing? A meeting room should be well-stocked with stationaries so that people do not have to step out in the middle of a meeting to fetch them.

There should be a large wall clock in every meeting room so that everyone is conscious of time. Even better would be a countdown timer which gives an auditory signal when the end of the meeting is near.

There should be a designated place in all meeting rooms to dump phones and laptops. One of the biggest distractions during a meeting is the constant barrage of notifications on devices and the pavlovian reaction to them. Do not rely on people controlling their will power to overcome this but design the right environment for people to achieve this.

Even though meetings have come to be associated with unproductivity, I believe collaboration and brainstorming are essential for crafting the right product. It is paramount that you do everything possible to facilitate communication and discussion between teams; this might make or break your product.

Photo by Drew Beamer on Unsplash

And you are like

Kafka is a distributed stream processor. There is a gulf of difference between a queue and a distributed stream processor. Kafka happens to be the most popular distributed stream processor; there are others too, for example, Kinesis from AWS. Even though you can use a distributed stream processor like a queue, it is a good idea not to. Also, you need to be very clear about the difference between the two; you have to approach and model them differently.

For the rest of this post, we will not talk specifically about Kafka. We will build a mental model of a distributed stream processor like how we did with a distributed key-value store in one of the earlier posts.

Imagine that there is a huge file which your applications can write to. Others can read from this file. Let us refer to this file as “Topic.” The applications which write to the file are called “Producers.” Applications which read from the file are called “Consumers.” Each line in this file is called an “Event/Message.”

There you go, we have clarified many distributed stream processing parlances.

The lines in a file have a sequence based on the order in which they are written. The same goes for messages in a distributed stream processor; they are ordered. Also, each message has a unique incrementing sequence number to determine the order.

Like how a file can have multiple writers and readers, a distributed stream processor can have many producers and consumers.

A distributed stream processor is stateless. All it cares about is ensuring the file is always available. The consumers are responsible for maintaining their state. The consumers keep count of the last line they read from so that they know where to start reading from again. Some distributed stream processors may help you in maintaining the state, but that is not their primary concern.

While reading a file, even though you may be reading from the 100th line, you can always reset and start reading from another line. Reading a line from a file does not delete it from the file. The same is true for consumers of a distributed stream processor; they can go back and forth between messages as they wish.

If we keep writing to a file without ever deleting, it will soon grow large and eat up the entire storage. Distributed stream processors have retention periods to counter this problem. The retention period is configurable based on how much storage capacity you have. Messages past the retention period are deleted.

If multiple processes write to a file, you will soon hit performance limits. Same goes for a topic in a distributed stream processor; shards/partitions are used to overcome this. Instead of writing to one file, imagine that the file is split into multiple smaller files and consumers write to these thus distributing the writes.

How does a consumer decide which file to write to? Partition key aids this decision.

A data point in the message is used to determine which partition to write to. Usually, this data point is hashed, and then modulo arithmetic is used to determine the target partition. There are other schemes too for deciding the target partition.

A queue has a straightforward mental model; first in, first out. Once you fetch a message from a queue, it is deleted; it will not be available again for processing. If a queue has multiple consumers, a message will be available to only one; this is in stark contrast to the consumers of a distributed stream processor. In a distributed stream processor, since we are reading lines from a file, a message can be processed multiple times by the same or different consumer; there is no concept of deletion.

Multiple machines back a distributed stream processor so that they are resilient to machine failures; this should be obvious since it is a “Distributed” stream processor.

Distributed systems are hard; tons of engineering goes into making anything distributed; same goes for a distributed stream processor.

I have written before too on similar lines.

In today’s world of umpteen open source libraries, cloud computing platforms, and StackOverflow, creating software is child’s play. You can look up things on the internet, glue together components, and voila - you have created something.

What is hard is creating quality software. Software that scales; software with very few bugs; software that is elegant and takes care of edge cases and boundary conditions; software that stands the test of time.

I believe the fundamental reason for this is the age-old problem of simultaneously holding opposing thoughts in mind and doing justice to them.

To create something, you need to be an optimist. You need to look at the bright side of things. Whereas, if you want to build quality software, you have to foresee all the probable future problems and develop interventions for them, i.e., you need to wear the pessimist cap for this.

Being an optimist and pessimist at the same time is hard, and I believe that is one of the reasons why quality software seems to be elusive.

Image by Gino Crescoli from Pixabay

Let us first define some of the terms in the CAP theorem.

A distributed system is a group of independent computers coordinating with each other to solve a problem. The group of computers is called a cluster.

A network connects the computers so that they can communicate and coordinate. Whenever a network is involved, there are bound to be delays and outages. Also, the individual computers themselves may go down due to hardware failure. An event which leads to some of the machines in the cluster not being reachable is called a network partition.

Now that we have defined some of the terms, we will build a hypothetical distributed key-value store and witness some of the distributed computing concepts and trade-offs in action.

A key-value store is a dictionary data structure with persistence. A distributed key-value store uses multiple machines for storage.

The user sees and interacts with the key-value store as a single unit through a client. The multiple computers backing up the store is abstracted from the user.

Let us say that we have three machines - m0, m1, and m2 backing our key-value store. m0 is the coordinator and m1, and m2 are the followers.

The coordinator is the one which handles the reads and the writes. The client only interacts with the coordinator. The role of the followers will become clear as you read on.

Let us say that the client writes key-value k0 to the store. The coordinator persists it and then asks the followers also to persist. m0, m1, and m2 now have the key-value pair k0. Along with the key value pair, a version number is also stored; in this case, the version number is zero.

How does a read occur? The read goes to the coordinator m0. m0 reads it’s own value, also fetches the values from m1, and m2. It compares the value it has with those fetched from m1, and m2 and sees that all of them are in sync, i.e., they are consistent. It responds to the client with the value.

Now the client updates k0 with a new value. The same sequence of steps follow, and the version number is updated to one. When this update is in transit, m2 is not reachable from m0 due to network congestion, i.e., a network partition occurs.

Recalling the CAP theorem - Our system can be either available or consistent in the event of a network partition.

If we want to design our key-value store to be consistent, we should reject the write and throw an error to the client.

If we want to design our key-value store for availability, we should accept the write even though we cannot reliably store it in all the machines in the cluster, i.e., maintain the consistency of the value.

Every distributed system has to make this trade-off.

Let our key-value store trade-off consistency for availability.

The update proceeds and the state now is: m0, and m1 have the version one of the key-value and m2 is still at version zero. Now, the coordinator m0 goes down due to a hardware fault, again a network partition; our cluster has only m1 and m2 now.

We have to figure out a new coordinator for our cluster. Let m1 become the new coordinator and m2 its follower. We will ignore how and why m1 becomes the new coordinator; we will tackle that in another post.

If you are a keen reader, you will see that the cluster is in an inconsistent state now as m1 has version one of the key-value whereas m2 has version zero.

Now the client tries to read k0. The same sequence of events as earlier occur for read, but coordinator realizes that the cluster is in an inconsistent state.

How to resolve this inconsistency?

Distributed key-value stores make trade-offs to resolve this.

One option is to let the last write win. m1 sees that the value it has is the latest and updates that value in m2 too and then returns the value to the client.

This is prone to the notorious clock problem in distributed systems.

Another option is to let the client decide which value to keep. In the event of inconsistency, all the conflicting values are sent to the client, and the client is free to determine which value to retain. Once the client decides, the chosen value is updated in the cluster.

In our scenario, with the version numbers, m1 could have taken the call that it has the latest value and update it in m2 too before responding to the client; this is what real distributed key-value stores do. But, it is not possible to do this in complicated scenarios involving highly diverged versions. Vector/Lamport clocks and version numbers aid this process.

At this stage, it should be apparent as to why we store copies of the key-value in multiple machines: to avoid catastrophic loss of data in case of machine failure.

All machines in the cluster keep sending heartbeat messages to each other so that they are aware of non-reachable machines in the cluster. Gossip protocol is one of the ways to achieve this. This cluster membership information is proliferated to the client too.`

We have glossed over a lot of details and simplified things in the interest of readability. A real key-value store is much more complicated and can involve thousands of machines coordinating with each other for storage. After reading this, you should be able to make some sense of the many seminal papers in distributed systems.

Dynamo paper, Aerospike paper, Bigtable paper.

There have been tomes written on team management, but, if you whittle it to the core, it boils down to the following.

You set crystal clear expectations so that everyone knows what is expected out of them. You create an environment where everyone can thrive and meet/exceed the set expectations. You periodically check-in to ensure people have what they need to excel. You hold people accountable. If you see people meeting/exceeding the set expectations, you positively reinforce it. If someone is not able to meet the set expectations, you figure out what is going wrong and fix that. Rinse and repeat.

The above is management in as few words as possible.

Multiple books can and have been written on each sentence in the above paragraph.

Image by Pexels from Pixabay.

The trick is to identify non-critical components of your tech stack to experiment and gain confidence. Once you are familiar with the nuances of operating in the cloud, you can make the big bang move.

One approach is to start with replicating your QA/development/staging environments in the cloud; this gives you a realistic picture of what it will take to move your actual production setup to the cloud.

Another approach is to move offline data processing workloads to the cloud. Data processing thrives on elastic capacity, and the cloud is amenable to workloads like this.

Also, start creating slaves of your production databases in the cloud; this will give you the valuable experience of running a datastore on fickle cloud servers. Operating persistent datastores in a transient environment like the cloud can be challenging. Plan for this and gain experience before you trust your production data to the cloud.

Once you have done the above, you would have enough foresight to plan your complete migration to the cloud.

Photo by Cris Ovalle on Unsplash

Nowadays, the barrier to entry for creating open source software has come down; this has lead to a boom in open source libraries. When it comes to picking one, there are tons of options; it can get confusing. Here are some simple and actionable tips to help you choose one.

Start with the documentation. More precise the documentation, the higher the chances of the library being the right choice. I feel Hashicorp has set the standard here; the attention to detail in their documentation is impressive.

Look at the Github issues. Browsing through the issues gives you a good understanding of the strengths and weakness of the project. An important tip here is to look at the closed issues too, do not restrict yourself to only the open ones.

If the project has a mailing list, lurk around the ongoing discussions. Go through old threads too. Usually, projects have two mailing lists - one for the users of the project and the other to discuss development. The latter gives you a good sense of the direction of the project.

Go through the past releases and the release notes, get a sense of how the project has evolved.

Combination of issues, mailing list, and release history paint the right picture of the evolution of the project, it’s velocity, maintenance, and the responsiveness of the community. All these are very important to a project’s success and durability.

Peek through the source code. More idiomatic it is, the better the chances that the library is well written.

Browse the test cases, the more comprehensive it is, the better the quality.

Following the above will enable you to make an informed decision.

Photo by Javier Allegue Barros on Unsplash

Now that polls are over, it is time for all to draw lessons from the mandate. It is the best time to witness hindsight bias in action. Media will lead the onslaught with a cacophony of op-eds—lessons from the election, advice to Modi, why we got the mood of the nation wrong, what Modi should be doing in the next five years, where Rahul went wrong, etc.

“Skin in the Game” and “Accountability” are best seen during these times. People who say one thing before the election write diametrically opposite words post-election. Renowned journalists who claimed this to be a waveless election, post-result write wordy articles on why the Modi wave was intense this time. Nevermind the fact that these are the same people who got the 2014 polls wrong too. The best part is, these personalities will soon start advising in television debates on what the government should be doing in the next five years to win over people’s hearts.

The thing I find fascinating is no one holds these individuals accountable for what they wrote before elections; no one questions their credibility and so-called expertise.

I believe it is almost impossible to predict the future, but what we can do is observe the broad trends; the trends should be the guide to the future. Yogi Berra captures it best:

It’s tough to make predictions, especially about the future.

The idea that the future is unpredictable is undermined every day by the ease with which the past is explained.

The above quote is from Daniel Kahneman; this should be the major takeaway from any massive upheaval.

PS: Contrary to the above, some people believe that the future is predictable; there are projects too on this.

Photo by Edwin Andrade on Unsplash

Eons back in internet history, Joel Spolsky wrote a post on leaky abstractions. The gist of the writing is that even though we create software abstractions to make life easy, they somehow unintentionally leak through.

With technology getting a lot more complicated, I believe that the software abstractions that developers create should intentionally leak to some extent to let non-developers design the right product experience.

Taking a concrete example, let us say that you are creating a search utility which goes over a lot of data. Your algorithm has two steps, one that works on recent data and the other, older data. Recent search is blazing fast while searching through old data is slow. You do both in parallel, combine the results and display it to the user. You have two options here; you can abstract this two-pronged approach from the product teams and act as if the search is a single step process or you can leak this abstraction so that they can take this into account while designing the product.

What can the product team do with this information? They have multiple options.

Show the faster results first without waiting for the slower ones. Where the older results are displayed, show a loading placeholder, and then populate the results as and when they become available.

Another option is to make this a two-step process by showing only the recent results and then asking the users explicitly whether they want to search through old data too.

They can do this only if your abstraction leaks through informing them of the internal search nuances.

You need to be judicious with what you expose, and what not, so that the product teams are not burdened with unnecessary details - striking the right balance is essential.

Image by Jesse Bridgewater from Pixabay

I see a lot of shared two-wheelers on Bangalore roads these days. The expectations of the companies which buy these for shared mobility would be very different from individual buyers of two-wheelers. As of today, I do not know of any company designing vehicles keeping shared mobility in mind; but I am sure it is not far away. Someone is going to do a deep re-think on this and come up with a design that is amenable to mass public ride sharing.

Samsung opened a large experience center in Brigade road, a high street shopping destination in Bangalore. MI has a similar one in Indiranagar, another bustling area of Bangalore. The emphasis in these places is not on selling but letting consumers experience the product. These centers would roll up to marketing functions than to sales. If you think about it, sales and logistics are no longer the thorny problems they used to be before; technology has taken the pains out of these to a large extent. I believe physical retail is going to go through a deep re-think where it will be more about product experience than selling.

Boston Dynamics has been releasing jaw-dropping videos of their robots in action. These robots seem to be getting better by the day. We are not far from a time when a lot of the hard and mundane physical labor will be carried out by these automated agents than humans. But none of our current products have been designed keeping these automated agents in mind. For example, if you want an army of drones to fix and replace the street lights, the light assembly needs to be designed in a way that is drone friendly. As of today, it is not; these have been designed keeping in mind a human carrying out these maintenance tasks. The same applies to almost everything that we see around. Someone is going to start re-thinking on these designs and changing them to be more automated agent friendly.

I firmly believe the trifecta of technology, automation and sharing economy is going to bring about a massive re-think in how we look at the design and assembly of almost everything.

Photo by Tincho Franco on Unsplash

Technology is getting commoditized. In the yesteryears, you could have counted on your software frameworks and technologies, hardware prowess, access to capital and talent pool as competitive advantages. In a world, which is getting commoditized, the access to these has been democratized, thus leveling the playing field. Today, it does not matter whether you are a single person startup or a large multinational corporation; everyone has access to the same resources.

Cloud providers ensure you have enough servers and hardware at your disposal to meet growing demands. Open source frameworks and technologies provide the technological backbone to build stuff. Umpteen services exist to recruit talent and develop products; co-working spaces take care of physical office requirement; social media gives you access to a wide audience. In spite of these, if you are short on capital or want to splurge, there is the ever-growing VC industry to take care of your capital needs. The icing on the cake are sites like Stackoverflow and Coursera which have levelled the playing field when it comes to access to knowledge.

In light of this, what constitutes your business competitive advantage?

One short answer, developer productivity.

Developer productivity dictates how well you can effectively and efficiently leverage all the resources at your disposal to build a great product and trounce your rivals. Developer productivity is a trifecta of having the right engineering tools, structure and processes. When you excel in these three, developers can move fast, and business can conquer markets with ease.

Developer productivity is too vast a topic to be covered in one post. I will be writing more on this in the coming months; it is an area which is extremely close to my heart, and I firmly believe companies do not focus enough on this.

Photo by Erwan Hesry on Unsplash

Daily standups are a hotly contested and controversial topic in the software industry. There are some who actively champion it, and then there are others who detest it. I belong to the former camp, and firmly believe every team should do a daily standup.

One of the primary problems that teams face is information asymmetry. Some members tend not to know what their teammates are up to; this results in individuals becoming islands and a breakdown in efficiency. I can write an entire post on the problems this causes but for some other day. One of the reasons for this is not everyone in the team communicates at the same level; all teams face communication asymmetry.

There is a vast diversity in humans when it comes to communication. Some excel in it; some do not. Efficient and timely communication can make or break teams and are a huge indicator of a team’s success. Standups force everyone to communicate. You do not have the option of not communicating during a standup.

A format I have seen work is going around with each one telling: 1. What they did yesterday. 2. What they plan to do today. 3. Any blockers they are facing which needs immediate attention.

This ensures everyone in the team knows what their teammates are working on.

Apart from information proliferation, this also helps in surfacing blockers. A lot many people do not know when to surface blockers, and they wait for things to go down before yelling for help. Daily standup gives everyone a forum to do it before shit hits the fan. It also allows others in the team to intervene and course correct when needed.

As with any meeting, it is paramount to keep daily standups short and sweet. Do not allow anyone to hijack the daily standup for lengthy or off-track discussions. Do it outside the standup.

One downside of standup is that people who communicate well and efficiently feel penalized and do not see the worth of it. I was in this camp when I was new to the industry, but as I have built teams and scaled organizations, I have become an acolyte of daily standups. If it is of any solace, think of it like taking one for the team and the organization.

Photo by Margarida CSilva on Unsplash

I have talked to multiple startups doing this. The intention is to future proof the product; this should be the least of your concerns during the early days. In your gestation period, you should be spending your waking hours building features to acquire customers and grow. That is what will make or break your startup, not the abstraction layers you create for the distant future.

If you are using MySQL, then go all in on it. MySQL has a lot of features that are above and beyond the ANSI SQL standards. Do not try to be standard compliant and avoid the esoteric non-standard features; try to leverage them as much as possible. It will take you faster to market at the cost of being locked in on MySQL for the foreseeable future. If your startup survives to see the day when MySQL becomes the bottleneck, you will have enough resources and expertise to make the switch. Same applies to cloud providers. Do not try to be cloud agnostic; utilize the features given by your cloud provider to the maximum.

This might sound orthogonal to some of the advice that you might get on the internet as well as sound engineering practices. A lot of these are written for mature companies with established product and revenues, not for young, early-stage startups. As with everything, it is all about context.

If you think about this, it boils down to a simple decision; whether you live in the present and try to maximize for today and the immediate future or plan out for the distant, uncertain future. A young startup, trying to make its mark on the world, should do the former.

This is a bit counter-intuitive and becomes challenging to do if you have a mindset that is more attuned towards a systematic way of thinking and doing. A lot of things with startups are counter-intuitive and go against the grain. If not, would it not be a cake walk?

Photo by Nancho from Pexels.

Never wait for a feature to be complete for deployment. Break it into multiple smaller chunks and keep deploying these to production even though no one will be using it. Smaller deployments always win over one big bang deployment. Longer the code runs in production, more data points you have to analyze and take corrective steps.

If you can test with real users without them knowing, there is nothing like it. If you put your mind to it, you will always figure out a way to do this. Taking an example, let us say you are building a chat feature for your application. You can stealthily release it without informing your users and send dummy messages on their behalf. The user does not see any chat related functionalities, but behind the scene, you create dummy messages and send them to one another. Apparently, this is how Facebook tested their chat before launch; this allows you to check your feature realistically in ways it will be ultimately used.

Another trick is to log the action without actually carrying out the intended side effect. Say for example; you are introducing an API rate limiter. As a first step, start logging whenever an API call hits the rate limit threshold. During this phase do not block the calls, the idea is to emulate the result; this lets you test out the feature with real traffic and fix any bugs before it starts negatively affecting your API consumers. Once you are confident of your approach, you can roll out the feature where you block offending callers.

Phased rollouts is a life saver. Release features to a small set of users, keenly observe your application and gradually ramp up to 100%; this minimizes the impact of bugs and lets you catch them early.

Another lifesaver is feature flags. The idea behind a feature flag is simple; you put all your features behind an on-off flag which you can toggle if and when needed. The ability to turn off features during their early lives gives you a lot of flexibility and room to maneuver and improvise.

If you are working on refactoring or replacing an infrastructure component, never do a stop the world cut off. Always run both the systems in parallel and only once you are confident of the new approach, do the switch over. Say, for example, you are moving your data source from MySQL to Mongo, start by writing to both the datastores in parallel. Then change your application to work with Mongo while still keeping the option to fall back to MySQL if needed. Put this behind a flag which you can toggle. Only once you are confident of everything working with Mongo, pull the plug on MySQL.

The common thread that holds all these strategies together is to plan your deployments such that they are small and gradual; if something goes wrong, it is not catastrophic. You increase the scope gradually as you observe and gain confidence.

As they say - Risk is what’s left over when you think you’ve thought of everything. A bit of planning goes a long way in reducing risk as well as giving an excellent experience to your users.

Photo by Magda Ehlers from Pexels

I have written about this before.

Please do not take this prose as an argument for being sloppy and building crappy products.

I firmly believe one of the reasons why this occurs is that people confuse knowing with doing. You being aware of something should not force you into using or implementing it. You might have a personal inclination towards a language or a framework; you might even believe that it is the best in the world, but that does not imply you rewrite your currently well-working production application in the said technology. You come across something new, and shiny does not mean it has to be part of your application stack.

When we see something novel and trendy, our brain actively tries to figure out ways in which we can make it a part of our lives.

Another arena where this plays out is in doing things if and when needed. Because you read a blog post touting Redis as the next best thing to sliced bread should not make you go about slapping Redis as a cache in front of your data stores. Your data size might be tiny that it entirely fits into main memory. Achieving five nines of reliability is a daunting task which takes a lot of engineering and effort. As a geek, it is a fascinating problem to work on, but your application may not need it. Just because you are aware of what others are doing to achieve this does not imply you too embark on this path.

If you ponder over this, you will realize this behavior is not only restricted to work but plays out in our personal lives too. When we come across something new and exciting in any domain of life, we try to adopt it irrespective of whether we truly need it or not, and when we do it, we go way out of line in justifying its usefulness to ourselves.

The first one is pretty straight forward - charge for support, maintenance, consulting, and custom development. Software takes an effort to understand and maintain. Either you can do it in-house or outsource it to an external firm. Big enterprises have specific needs which require custom enhancements. They also need consistent support and suggestions. There are a lot of companies which have used this model to generate revenue from open source software like Redhat, Percona, etc.

The SAAS option for open source software has gained immense traction in the last decade or so especially since the advent of cloud. Instead of you taking the pain to host and maintain, the company behind the software deploys and manages it for a recurring fee. Most of the popular open source software is available under this model nowadays. Wordpress, MongoDB, ElasticSearch are some prime examples of this strategy.

Another revenue strategy is the open core model. The core is open and free, but features which enterprises need like security, high availability and user management are part of the commercial offering. For example, the core database could be open, but clustering and high availability might be available only in the retail version. InfluxDB uses this model.

Then there is the licensing play. Software licensing is nuanced and comes with a lot of baggage and restrictions. The open source version of the software is released under a restrictive and commercial antagonistic license. If you want to use the software in a business setting, you have the option of buying a more permissive and commercial friendly license; this is very prevalent in software bundled in commercial products.

It is not uncommon for a company to use a mixture of the above strategies.

In the next part of the series, we will go through some recent developments in the open source world in an attempt to ward off the threat from big cloud providers like AWS.

Image Credit: lucas Favre

One of the most common conflations I see people making is mistaking open source software for free software; both are not the same. Being free is just icing on the cake, the more significant advantage is the freedom and flexibility that comes with open source software.

Let us say you are an enterprise with millions of dollars that has built its entire product on a closed source database. Your business is now profoundly entwined with the success of the database company. What happens if the database company goes kaput and shuts down? You now have to replace the database. Depending on the complexity of the product and the business, this might take significant effort and end up derailing your entire business. Open source software severely mitigates this problem.

There is no concept of shutting down in open source world. Open source software development is inherently decentralized. In a lot of cases, committees govern open source software development. These committees have many stakeholders whose best interest is in keeping the software alive. Apart from this, many boutique firms provide development and maintenance services. All this leads to a robust eco-system that prevents a project from abruptly shutting down and taking you hostage.

Commercial closed source software reminds me of a famous line from the Eagles song Hotel California - You can check out any time you like, but you can never leave. Once you are locked into a piece of software, it is not easy to get out. During pricing discussion, there is an asymmetry. As a locked in customer, it is difficult for you to leave, there is no BATNA. Open source software does not have this problem.

Having access to source code is a huge advantage. When I was building Kwery, I used Apache Derby as the database. I started seeing a weird bug in Kwery which lead me to tinker with the Apache Derby source and finally unearthing a bug in the Derby database. A couple of mail exchanges on the Derby mailing list confirmed this. If I did not have access to the source code, there would be no way for me to figure this out.

I am not saying that open source software is a panacea to all problems and that you should completely shun commercial closed source software. Each has its place but equating opensource software with only free is folly.

You can read the second post here.

Image credit: Photo by Adailton Batista from Pexels

I think the minute you have a backup plan, you’ve admitted you’re not going to succeed.

Elon Musk said so. Chew on this for sometime before you read the rest.

I was not honest with the above. It was not Musk who made the statement but Elizabeth Holmes, at the peak of her popularity. It has been a hard fall for her from then to now; she is now accused of fraud.

Did your opinion of the quote change with the source? Did you go from awe to retching?

I believe we give as much importance to the source of a quote as to the quote itself. We should be internalizing quotes and aphorisms by divorcing the source. A quote should be evaluated solely on its content, not who it came from. When we do not do this, the aura of the person shadows the import of the saying, reducing it to personality worship. The significance of the quote tends to get lost.

The act of viewing a quote objectively also acts like a shit umbrella against famous people getting away with baloney like the above from Holmes. If a successful person makes a statement, we take it at face value thinking if she says it, it must be true. We should guard ourselves against this attitude.

Internalizing a quote just on its content is not easy to do as we all love narrative fallacy, but it is worth trying. As with everything, we get better at it with practice.

Image credit: Ilyass SEDDOUG

How many times has the above happened to you?

I believe almost everyone goes through these in life.

One simple hack to get better and faster at problem solving is to backtrace through your thinking. Once you successfully arrive at a solution, replay your unsuccessful attempts and figure out what you could have tweaked in your thinking process to arrive at the answer faster.

High performance teams do post-mortem analysis after a critical issue. Members create RCA(root cause analysis) document which contains what went wrong, what could have been done to prevent the untoward incident from occurring and what are the steps to be taken to avoid a relapse. We should be applying the same steps to our thought process when we do not arrive at solutions on time; think of this as an RCA of your thinking process.

This simple trick I believe helps us in getting better and faster at problem-solving.

Image credit: Diego PH

W A T A R I

The first step towards embracing observability is figuring out what to track. Broadly, we can categorize software observability into: 1. Infrastructure metrics. 2. Application metrics. 3. Business metrics. 4. Distributed tracing. 5. Logging. 6. Alerting.

Infrastructure metrics: Infrastructure metrics boil down to capturing the pulse of the underlying infrastructure where the application is running. Some examples are CPU utilization, memory usage, disc space usage, network ingress, and egress. Infrastructure metrics should give a clear picture as to how well the application is utilizing the hardware it is running on. Infrastructure metrics also aid in capacity planning and scaling.

Application metrics: Application metrics help in gauging the efficiency of the application; how fast or slow the application is responding and where are the bottlenecks. Some examples of application metrics are the API response time, the number of times a particular API is called, the processing time of a specific segment of code, calls to external services and their latency. Application metrics help in weeding out potential bottlenecks as well as in optimizing the application.

Infrastructure metrics give an overall picture whereas application metrics help in drilling down to the specifics. For example, if the infrastructure metric indicates more than 100% CPU utilization, application metrics help in zeroing in on the cause of this.

Business metrics: Business metrics are the numbers which are crucial from a functionality point of view. For example, if the piece of code deals with user login and sign-up, some business metrics of interest would be the number of people who sign up, number of people who log in, number of people who log out, the modes of login like social login versus direct. Business metrics help in keeping a pulse on the functionality and diagnosing feature specific breakdowns.

Business metrics should not be confused with business reports. Business metrics serve a very different purpose; they are not to quantify numbers accurately but more to gauge the trend and detect anomalous behavior.

It helps to think of infrastructure, application and business metrics as a hierarchy where you zoom in from one to the other when keeping a tab on the health of the system as well as diagnosing problems. Keeping a check on all three ensures you have hale and hearty application.

Logging: Logging enables to pinpoint specific errors. The big challenge with logs is making logs easily accessible to all in the organization. Business metrics help in tracking the overall trend and logging helps to zero in on the specific details.

Distributed Tracing: Distributed tracing ties up all the microservices in the ecosystem and assists to trace a flow end to end, as it moves from one microservice to another. Microservices fail all the time; if distributed tracing is not in place, diagnosing issues which span microservices feels like searching for a needle in a haystack.

Alerts: If you have infrastructure, application and business metrics in place, you can create alerts which should be triggered when they show abnormal behavior; this pre-empts potential downtimes and business loss. One golden rule for alerts is, if it is an alert, it should be actionable. If not, alerts lose their significance and meaning.

Both commercial, as well as open source software, are available to build observability. NewRelic is one of the primary contenders on the commercial side. StatsD, Prometheus and the ilk dominate the open source spectrum. For log management, Splunk is the clear leader in the commercial space. ELK stack takes the crown on the open source front. Zipkin is an open source reference implementation of distributed tracing. Most of the metrics tracking software have alerting capabilities these days.

If you already have microservices or are moving towards that paradigm, you should be investing heavily on observability. Microservices without observability is a fool’s errand.

Market size is defined as - The number of individuals in a certain market who are potential buyers and/or sellers of a product or service.

When building any business, gauging the potential market size is the most critical step. The size of your team and the amount of money you want to raise should correlate with the market size. If the intention is to create a big business, you should be going after a big market. A lot of would-be startup founders are fixated on their idea and raising money that they do not analyze the market size or fool themselves into thinking that the market is vast. Peter Thiel, in his book Zero To One, beautifully explains the significance of market size.

The story that you build around your startup should also reflect the market size that you are after. In 2014, professor Aswath Damodaran estimated the value of Uber at ~6 Billion dollars with the story that Uber is an urban car service company. Bill Gurley, the lead investor in Uber, refuted this saying that Uber is not an urban car service company but a logistics company that is going to be everywhere, not just urban areas and is going to exploit global networking benefits, not local networking benefits. With this story, Bill Gurley was valuing Uber at ~53 Billion, a ~9X increase from Aswath Damodaran’s valuation. With a change in the story, the market size that Uber is after went up. This incident illustrates the importance of framing your startup story for big market size.

Some parting words on market size from Kunal Shah, founder of FreeCharge - Market > Product > Team.

Since the two are disparate distributed systems, one way to achieve the above is to use an anomaly detector. There are off the shelf products for this as well as open source projects. If you do not have the time, need and resources to invest in deploying an anomaly detection system, having a reconciliation system is the way to go.

Reconciliation is deeply entrenched in the financial domain where it is a way of life. The technology world can borrow this and use it as a poor man’s anomaly detector. For the scenario that we started with, we run queries on the data repository of the sign-up and wallet systems at regular intervals. These queries fetch the count of sign-ups, wallet creations, and credits that occurred during the period. Once we have the numbers, all we have to do is ensure that they match. One can do this with a simple bash script; this is extremely simple to develop and deploy.

Reconciliation can play a role in all places where two-phase commit flows are involved. For example, most payment flows follow a two-phase commit process. You first deduct money from the user’s payment instrument and then fulfill the commitment. There is a good chance that post payment debit, your system dies not doing the fulfillment. Having a reconciliation system in place helps you to take corrective action in these scenarios.

Reconciliation is a simple way to achieve anomaly detection until you have the need and the resources to invest in a more robust distributed anomaly detector.

1. Is the VC crazy to invest so much money in that startup?
2. Why does that startup need massive amounts of money?

In my opinion, these questions stem from not having an understanding of how the VC business works.

Let us say that there is a contest with prize money of 1 billion dollars. You have to form teams, do some tasks; whoever wins, walks away with the prize money. There are a hundred individuals who are eager to participate but do not have the financial means and resources to recruit people to form teams and get to work. There is a wealthy benefactor who is ready to fund these hundred individuals with 10,000 dollars each. He strikes a deal with each of these hundred people saying, if you win, you give me 50% of the prize money; if you lose, you do not have to repay the 10,000 dollars I gave you. The benefactor is sure that one of them is going to bag the prize, she does not know who. The benefactor has invested 1 million dollars in total(10000 * 100) to ensure a return of half a billion dollars, i.e., 500x returns.

The above is a crude explanation of how the VC industry works. The individuals are the startup founders, and the wealthy benefactor is the VC. The only difference is, in the real world, there is no guaranteed success, i.e., there is no way for the VC to tell with cent percent certainty that someone is going to win for sure; it is all about probabilities, data, reading the trend and gut instincts. To bring a semblance of certainty to returns, VC spreads her bet on a wide array of startups in different domains; at least a handful of them is going to hit the jackpot.

Why does this startup need so much money and are the VCs crazy to give them that? Startups that VCs invest in are not chasing organic growth. Organic growth is steady, sustainable growth which compounds to big returns over decades. VCs are chasing exponential returns in a few years; not steady growth. The money that VCs give to startups helps them with this exponential growth.

Why so? Double-digit returns are achievable by investing in public companies and debt. Why take the risk of investing in startups with absolutely no guarantee of returns when there are better ways to get there? For the chance that VCs are taking, the gains have to be comparable, i.e., exponential returns.

Why do startups need so much money? To short-circuit the whole organic growth curve and achieve exponential growth so that investors get the returns in a relatively short period. The money is spent on scaling teams, building technology and more importantly in acquiring users through discounts and advertising. In short, the VC capital infusion empowers startups to compress multi-decades of growth to a decade or less.

Why is this startup spending money on discounts when they are not even profitable? Big spending circles back to the concept of exponential growth and building market dominance for a winner take all effect. The idea is that once the startup dominates the market or has enough users, they will eventually figure out a way to make money, dominate the market and turn profitable. Case in point; Amazon, Google, and Facebook. When VCs invested in these firms, none of them were revenue positive, but look at where they are today. Of course, there is a risk of startups never figuring out how to make money, but VC investment is about calculated risks and spreading the bet.

You might have read of startups of previous successful founders raising astronomical capital without even having a product. VCs are not crazy to do this; they are betting and playing with probabilities and sentiments. The chance of the founder hitting the lottery again is high, hence the clamor to invest. VCs know for sure that not all the startups they invest in are going to be home runs, some are going to go dud and die, and they take this into account while investing. Only a very handful of the startups they invest in turn out to be successes and these handfuls generate the returns. A minority of the successes cover up for the majority of the failures. Returning to our lottery example, out of the hundred, the VC is betting on only one person to win, she knows the rest of them are going to fail, but she gets 500x returns with just one win and 99 failures.

Treat this as a dummies guide to startups and venture capital. I have watered down a lot ignoring VCs and funds who invest for more extended periods. Like any business, the VC industry is varied and diverse.

I have been writing for quite some time now. I started blogging as soon as I got out of college. I have been writing on and off since then, but in the last one year, I have been successful in maintaining a writing streak. I have always wanted to write regularly; it is only now that I have been successful at it. A significant impetus to the writing streak was the realization that I was becoming a passive consumer and not a producer. I am a voracious reader, and at one point, I realized that I was consuming stuff without putting out anything of my own. Off-late, I have been promoting my posts a bit more aggressively due to which I have received bouquets as well as brickbats. A couple of questions have come my way along the lines of how do you manage to write so regularly, where do you get the ideas from and some trepidations people go through when they want to express their opinion in public.

I write for myself; this might sound like a cliche with a lot of writers claiming so, but you need to start writing to get this. Writing is like having a structured conversation with oneself; it is therapeutic. It helps to bring clarity to a lot of ideas I have. It also helps me to develop a rubric for thinking and decision making which aids me at work as well as in my personal life. Every day I make tons of decisions, creating a framework for this helps me to evaluate the choices objectively and get better at it. Writing helps me to reinforce the process to make these decisions.

One of the trepidations that most face when it comes to putting out their opinion in public is whether it is good enough, what will others think? Whenever I look at my past writings, I have a terrible feeling in my stomach. I can spot quite a bit of grammatical errors, half-baked ideas, and poorly constructed phrases. I am sure I will feel the same about this post in future. But, you get better by doing, not by not doing, hence, if I write more, I should get progressively better at it, at least, that is the hope. Seeing this from the lens of a journey rather than a destination helps. Ask yourself the question; what is more important to you? Is it your image in other’s eyes or your improvement? What do you prioritize?

What if I write something which I no longer believe to be true? It does not matter. If your views and thoughts are not constantly evolving, there is something wrong. If your beliefs are changing, it means that you are getting exposed to diverse ideas and opinions and you are continually updating your thoughts. A better way to put it is; strong opinions, loosely held. My writings are a reflection of my views at a particular point in time; they may very well change in the future. Wise people understand this.

Another common misconception everyone has is that one has to write something long and unique. Even I held this view earlier, but I have changed. I no longer believe that writings have to span pages to be effective. Shorter essays are equally palatable and powerful. Haikus, proverbs, and parables are some examples of short useful musings. Even if you voice something that someone has written before but color it in your perspective, it adds a lot of value.

How do I write?
I have a rough idea in mind. I draft that. I sleep over it for a couple of days, editing it now and then. I try to shorten it as much as possible nuking unnecessary words and sentences. When I feel I have been moderately successful in communicating the concept effectively, I publish it. After posting, I find numerous ways in which I could have done it better, but I have to draw the line somewhere.

I am writing this in the hope that I can enthuse others to start writing more often.

What does writing idiomatic code mean?

Let us say you are using Python to populate a list with numbers.

One way to do this is

nos = []
for i in range(5):
    nos.append(i)

Another is

nos = [i for i in range(5)]

The second one is the idiomatic code. It leverages the Pythonic way of coding. Python is just an example here; every programming language has a philosophy and a method of doing things. Code that adheres to this is said to be idiomatic.

The advantage of idiomatic code is that it takes less mental work and space to understand. The first one spans a couple of lines which translates to spending more grey cells to understand what is going on. The second one is concise and to the point. You end up expending less mental bandwidth to understand the second piece of code.

Also, idiomatic code is an inside speak between people in the know; like how a society functions with social norms and conventions, programming languages and communities achieve the same through idioms and conventions. You look at idiomatic code and you know instantly what this piece of code is trying to do, it is embedded in your subconscious like muscle memory.

Learning a programming language is not just about learning the syntax, it is more about learning the idioms and standard conventions of the language and the community behind it.

PS: You can populate a list in Python in a lot of different ways including using built-in library functions. Debating that is not the point of this post.

Given a choice between the two, developers usually gravitate towards from scratch development. Developing something from scratch is an intensive creative work where you have the freedom to shape the product the way you see fit. Hence, it is pretty obvious why people prefer this. I draw a parallel here with Calvin Ball. For those of you not familiar with Calvin ball, it is a game that Calvin invented where he makes rules on the fly during the game. From scratch development is akin to Calvin Ball, you can create and amend rules on the fly. If you chose a framework and in the course of development you see it does not fit the bill, you have the freedom to swap it with something else. You are operating under a lot of degrees of freedom.

Maintaining and enhancing existing software is more like solving a puzzle or playing a game with well laid out rules. Someone has already laid the foundation or in a lot of cases built the entire structure. You first have to expend time and effort in groking this and familiarising yourself with what is already there, only then you will be able to do something. A lot of times you need to get into the mind of the original developer and decipher things from her perspective. Working on code written by others is more like Sherlock Holme’s work. When you do changes and enhancements, you have to ensure what you are doing fits well into the existing framework. You are working in a constrained environment; you have to stick to the rules of the game. All this is as much or sometimes more challenging than developing software from scratch.

Debugging is an acquired skill which carries over to all areas of development. When you troubleshoot code written by others, you become more attuned to add enough debugging information in the code you write. You will start empathizing with the person who will maintain your system in the future and ensure that person has enough data points to debug when things go wrong. It might as well happen that that future person is you only. Injecting debugging information and future proofing your project is a fundamental behavioral change that maintenance induces in you.

There is nothing wrong in preferring to create something from scratch, but it is imperative to have the second skill set under your belt. The real world requires more of type two work than type one. If from scratch development is all you have done till now, it is high time you challenge yourself with category two work. You will feel a bit frustrated and handcuffed in the beginning, but the way to approach it is like solving a mystery. If you see it that way, it becomes a fun and entertaining experience.

PS: Calvin and Hobbes image taken from Wikipedia.

Imagine that you run a store with only one customer service representative. As soon as a customer walks in, the customer service representative greets the customer with a quick hello saying – “If you need any help, give me a shout, and I will help you out.” She then waits for the customer to seek help. She aims to complete the interaction as soon as possible and wait for the next communication. When a customer asks for help, she quickly answers the query and goes back to waiting. If a customer asks where is the washroom, she points in the right direction quickly and reverts to waiting. If a customer asks her for the price of a product, she quickly conveys the price and goes back to waiting. The point to note here is that there is only one customer service representative for the entire store servicing all customers. This model works exceptionally well when the representative is fast, and the answers to the queries are quick. Concurrency based on events works like this.

Now consider the situation where you have five customer service representatives in your store. As soon as a customer walks in, a representative is assigned exclusively to that customer. When another customer walks in, one more representative is picked from the pool and assigned to the customer. The critical point to note here is that there is a one to one relationship between the customer service representative and the customer. When one representative is servicing a customer, she does not bother about other customers; she is exclusive to that customer. Since our pool has five representatives, at most, we can serve only five customers at a time. What do we do when the sixth customer walks into the store? We can wait until one of the customers walks out or we can have a rule saying that a representative services a customer for a fixed period after which she will be assigned to another waiting customer. She is reassigned to the original customer once the time elapses. Concurrency based on threads works like this.

Coming back to the scenario wherein the sixth customer walks in. Now, we have to ask the sixth customer to wait until a representative is free. On the other hand, we have to wean away a representative from one of the existing customers and assign her to the new customer. When this happens, the customer who was initially being serviced by this representative has to wait. After the elapsed time, we have to assign the representative back to the original customer. When a lot of customers walk in, and you have a fixed no of representatives, quite a bit of coordination is needed to service all customers satisfactorily. In a computer, the CPU scheduler takes care of switching between tasks. Switching is a comparatively time-consuming operation and an overhead of the thread based concurrency model when compared to an event based one.

In the single representative scenario, what happens if one of the customers starts a long conversation with the representative? The representative will be stuck with the customer, and if other customers have queries, they will have to wait for the representative to finish the ongoing conversation. Also, what if one of the customers sends a representative on a long-running errand like fetching something from the depot a mile away? Until the representative returns, all other customers have to wait to get their queries resolved. One egregious customer can jeopardize all other customers and hold up the entire store operation.

Hence, when working with event based concurrency, it is essential not to: 1. Carry out CPU intensive tasks akin to having a long-running conversation with the representative. 2. Carry out blocking IO tasks similar to sending the representative to the depot.

NGINX and Redis are probably the most commonly used software that leverage event based concurrency. The workloads that these cater to are quick. Hence event based concurrency makes perfect sense here.

Taking the case of NGINX used as a reverse proxy, what does it do? Pick a client connection from the listen queue, do some operations on this and then forward it to the upstream server and then wait for the upstream to respond. While waiting for the upstream, NGINX can pick more client connections from the queue and repeat the above. When the upstream sends a response, it relies on this back to the client. Since all these are short-lived operations, this fits beautifully into an event based concurrency model. Good old Apache HTTP server creates a thread/process for each connection to do the same. The no of threads it has constraints apache. If the number of incoming requests is more than the number of threads in its pool, it has to deal with switching and coordination. NGINX does not have this overhead which makes it comparatively faster than Apache in real-world workloads. All of this is a bit simplistic and hand-wavy but should convey the idea.

Event based concurrency cannot leverage multiple CPU cores which all modern processors have. To do this, you create one event unit for each core usually called a worker. Also, most software that leverage event based concurrency adopt a hybrid model where they use event based concurrency for short-lived quick operations and off-load long-running tasks to a thread/process.

I have glossed over a lot of details and nuances to explain a complex topic like concurrency in simple terms. Treat this as a good starting guide to dig more into this fascinating world.

The regular interview process is biased toward extroverts and people who can speak well. In my opinion and observation, this does not correlate to someone who is good at what she does. I have been observing this since my college days. During campus placements, flamboyant and boisterous students made their way into companies while the quiet ones who were good at studies and who could get along with others were often overlooked. The same thing happens to a lesser extent with the regular interview process too.

Taleb says the following of doers.

For those who do things, it is harder to talk about what they do. Reality doesn’t care about talk. It is full of pauses and hesitations. Clear, non-hesitant speech is the domain of non-doers.

Interviewing is an intimidating experience, and not a lot of people excel in this, especially people who are good at what they do. Also, the kind of questions that are asked during an interview has hardly any resemblance to the day to day work. I have written about this before too. A big step in this direction is to craft an interview process that mimics your organization’s day to day tasks. The best way to do this is to have an assignment based interview process. You can create assignments that resemble tasks that you do on a daily basis and ask candidates to work their way through this. An approach of this sort takes the whole adversarial, and interrogative tone out of interviews and makes it an enjoyable experience.

An important aspect to take care while creating an interview process is to take subjectivity out of interviews. Who conducts the interview should not have a bearing on the outcome of the interview. Another thing to keep in mind is to subject all candidates of a particular level to the same questions. Following the above ensures that you are evaluating every candidate on the same yardstick. It is essential to do this to create a quantitative interview process. This sort of standardization also helps in creating a checklist of what you expect in answers to questions. If you have these covered, anyone can take anyone’s interview and evaluate candidates. Is it not ironic that startups claim to be data-driven, but when it comes to interviews, the whole data-driven approach takes a corner seat.

An example interview template would be to have an assignment that closely resembles a task related to work. Try to keep the assignment practical yet straightforward. A good candidate should not take more than 3 to 4 hours to complete the assignment. On submission; evaluate the assignment on code quality, conventions, comments, instructions to run, test cases, boundary condition handling, error handling, and documentation. Post that, have a phone screen with a bunch of questions that mostly have a one-word answer. Make this as practical and broad as possible. Try to touch all aspects of work from data structure choices to computational complexity to databases, web servers, etc. The idea here is that if someone is good at what they do, they should be able to answer these without any preparation. Then invite the candidate for face to face round where you ask the candidate to expand on the take-home assignment. Add a bit more complexity and see how the candidate fares. Finish off the entire process with a design round. I have presented a very rough template, tweak it, add more rounds and improvise to fit your needs; every organization is different.

Along with all the above, soft skills are critical. The sad fact is a lot of people do not know how to conduct interviews. I have heard of instances where interviewers were rude and outright obnoxious, not punctual, candidates not kept informed in advance of the interview schedule, candidates not being adequately attended during the face to face rounds, etc. All these play a significant role in shaping the image of the organization in the candidate’s mind.

In the talent market, big companies, as well as startups, are competing for the same piece of the pie. By adopting a hiring process similar to big companies, startups are not going to get an ace up their shoulder. What startups should be focussing on is coming up with a flexible hiring process that is unique to their organization. It is not easy to execute this in a big corporation but very much doable in a startup. Starting from how you engage with a candidate for the first time to the interview to the offer rollout process, everything is ripe for disruption. Differentiate yourself from others. An interview is a big opportunity for you to make a mark on the candidate, seize it right there.

The same principle applies to systems and application design. For example, let us say that you have a development server where someone is running a production cron job. Since this is a development server, someone might take it down for experimentation. No one expects the non-availability of a development server to have untoward consequence.

Whenever you deviate from the expected, ensure you scream from the top of your voice so that no one misses it. Documentation, common conventions and putting in the right processes are some of the ways to mitigate this. The best is not to do it. Whatever you are doing, it always helps to ask, is this a deviation from the expected? If I am not part of the inner circle, would I expect it to be like this?

You have an established product, and you introduce a radically new/different feature. You are enthusiastic, but the metrics show users are hardly using the feature.

There could be two reasons:
1. Users do not see a value.
2. Could be resistance to newness.

It is essential to zero in on the above to decide whether to put more effort into the feature. If this is indeed resistance to newness, there might be multiple ways to nudge your users towards the feature gently.

A straightforward way to figure this out is to measure the stickiness of the feature. Of the small percentage of your users who interact with the feature, how many of them come back to it subsequently, i.e., once people are acquainted with the feature, do they come back to it later? If you see stickiness in this cohort, it is a good indicator that the feature is of value; you are doing a lousy job educating your users and leading them to the feature. If not, it is time to cull the feature and invest that time and energy in something else.

A straightforward hack to keep discussions on track is to write things down. Project a shared document where you note the objective and the point of contention. Whenever matters go awry, point people to the shared document; this helps everyone stay focused and not shift goal post as the discussion progresses.

Irrespective of how rational and mature one is, when someone disagrees with something that one believes to be true, one tends to become defensive and shift goal post without genuinely being aware of it. Writing things down makes one aware of this and helps in shepherding the discussion in the right direction.

Whenever I explain startups spending marketing dollars to acquire users even when they are not generating any profit, I get a stunned look from people coming from a traditional business background. It is difficult for them to grasp the concept of betting on explosive future growth at the expense of today.

Phil Knight, in his book Shoe Dog, writes a lot about how his bank was asking him to preserve capital when all he wanted to do was grow Nike at all costs during its fledgling years.

A lot of prolific US citizens opinionated that Trump had a naught chance at US presidency. The same goes for Brexit.

What is common in all these situations is a difficulty in viewing the world from a lens not tarred by our own experiences. Even if you want to do this, it is tough to implement because you do not know where to draw the line. Tomorrow, if a person tells you that she has invented the perpetual motion machine, what do you do? Do you dismiss it outright or be skeptical of this person’s claim?

In all these scenarios you have to suspend your rational mind and view things from a radically incongruent perspective. It is easy to write this but extremely difficult to implement.

https://twitter.com/zoho/status/1044249648993525760

Whenever I hear of events like this, I am reminded of a couple of pages in " The Black Swan". Taleb calls it “A new kind of ingratitude”.

The idea presented by Taleb essentially boils down to a person who takes steps to prevent something catastrophic from happening. Since that person has taken steps to prevent the catastrophe, the catastrophe never occurs. Thus the person never gets his due and dies a silent hero.

This is a very fascinating thought that keeps repeating in all aspects of life. Whenever it floods, we make a big deal of politicians who fold their sleeves and get into action. What about that politician who took the necessary steps to prevent flooding?

Whenever there is a production issue at work and a team goes out of their way to put out the fire, that team is lauded. What about those teams that took steps to prevent something like this from occurring in the first place?

Software security is one big area that falls in this category. If you have a great security team, life would go on humming silently. You need to have the right tech leadership to recognize this otherwise it falls bang into ingratitude category.

This is a very obvious thought but Taleb has done a great job of giving structure to this idea. If you keep your eyes open, you see this happening around you all the time.

A radical solution to the problem is to use some other means of transportation instead of walking. If you go to a walking expert, you are going to get tips on improving your walking speed. The expert is not going to ask you to forego walking and use a different mode of transportation. Also, if you are deeply attached to the idea of walking, you might not think of a solution beyond walking. Improving your walking speed is a micro solution, whereas using some other means of transportation is a macro solution.

The above is a contrived example but something we come across in our professional and personal lives, both as problem solvers as well as ones facing a problem. Programmers sometimes try to optimize the hell out of a piece of code while the right approach might be to throw away the code and use something else. Organizations seek to nail down a process to the last mile while a sensible solution might be to do away with the process entirely.

We lean towards micro solutions when we are either deeply entwined in a problem or are the domain expert in that particular area. In these situations, we tend to think within the bounds of a problem and not outside.

When you come up with a solution, bracket it as micro or macro. Being aware is the first step towards becoming better at anything. Also, an outside view helps. Find someone who is not an expert in the domain or one who is not acutely aware of the problem. Run your solution through them. They might lead you to a macro solution or make you aware that what you have is a micro solution. Taking time and mind off a problem helps, like how Archimedes had his eureka moment.

Last but not the least, take a walk.

If you have a workload that need not be addressed synchronously, use a queue.

If your customer-facing application is riddled with workloads that can be deferred, move these to a queue thus making the customer-facing application lean and mean.

Think of a queue as a shock absorber.

There are workloads that need to be processed immediately with sub-millisecond latency, and then there are ones where you have the luxury of taking the time. It is advisable not to mix these in an application. The second kind of workload can be addressed by moving it to a queue and having a consumer process them.

For example, consider a scenario where you are consuming messages and persisting them in a data store. These messages are coming in at a variable rate, and at its peak, the data store cannot handle the load. You have two options. Scale the datastore to meet the peak load or slap a queue in between to absorb the shock. Queue solves this problem in a KISS manner.

Queues enable applications to be highly available while giving enough room to maneuver. As long as the queue is highly available, the chance of message loss is almost nil. Since a queue is durable, you need not perfect your consumer’s high availability; you get leeway to manage.

With applications embracing microservices paradigm, there is a lot of API back and forth. Not all API consumption has to be in real-time. Whatever can be deferred should use a queue as the transport mechanism.

Queue introduces a bit more complexity into an application but the advantage it brings to the table makes it a worthwhile investment.

Category one is due to human nature. Deviation from an established routine causes queasiness and a yearning for the old way. It takes over-communication, repetition and sometimes “just giving it time” to tide over this initial phase; this is usually a short-lived phenomenon.

Category two is the troublesome one. When someone complains about a newly introduced process, it is essential to get to the source of this discomfort. Prod as to whether the reason for disapproval falls into category one or two.

A suitable process has to roughly follow the Libertarian Paternalism idea popularised by Behavioural Economist Richard Thaler. The process should be a nudge towards better behavior rather than a dictatorial dictum. A process whose intention is to police people does not end up well.

A new process introduces some amount of friction, but this friction has to be local, not global. This friction should not slow down the task at a global level; instead, it should aid speed, agility, and stability.

Take the checklist process as an example. It nudges people towards being more aware and aids better behavior. It does introduce friction at the local level, but on the whole, globally, the task speeds up with a much better result on an average.

It always helps to think along these lines to figure out whether a new process is worth its salt. Instead of introducing a new process and then reneging, put in the effort to evaluate the efficacy of a process beforehand.

Hospitals have figured out that by following simple checklists they can reduce the surgical mortality rate by a big percentage. The airline industry, which is an epitome of the highest possible standard in safety and training, relies on checklists for fail-safeness.

Who does not forget daily chores? Who has not kicked themselves for being late on bill payments? A to-do list solves this problem. There are tons of to-do apps out there, just pick one to start with. Apart from solving daily trivial problems, a checklist is a great way to keep track of long-term goals too, be it saving more, losing weight or adopting a healthier lifestyle.

Professional life requires one to regularly follow-up. A checklist is a great way to list down the follow-ups needed. On the flip side, no one likes doing follow-ups. If you want to be one of those rare people who respond without requiring to follow up, use a timebound to-do app to keep track.

Checklist brings efficiency to project management. Having a checklist of all the tasks to be accomplished before taking a project live leaves little room for ambiguity, misses and last minute scurrying.

A short-term or throw away project may not justify the time spent in automating disparate tasks related to the project. In this scenario, a checklist is a great substitute.

All organizations have an onboarding process. Make a checklist out of it and hand it over to employees on the day of joining. All software projects have some common alerts and metrics needed. Checklist them. Once you do this, you do not have to bother about this for every new project. View checklist as a temporary substitute while you scout for software to automate these.

In short, a checklist can be used as a Swiss Army knife to organize and automate life. A checklist institutes repeatability eliminates ambiguity and improves efficiency.

PS: Link for the above comic from XKCD.

Many security bugs can be avoided by making a clear distinction between authentication and authorization. When one logs into Facebook, one uses a username and password. Facebook lets you log in only once it is sure that you are the owner of the account by verifying your password. This is authentication. Once you log in, you cannot view all your friends’ photos. You can only view those photos which your friends have authorized you to view. This is authorization. There is a class of security bugs that arise because developers have not made this distinction.

A lot of security is knowing what not do. Security by obscurity and hand rolling security algorithms and protocols are the two things that immediately come to my mind. For example, while storing passwords, instead of coming up with an elaborate custom secure storage scheme, employ the industry standard bcrypt.

There is a thought process that you will do better security by having tons of access control. One of the manifestations of this is restricting SSH access to production boxes. Unless you have invested tons in tooling, this slows down teams drastically. In today’s world, where speed is paramount, this does not work. Under pressure to do things fast, teams find ingenious ways to circumvent these controls. Strict access control only works in organizations which are fine with taking things slowly but this usually stifles productivity and leaves a bevy of frustrated developers. The only way around this problem is to have the most necessary access control and take care of the rest through tooling. An example is how Netflix uses tools to enable developers to SSH into production boxes without compromising security.

Security implemented in a naive manner goes against the human nature of seeking to accomplish tasks in the least restrictive manner. If you do not invest in tooling, security always gets in the way of accomplishing things.

A less intrusive way of doing security is to configure systems with sane defaults. An example—when you provision a server, ensure that it is fortified by default. If you are using external tools, configure them with defaults. For example, if you are using Slack, configure it so that only people with your organization’s email address can sign up. Carry out a periodic audit of systems. This could be anything from periodically scanning SSH access logs to repository audits to ensure secrets and passwords are not leaked.

No writeup on security can be complete without touching upon compliance. There are tons - PCI, HIPAA, SOX etc. All these come with their own baggage. One simple way around this is to first understand what and all parts of your application have to be under the scope of compliance. For example, if you have an e-commerce application taking credit card information, you have to be PCI compliant. But this does not mean your entire application has to be under the scope of PCI audit. You can smartly bifurcate the application into parts that deal with payment and parts that do not. Once this is done, only the parts that deal with payment have to be under PCI scope.

A final note, security is a never-ending concern, there is nothing called enough security. Where you draw the line is up to you.

Here is a hilarious comic by XKCD on teaching a lesson to people who do not follow security practices.

Long time back, in one of the social gatherings, someone asked me what do you do in your pastime? I said I read a lot. That person asked – What was the last book you read? I stumbled a bit and took a name. I realized that it had been quite a while since I read a book. Unbeknownst, I had gotten into the habit of reading blogs and articles online with social media and messaging boards acting as the source. There is always more than you can chew with articles touching a wide variety of subjects. Also, it feels a bit like going down a rabbit hole; one blog/tweet leads to another which leads to another and so on until you lose track of time.

I have been a voracious reader of books since my childhood. I got hooked onto online articles and blogs only during the later part of my life; this got me thinking about the difference between reading blogs versus books.

Even though most books revolve around a core central idea, the author takes the pain to reinforce this with different thoughts and anecdotes. The author builds a structured case around the view and presents a lot of scenarios leading to the core idea. Reading blogs and articles in most cases feels like reading a summary of a concept or just one facet of it.

When you read a book, you are enjoying the journey whereas reading a blog feels more like focussing on the destination. Both have their place, but one should strike a balance between the two.

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

Even though the above might have been in jest, naming variables while writing code is a head-scratching experience. Should I make it short? Should I make it descriptive? If descriptive, how descriptive? These thoughts keep running in one’s head.

A simple strategy is to keep the descriptiveness of a variable’s name in line with the reach of that variable. If the variable is short-lived i.e within a small block, stick to a short name as the cognitive load of the variable is negligible. If the variable’s reach is much larger, as in if it spans a large number of lines, make it as descriptive as possible.

Goes without saying that names should adhere to the conventions that your team has adopted.

It’s understandable when children act out in self-defeating ways. What’s troubling is how many adults do the same. A common refrain: The organization doesn’t treat me well, so I won’t give it my best.

Set aside questions of morality or responsibility—this attitude hurts you more than anyone else. When you don’t give your best, you stop growing. And when you stop growing, you stall your own progress.

The smart move is to give 100%, no matter how you’re treated. If you feel undervalued, speak up. Try to change things. If that doesn’t work, walk away. But don’t slack. Don’t punish yourself trying to punish someone else.

1. A new product or feature gets launched.
2. The PM wants to schedule a report for her feature.
3. The report should be generated at a particular frequency.
4. The PM and others from the C-team should get this report in their inbox.

We had hundreds of reports like this at FreeCharge. We explored tools like Redash, Metabase, etc. However, all these tools geared towards creating beautiful dashboards and visualizations; report generation, and email delivery were treated unfairly. We hacked together a solution to solve this problem with bash/Python scripts, Git, and Jenkins. The business analyst team wrote the query. Still, they had to depend on the engineering team to schedule the report, format the data fetched by the SQL query(using HTML), and deliver the report over email. One of the goals of the solution was to make the business and product teams self-reliant in report generation. Since the solution was stitched together using different tools that were not meant to solve this problem, it worked but had its chinks; this seeded the thought for Kwery in my mind.

Kwery is a tool that solves the above niche problem. It was also an attempt at building a single person lifestyle business that could supplement my regular income. I followed lean startup principles while building Kwery—made an MVP, deployed it at an organization, and then iterated from there.

With Kwery, I had to decide whether to make it a SAAS product or an in-house deployment. I decided against the SAAS route as I was unsure how many organizations would be comfortable giving access to their data sources to an external fledgling SAAS tool. I wanted to make the onboarding process of Kwery simple. Hence, I shunned all external dependencies for Kwery and opted for an embedded database so that I could package Kwery as a single binary. The sole dependency needed to run Kwery is a Java 8 runtime.

I deployed Kwery in a couple of places that use it to date. Kwery has received a lot of love from them, but I never hit the numbers I had in mind when I started. The opportunity cost was bearing on me. I had to take a call between continuing to work on Kwery or cutting my losses and moving on; I chose the latter.

Like the old saying—every cloud has a silver lining; I have open sourced Kwery under the MIT license. It is straightforward to get started with Kwery. Give it a spin, open Github issues if you face any trouble. Pull requests are very much welcome.

Most of the time, one tends to do only one conveniently ignoring the other. Our mind goes into overdrive and quickly implements a hacky solution and then we forget the problem only to find it re-occurring. Or, we languish to implement a long-term solution while the problem drags on.

One of the reasons why we might find it difficult to implement both is that they require a fundamentally different kind of thinking. The quick and dirty solution involves hustling and running around to get things done. Somehow by hook or crook, one wants to plug the hole. A long-term solution requires one to analyze the problem from all angles, think deeply about it and figure out a well-rounded solution. A vague comparison would be system 1 thinking versus system 2.

Both are equally important - neglecting either is a recipe for disaster.

Many are apprehensive about switching programming languages. It is OK to have preferences—I am heavily biased towards statically typed languages with great tooling support—but being dogmatic is not something one should aim for.

What could be the downsides of switching programming languages? I am disregarding the psychological aversion to change and sticking to hard facts.

1. One will lose syntax fluency.

This is a non-issue; the syntax is similar to muscle memory; one will get it back in a day or two. This is akin to swimming or driving after an extended break; one naturally gets it back.

2. One will forget the way of doing things.

Every language has a culture and a community-accepted way of getting things done. Regaining this might not be as easy as syntax retrieval, but one should be able to recoup with some effort and thought.

3. One will not be up to date with the language.

Languages keep evolving, core ideas and philosophy remain the same. The standard library might become more expansive, VM might become faster, some earlier prescribed way of doing things might be an anathema now, but the foundational principles remain intact.

4. There is no demand for this language.

As long as your fundamentals are excellent, this should not be a concern. Some roles require deep language know-how, but these are far and few. In fact, it is the opposite—the more the languages in your kitty, the more the opportunities.

The most significant upside to learning a new language is the exposure to novel ideas and thought processes. Any new language immensely expands one’s horizon. For example, the way Java approaches concurrency is different from Go’s way of handling concurrency. Having this sort of diverse exposure helps one build robust systems and mental models.

Programming languages should be viewed as a means to an end, not an end in itself. There are cases where programming languages make a difference. Otherwise, there would not be so many around. But, you are doing a disservice to yourself by restricting yourself to a few programming languages.

Obstacle racer Amelia Boone says that she is not able to devote enough time to friends and family due to the demands of her tough training regime. That is the price she pays for being on top of her sports.

In the movie HEAT, Robert De Niro says - “Don’t let yourself get attached to anything you are not willing to walk out on in 30 seconds flat if you feel the heat around the corner.” That is the price he pays for being a master thief.

Michael Mauboussin says his quest for knowledge means he misses out on latest series like Game Of Thrones. That is the price he pays for being a crème de la crème investor.

I feel one of the reasons why people give up something too soon or midway is they have not figured out the price they have to pay for doing it.

Everything that one does has a price. Sometimes it is implicit, sometimes not. Better to figure it out beforehand.

We all know that worrying over things that we cannot control is a pointless exercise. We are aware of the many cognitive biases that we have, we still fall prey to them. Why does this happen? There is a huge difference between knowing something and internalizing it.

Daniel Kahneman says that in spite of studying biases throughout his life, he is no better at avoiding them compared to others. Dan Ariely believes Kahneman was playing to the audience with that quote and we do get better at recognizing cognitive biases and sidestepping them.

Two simple practices that I find useful in becoming more aware of my emotions and biases: 1. Carrying out a daily audit. Every night, I go over circumstances that day where I believe I could have reacted better. Along with this, I also ruminate situations where my cognitive biases one-upped me. 2. Whenever I know that I am getting into an unpleasant situation, I keenly observe my emotions. This might be something as mundane as getting stuck in a traffic jam to dealing with an unpleasant situation.

I am not sure whether anyone will be able to completely eliminate these but I believe we can get incrementally better at it. Minuscule daily improvements compound to mammoth changes over a long time.

How does this help?

A simplistic view is that it helps to keep code consistent, especially when multiple people work on it.

A deeper way to look at this I believe is in reducing the cognitive load.

In cognitive psychology, cognitive load refers to the effort being used in the working memory.

If you have conventions, it is one less thing to think about. You do not have to spend mental capacity on thinking whether to name variables small case, capital case, camel case, with hyphen, underscore etc. You blindly rely on the convention. Same applies to code patterns. You look at the pattern and automatically grok the idea; without expending grey cells.

I strongly believe that all tech teams should have conventions wherever possible; outside code too. Freeing up any amount of working memory for things that matter will go a long way towards increasing productivity.

For example, let us take NoSQL databases. A lot of time is spent on singing paeans to the scalability, malleability, etc of NoSQL databases while hardly thinking about the negatives that come with it.

Two simple techniques which give good visibility on anti-features: 1. The very obvious one, Google for the shortcomings. Someone would have written a blog post on the interwebs highlighting how a framework or technology let them down. For example, take this post by Uber on how Postgres did not work as expected for them. 2. Comb through Github and/or JIRA peeking at the bugs raised and enhancements requested.

Both of the above will provide a good picture of the shortcomings. If you are evaluating a closed source proprietary technology, the above may not make the cut.

Once a mental note is made of the negatives, ponder on the scenarios where this might affect your usage. It helps to spend quality time on this as this will save one from a lot of future trouble.

If you think about this, this might sound very obvious but tends to be highly neglected. We get so caught up in the positives of something that the negatives tend to be ignored and this usually comes biting us back later.

An excerpt from the article:

Lucky people generate their own good fortune via four basic principles. They are skilled at creating and noticing chance opportunities, make lucky decisions by listening to their intuition, create self-fulfilling prophesies via positive expectations, and adopt a resilient attitude that transforms bad luck into good.

Patrick O’Shaughnessy, in his podcast " Invest like the best", talks to interesting people. He mainly concentrates on investors who have made it big, but once in a while he also chats with people from other walks of life. A common theme that keeps repeating in his interviews is how these people jumped at opportunities which others had shunned, their optimism and an attitude that stresses on continuous learning and development. These qualities eerily match with what Richard Wiseman says makes one lucky.

In the recent Farnam Street podcast, behavioral economist Dan Ariely says the following - “I gamble with my time. I take risks, I do things that do not seem like the right things to do”.

Two of the most successful and rich people of our times, Bill Gates and Warren Buffett are gung-ho about the future. Bill Gates actively champions positive thinking and wants all of us to cultivate this.

Probably luck is not luck after all. I am sure it is more nuanced than this, but something to ponder about.

One route to get there is to create frameworks, rules, and principles for decision making. When your team wants to do something and are confused about how to get there, they clawback to the principles and use them, for example, take hiring. Having a clear-cut framework for hiring that covers all aspects starting from what questions to ask, how many rounds of interview, how to reject or accept candidates, what qualities to look for in candidates aids the hiring decision process. By having this, teams are allowed to make hiring decisions on their own.

Also, when taking calls, articulate your thought process. Make everyone aware of your assumptions, questions you asked, data you looked at, and what trade-offs you weighed in. Laying out in the open the way you arrived at a decision helps others to traverse the same path on their own the next time.

To summarise, instead of taking calls on behalf of others, go that extra mile to create a framework that enables them to do this independently the next time. Also, laying out the decision-making process in the open allows everybody to peek at your thought process so that they can borrow it the next time.

A practical way to approach testing legacy applications is to first scope out the intention behind the test. Is it to test the behavior of a particular method, an API response or how an application behaves post an HTTP form submit? Next step is to jot down what and all has to be done to enable this. For example, if a database is involved, it can be mocked or a test database with bootstrapped data can be used.

The gamut of changes needed to inject testability into an application that has never seen testing before should never be underestimated. The way you would structure testable code is vividly different from coding being incognizant of testing.

Take a look at the code below, how would you unit test getUser method without creating a database connection?

public class Foo {
    DbConnection connection = null;
    public Foo() {
        connection = <establish db connection>;
    }

    public User getUser(int id) {
        ////Query db and get user data
        User user = new User();
        //Fill user with data from db
        return user;
    }
}

To mould this into testable code, DbConnection creation needs to be decoupled from object creation, like below:

public class Foo {
    DbConnection dbConnection = null;
    public Foo(DbConnection dbConnection) {
        this.dbConnection = dbConnection;
    }

    public User getUser(int id) {
        //Query db and get user data
        User user = new User();
        //Fill user with data from db
        return user;
    }
}

Since the DbConnection is independent of object creation, DbConnection can be mocked to unit test any method in the class. An application written without testing in mind would be replete with code like the above. Code patterns like these are one of the biggest hurdles in testing legacy applications.

Next step is to eliminate the resistance to testing. This would mean all the infrastructure and libraries needed to carry out testing are set up and a reference is readily available to follow. Bunch test cases into categories like unit tests, tests that need a mocked object, tests that need a mocked database, tests that need a database seeded with data, tests that need a web server etc. Post this, implement one test case for each of these categories. This will serve a dual purpose, the setup would be ready for each category and a reference readily available to others to emulate.

One aspect that is usually neglected is the effect of testing on the product release cycle. As a result of testing, more code, dependencies, and infrastructure is introduced which needs to be maintained. Along with working on new features, writing tests for these also has to be taken into account. While refactoring, it is not just the code that has to be refactored, even the test cases have to be refactored. This is a tradeoff between time to market, and maintainability and reliability.

Testing is no longer a chore it used to be, testing tools and frameworks have grown by leaps and bounds. With the advent of docker, headless browsers, Selenium etc; testing is very much within reach of most of the teams provided the intention is there and effort is put in.

The way we account for the build versus buy decision varies. Some of the frivolous reasons for building in-house are NIH syndrome, hubris, and planning fallacy. We generally tend to overemphasize our expertise, knowledge, and capability which naturally lead to building internally. Also, we underestimate the amount of work involved in creating software, only once we get our feet wet does the reality set in. A very valid reason for building internally is the cost, but when accounting for cost, we usually overlook the hidden cost of developing software. Buying a software has an upfront monetary cost whereas by building internally we pay in the form of opportunity cost, talent cost, feature cost, etc.

Build versus buy arguments are reminiscent of qualitative speak - “This is not our core expertise, we should be concentrating on solving our business problems”; “This is going to cost us a bomb, let us build in-house”; “We should have had this yesterday, building in-house will cost us another 6 months”; “Will that external product be able to handle our scale”; “Can we trust them with our data”; etc. In most cases, build versus buy decisions are qualitative, it is not an easy exercise to quantify them.

When evaluating a product that is already out in the market versus building something similar, a cardinal mistake people commit is mapping features one to one. Even though having 100 different features looks rosy and attractive, usually we end up using only a select few. Instead of trying to match an external product feature to feature, scope out the features that you need or would probably use and then estimate the effort. Another is refinement. An external product will be refined and polished, but you may not need the same level of sophistication. For example, you might not need a web interface for the product; a terminal would work fine for your use case.

When faced with the build versus buy decision, asking the following help:

• Is this my core expertise or is it something I can let others do for me?
• What is the cost of getting this done externally versus hiring people to build this?
• How much control do I need over this, i.e. can I live with some error, downtime or opaqueness?
• Will I do a better job building this internally?
• Do I have the expertise needed to build this?
• Once I build this, will I be able to maintain and enhance?
• What is the opportunity cost of having this sometime in the future versus having it now?

Use the answers to the above as a beacon for the build versus buy decision.

In my professional life, I have heard a lot along these lines; Netflix does micro-services, let us also do that; Google and Facebook subject interview candidates to data structure and algorithm questions, let us adopt the same. Embracing something without understanding the context is a recipe for disaster. As a thought experiment, let us take microservices. Microservices evolved in tech organizations with complex products handled by multiple independent teams craving for autonomy and control without stepping on each other’s toes. Also, for microservices to succeed, you need to put in a lot of effort into alerting, monitoring, orchestration, and devops. Without these, microservices is a bomb waiting to explode.

When borrowing technology and processes from other places, one needs to put significant effort into understanding the context around which these evolved and also the required pre-conditions for these. Blind adoption usually leads to unmitigated disaster.

If you are doing anything with data, you need to know the schema of that data. What this boils down to is whether this schema is explicit or implicit. In relational databases, the schema is explicit and well documented whereas in NoSQL databases, the schema is implicit i.e it is maintained in code. All the goodies that you get with relational databases like integrity checks, constraints etc should be taken care of in code.

Products go through a multitude of changes, features get added and axed and in order to accommodate this, the schema has to change accordingly. In relational databases, carrying out schema changes on tables with humungous data is a daunting problem which requires considerable planning and effort. This problem is not present in NoSQL databases, but again, you end up managing this in code. In order to handle multiple versions of the schema, code gets littered with if else statements. This gets even more messy as this is spread across the code base in multiple places wherever data is being handled. In relational databases, you handle schema changes once and for all, in NoSQL databases, you continue to do it in code long after the change.

Multiple people work on code at different points in time, people join and leave teams but code lives on. This is when the maintenance problem starts raising its ugly head.

A big reason a lot of startups opt for NoSQL databases is due to the constant flux in these organisations. Requirements change frequently due to which a stable schema design becomes next to impossible. In this landscape, NoSQL databases look like God’s gift to mankind. If your company does not fall in this bucket and you find yourself tempted to use a NoSQL database, take a good hard look at the problem and ensure that it is worthy of a NoSQL database. There are problems which are better solved with a NoSQL database but these are far and few.

During our travel, we visited Ho Chi Minh City, Da Nang, Hoi An, Hue, Hanoi and Halong Bay. In all these places, infrastructure was amazing, almost on par with western countries. In Hanoi and Ho Chi Minh city, there are abundant parks and public spaces. They are very well maintained too.

There are walking streets everywhere. Most of these are regular streets which get converted to pedestrian only movement during the night. You will spot umpteen eating and drinking options in these streets. These walking streets are not as scandalous as the ones in Thailand.

Food is a big part of the culture, street food is abundant. It is a paradise for non vegetarians, not so much if you are a vegetarian. Noodle soup(Pho) and banh mi are two of the most popular delicacies. Banh mi means bread in Vietnamese, usually it is sold stuffed with veggies and/or meat.  Banh mi is a relic of the French past like our Pav(Vada Pav). There is a version of noodle soup called hot pot which is cooked right on your table which is delicious. People watching seems to be a big thing, lots of cafes lined with chairs facing the street. I am in love with Vietnamese coffee, it is black coffee with condensed milk, tastes amazing.

Vietnamese seem to have a thing for “The North Face”. Fake North Face products abundant on the streets. Giving them company are Nike, Superdry and Under Armour. Adidas, Reebok and Puma are conspicuously absent.

Traffic is chaotic. Two wheelers zipping past everywhere breaking all rules. You can rent two wheelers in all the cities, which we did. There is neither license check nor passport deposit, it is a honour based system. Toyota is everywhere. We checked with a taxi driver as to why Toyota seems to be omnipresent. Since there are so many two wheelers, scratches and skirmishes are common it seems. Hence Vietnamese prefer Toyota which is easy to maintain and spare parts are economically priced.

Communication is a challenge. Google translate was a life saver. It was much easier to just type in translate and show to people.

Vietnam is blessed with abundant water, water bodies everywhere, in cities as well as countryside. From the 60th floor of Lotte tower in Hanoi, we could spot innumerable lakes.

Museums and historical places scream jingoism. Glorification of Vietnamese struggle against the French and Americans seem a bit in your face.

Vietnam seems to be on the path to becoming an economic powerhouse. Signs are everywhere, from the burgeoning constructions to commercial towers competing for the most number of floors title.

Vietnamese are a friendly lot, they try their best to make things happen for you. We have hardly had bad experiences with people apart from a parking attendant in Hanoi who lost his cool due to the language barrier.

All hotels including our room in the cruise had slippers as part of standard accessories which was new. Also, in some of the public toilets, you had to change your footwear to slippers present there, a nice hack to keep the toilets clean.

Indian cinema and tv shows seem to be very popular. Our van driver was glued to Telugu movie scenes on his phone and a taxi driver told us he is a big fan of Hindi movies.

Internal air transport is economical, couple of low cost carrier options.

Souvenirs and trinkets are sold everywhere. Propaganda posters are really cool. Vietnamese seem to have a soft corner for Tintin, various posters and fridge magnets of “Tintin in Vietnam” or “Tintin in Saigon”.

Vietnamese script is English with accents. A French person created the script for Vietnamese. Original Vietnamese script is long lost it seems.

Vietnamese seem to love kids, we had three toddlers in our group, they were greeted with chocolates and souvenirs by strangers almost everywhere.

From a cultural perspective, Vietnam and India share a lot in common. Vietnam has a rich cultural past, dating back centuries. In fact, at one point of time, Hinduism was the dominant religion in Vietnam.

While coming back, we took Malindo air. Malindo air was a pleasant surprise, friendly stewards, ample leg room and in flight meals.

When we told people that we are going to Vietnam, all asked us why and we did not have an answer. It was well worth it though.

Phil Knight, founder of Nike, pens down the early days of Nike in this part memoir, part biography, part business book; “Shoe Dog”. The book starts with Phil Knight’s idea to sell Japanese made sneakers in his home country USA, his business school thesis supporting the same, his journey to Japan to ink the deal and the subsequent travel around the world post which he dedicates his life to building Nike and the struggles thereafter.

In today’s interconnected world where communication is instant and automation is the norm, reading about running a business when these were non-existent is fascinating. Nike during it’s founding days was closer to today’s new age startup than to any old school business as the company prioritized growth over security and cash in the bank. His anecdotes regarding the dearth of capital, constant tussle with the bank on rapid expansion, his bank asking him to scale down his ambition and focus on capital accumulation; give insights on the business mentality of the yesteryears.

Throughout the book, Phil Knight talks about his hands off management style. He styles his management on the below quote from Antoine de Saint-Exupery.

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

Profile of the Nike founding team is far-fetched from the general imagery of traditional old-school businessmen. They seem to be misfits who found a home at Nike, at least Phil Knight thinks so. No wonder the whole company rallied around doing iconoclastic things that many businesses would not even think of during those days.

All in all, the book is a great read. I was hoping to read more about Nike’s marketing strategy and how it evolved, but there is hardly any mention of this. Probably for another book I guess.

Reading Shoe Dog brought back fond memories of reading “Made in Japan” years back.

Whenever you ask a question on unit testing in a forum, there is always that one person whose only job is to point out what you are doing is not unit testing but integration testing. It is important to know the difference but it is more important to not lose sight of the goal, ensuring a reliable and a bug-free application. Also, you need to adopt a terminology that works for you and your team, rather than what purists think or say.

In absolute terms, if a test depends on anything that is not in your control, it is not a unit test. For example, if a method that you are testing uses a public function, a method from an included library, database or an external API, it is not a unit test but an integration test. For a test to qualify as a unit test, you need to mock all these dependencies and get them under your control, only then you can claim your test as a unit test. Now that we have the purists happy, let us move to a more practical worldview.

When a regular joe developer refers to a test as a unit test, what she means is, she is trying to test a functionality in a massive gnarly application that she thinks is a small independent unit. This unit might have some components that are not under her control. Instead of debating whether she is unit testing or integration testing, a better discussion is trying to figure out what is the intention of the test, what needs to be controlled/mocked and not. Helping her to figure this out and achieving this will add more value than debating whether a test is a unit test or integration test.

In my new workplace, I was assigned a brand new shiny MacBook. My first reaction was to ask for an Ubuntu laptop. My brain justified by giving several reasons, it is developer friendly, software that you run on a server runs as is in Ubuntu, etc. I was almost about to voice this opinion, but system two took over from system 1. It started asking questions along the lines of is this the reason you want an Ubuntu machine or are you just trying to avoid the unfamiliar? You have not used a Mac before and are you trying to run away from something new? The two debated for some time and settled on giving the Mac a chance. So far, the experience has been, and I am learning some cool things like using gestures for different actions, etc.

Whenever something new and unfamiliar comes across, the instinct for most of us is to fight it. Take a step back, analyze whether this is the primal part of the brain trying to fight against the unfamiliar or you have a valid reason not to.

I think this is a great way to ask for more data in this world of user generated content. Instead of asking to review a restaurant how about wording it as “Review this place so that others can discover great food”. Instead of asking people to rate your app how about saying “Help your friends discover the app on playstore, rate us”. It would be interesting to A/B test this and see the result.

Down the road, there was another used book store. Again got in and repeated the question. The person searched, said no, but he put a new copy of the book in my hand and said he can give me a 20% discount. I am sure he knew before hand that he did not have a used copy of the book, searching was a ruse. I made the purchase.

We tend to neglect the impact of putting in that little extra. Sometimes all it takes is a little extra effort for a great outcome.

What makes Jenkins such a gem for executing periodic jobs?

1. You get a great web front end which is comfortably accessible from the browser.

2. The front end gives you a complete history of the previous runs with detailed information like when did the last execution occur, how long it took, what was the output during this execution, historic trend of the execution time and other diagnostic information.

3. You can leverage the Jenkins plugin eco system and do some nifty things. For example, you can use log parser plugin to parse the execution output and alert if a specific format is found in the output. The great part here is that your job need not have alerting logic baked in, your job concentrates on doing what it does best, let Jenkins take care of the rest.

4. You can configure regular Jenkins build rules like alerting on execution failure, preventing subsequent executions if the current one fails, etc.

5. You can chain multiple jobs and the chain is very obvious thanks to the great Jenkins UI.

All this is great, but one problem I faced with Jenkins is that you cannot have a Job call itself recursively with a delay in between, you have to schedule the job execution using cron expression. The difference is subtle, but there are implications of this limitation which I will expound with an example. Let us say that I have a job which ideally should run every 15 minutes, but sometimes this job execution takes more than 15 minutes to complete, in that case what happens is, job executions queue up and fire successively one after the other. The way I want this scenario to pan out is, once the execution finishes, it should wait for 15 minutes before the next execution starts. I could not find a way to do this in Jenkins and hence selfie was born.

Selfie is a Jenkins build trigger plugin which lets a project to trigger itself after a configured delay. The plugin appears as one of the build triggers while configuring a new project.

This is my first attempt at writing a Jenkins plugin, pull requests and code reviews are more than welcome.

SQS is a queue, adheres to FIFO and promises at least once delivery.

Kinesis is a distributed stream processor. A simplistic and hand-wavy way to think of Kinesis is like one large log file; items that you write to the stream as lines in this log file. When you want to process the stream, you get a pointer to the log file. When you read a line from this log file, the pointer moves to the next line. Kinesis is stateless, as in, it does not maintain the pointer for you, it is up to your reading process to maintain this. What this means is that, say you are reading off a Kinesis stream, and your process goes down, when you bring the reader process up again, it will start processing from the start, not from the last line before the crash. There is no concept of popping items out of Kinesis, data is always there(expires after seven days), you manipulate the pointer to this data. Hence, if you want to reprocess the stream, you can replay i.e., you can start from the beginning and do the whole thing over and over again. AWS provides a client library for Kinesis which maintains the state for you. This client library uses dynamodb to persist the state.

This should give you a fair idea of when to use Kinesis and when to opt-in for SQS.

If you plan to carry out multiple deployments a day, you should not have downtime during these deployments. When your shiny new code is getting deployed, your end-users should be able to access the site; this is usually done by routing all your traffic through a load balancer. The end-user does not directly hit your application server, she hits the load balancer, and it is the load balancer’s responsibility to route traffic to the application servers. In addition to this being the recommended architecture for server-side software, it gives you the agility to deploy code without having to worry about downtime. You take a server out of the load balancer, deploy your code on it, add it back to the load balancer, and do the same with the other servers that are part of the group. Two of the most popular load balancers out there, Nginx and HAProxy, allow you to do this dynamically; while the load balancer is up and running, you can add and remove back end servers. If you are running on AWS, Elastic Load Balancer lets you do this.

Also, your deployments should be as simple as possible; even a monkey should be able to deploy your code to production. More the complicated it is to deploy software, less the enthusiasm of developers to do it. Using a continuous integration tool like Jenkins helps to make this a painless process.

Enable a kill switch for all new features. Your app should have the ability to turn on and off features in seconds; this gives you the power to turn off a feature if you detect any problems with it in the early days.

Also, gradually releasing features is a good idea; this lets you whet performance and other issues on a small scale before it becomes a site-wide problem. Release to 1% of your users and then slowly ramp up to 100%, keeping a close eye on the feature all the time.

If you are working on a feature, you do not have to wait for feature completion to release. As and when you finish off logical steps, keep releasing. Your end-users might not see the feature yet, but this helps you to get away from the situation of one big bang release and everything going down. These incremental releases help to detect bugs early and build confidence for the final version.

Create alerts for all your critical performance and business metrics. After deployment, if any of these metrics go awry, you get an alert, and you can set things right. In addition to alerting, having the ability to graph these is tremendous. Post a deployment; you can check your dashboard to see whether the deployment has hurt your response time and critical business metrics; this adds to your confidence to do multiple deployments without having to worry about your new code degrading performance and adversely affecting business.

These are some of the simple tips that help you deploy code to production regularly. I have not touched on more advanced topics like automated testing, integration testing, and automated provisioning.

Photo by Giftpundits.com from Pexels

I was stuck in a traffic jam caused due to BWSSB(Bangalore Water Supply and Sewerage Board) closing a section of the road to fix a sewer. This was on an afternoon on a main road in Bangalore and no wonder, there was a traffic pile up for close to 3 kilometers. I cursed myself for taking that road instead of some other alternate route. I did mentally spew venom at the people digging the road, but it never occurred to me to hold the authorities accountable for this and expecting better from them.

The curious case in both the incidents is the expectation. Even though the right thing to do when you find something that is not yours is to return it to the owner, people nowadays expect that not to happen. It is the responsibility of the government to execute tasks in such a manner that it causes least annoyance to the citizens but we no longer expect that from our administration. If things do happen the right way, we are mesmerised and think of ourselves as lucky, it does not occur to us that this is how it should be. This is not a good sign because when expectations hit rock bottom, it becomes the norm and from there on, it is a vicious downward spiral.

The same is true for organizations as well. Holding people to higher expectations and making them accountable is the key to success. If in an organization you are happy that your team members are not taking impromptu leaves, there is something really rotten. There is nothing stupendous about people not taking unplanned leaves, this should be the norm and if someone following the norm is an aberration in your organization, it is a warning sign that there is something deeply wrong which needs immediate fixing.

One of the tests for any organization to see whether it is on the path to greatness would be the expectation test. Mentally go through your expectations from employees/company and see whether these are really higher expectations or something that should have been the norm. If these are expectations that breach the bar, well and good, if not, do a reality check and fix them.

I am privy to startup lores where everyone writes components in the language of their choosing. If today’s fad is Node, then the developer uses Node and if tomorrow hacker news has a post touting the next most magnificent language, the developer switches to that. We are a startup; we do not have a QA and pre-production environment for testing; we test code in production. We are only a couple of people. Hence we do not need version control. Since I am a single guy who manages servers, configuration files are not in version control; my brain is my GIT repository.

Being part of a startup is an opportunity to move fast without being burdened by the bureaucracies and politics of big organizations. Say, for example, you find an employee kicking ass. As a manager/lead in a startup, you can give that person a raise by having a quick chat with the CEO. In a big organization, you have to make a case for that. You need to present it to a committee, justify why someone should be given an out of turn raise. There are a lengthy process and guidelines for these sort of things. It is also an opportunity for you to trust people and call them out when you see something odd without introducing a process around it. If you see someone working from home on a very regular basis, and it is hampering work, instead of floating a blanket rule saying work from home is prohibited, thereby inconveniencing other employees who use it judiciously, you talk to the person and see what is the problem that person is facing and try to solve it. This is difficult to pull off in a big organization that has thousands of employees but doable in a startup.

A startup lets you create an atmosphere based on trust rather than draconian processes. Let us take a technology choice, for example. Big organizations have lots of rules and regulations are technology choices, buying licenses, etc. The reason is that individuals are not responsible enough, so let us decide by committee. In a startup, you do not need to introduce a process for this. You can trust your employees to make the right technology choices as long as they do not go bonkers with it. If something goes wrong with a technology choice, you can talk to that person and figure out where he went wrong instead of mandating policy for all employees from then on. Individual responsibility trumps rules any day, it is challenging to implement and follow in a large company but can be easily pulled off in a small organization.

Next time when you use the phrase “We are a startup,” think hard whether you are trying to mask inefficiencies in the guise of being a startup.

Photo by Clark Tibbs on Unsplash

Setting up an office might not seem a daunting task, but trust me, it is. A previous company that I was part of had opted not to set up its own office and worked out of an incubation center. I always used to wonder— why not move into a proper office space? After personally going through the hassles of setting up and maintaining an office, I appreciate their decision.

If you are a small company, try to avoid moving into an office space of your own as much as possible. Do it only if you should and must. There are other options available, like co-working spaces, incubation centers, etc. Despite this, if you want to get into the adventure of setting up an office, read on.

The first step of the process is prospecting potential office spaces. Office spaces in tech parks are expensive and come with long leases, but they are the most organized. If you are reading this, you probably do not fall into the category of people who can afford office space in a tech park.

Some of the things to keep in mind while prospecting:

• Keep enough leeway for growth for at least a year.
• Is there enough parking space? If there is a shortage of parking in the building and the building is located in a busy locality or road, your employees will curse you. You do not want people to be grumpy in the morning before they start their work. On the other hand, if the office space is in a residential area, this should not be a problem.
• Does the building have power back up?
• Do all the major internet service providers(ISP) serve the area? Confirm this before you sign on the dotted line as some areas have restrictions against digging etc., due to which ISPs refuse internet connectivity.

Once you zero in on the place, the second step of the process starts. Things to keep in mind:

• Interiors and furniture.
• Networking. Do you go for wi-fi or ethernet? If ethernet, you have to plan your layout. If wi-fi, you might not be able to get away with a home router as the signal strength may not be strong enough throughout the office, or the router might not be able to accommodate all the people in the office. In all probability, you need an industrial-strength router/switch and repeaters.
• Drinking water, do you buy a water filter or go for water cans?
• ACs, fans, etc. Do keep in mind that you do not have to buy these, you can also rent them.
• Cleaning and maintenance—best to outsource this to an external agency. Keep phone numbers of electricians, plumbers, etc. handy; you will need them all the time.
• Office keys—you might have to have a security guard 24/7 because you will have people coming in and leaving at different times, and coordinating the office key becomes a headache.
• Also, it is best to hire an office manager who can coordinate all these tasks for you. These tasks eat into your precious time, and I am sure your time is not well spent on these mundane yet essential tasks. If you are working out of an incubation center or co-working space, you get most of the above as part of the package. As I said before, try to push off moving into an office of your own as much as possible. Do it only if you must.

Photo by Georgie Cobbs on Unsplash

It appears that validation of configuration properties is performed in the ConsumerConfig and ProducerConfig constructors. This is generally bad practice as it couples object construction and validation. It also makes it difficult to mock these objects in unit tests.

Ideally validation of the configuration properties should be separated from object construction and initiated by those that rely/use these config objects.

http://misko.hevery.com/code-reviewers-guide/flaw-constructor-does-real-work/

It links to an article by Misko Hevery on writing testable code. If you have not read posts by Misko Hevery, I urge you to. For an open source project like Kafka, it might make sense(Jay Kreps, the person behind Kafka does not agree with the bug as visible in the comment) to follow all the guidelines of writing testable code but for your project it might not. If you are a small company with a two or three person team, do not blindly follow practices because someone on the internet says so.

Follow rules and guidelines only if it helps you to make your code more secure, performant, easier to maintain etc, do not ape guidelines without understanding why they are laid out in the first place. To go back in history, checked exceptions were all the rage in the Java land a couple of years ago, but these days, after frameworks like Spring sprang up, people look down upon checked exceptions. Same with TDD. TDD was expounded as the next best thing after sliced bread, but now programmers are raising their doubts about TDD.

A lot of times, mocking objects, interfaces etc takes more work than writing the actual functionality. In many projects, there might not be an ROI in writing/maintaining this elaborate test framework/infrastructure. It is true that injecting dependencies into an object makes it easier to test, but it also comes with the downside of having to take care of injecting dependencies every time you create that object. If you are injecting dependencies by hand, object creation becomes an elaborate exercise each time, else you have to delegate this to some framework like Spring, Guice etc, now your project is bloated. Maybe you should side step dependency injection and create the object with the dependencies inside it.

The situations and background under which these rules/guidelines crop up might be radically different than the one in your organisation/team. Taking the call of what to follow and what not to is more of an art than science. Your instincts, past experiences etc help you in formulating them.

Most of the time, I get a feeling of disenchantment from people about their occupation. To put it bluntly, it is fashionable to be cynical about one’s job and complain about it. A lot of people feel that they want to do something different but most of them do not know what is this different path they want to take. And to fuel this fire, you have hundreds of books and blog posts which extol you to give up your job and follow your passion.

You go on a scuba diving holiday and all of a sudden you want to be a professional scuba diver. You do a couple of hikes and a la, you aspire to be a travel writer. You have started to cycle to office and you have this strong urge to start a bicycle touring company. You purchased this new DSLR and now your single aim in life is to be a wildlife photographer. An article about a so and so who gave up his cushy corporate job to start a local bike store or became a wildlife photographer reinforces these thoughts.

Even though these thoughts are romantic and warm the cockles of our heart, the reality is a bit different. When a hobby becomes a job, the fun aspect of the hobby goes out of the window and the boring part kicks in. When you do something repetitively, the novelty wears off. Some paragraphs from this article on becoming a travel writer:

But while free trips, global travel and your name in print sound glamorous, there are down sides. It’s hard work, hugely competitive and – unless you are the second Bryson – you won’t earn much. Roving overseas with a notebook, a deadline and a pack of other journalists can also take the fun out of travelling altogether. Not put off? Read on to find out how you can get this dream job.

Below is a paragraph from an article by a scuba diving instructor:

The job duties of an instructor aren’t what most newcomers expect, either. And to many it comes as a sad surprise. The reality of being an instructor at least full-time is that teaching is only a small portion of what you’ll do. Mainland-based instructors often work 40 hours a week at a dive store counseling customers or repairing equipment, then teach one or two classes a week on top of that. That means 60-hour weeks are commonplace. Think it’s easier at a resort? The norm for resort-based instructors is six days a week, and during busy periods seven days isn’t uncommon. Here, too, teaching is only a minor part of the job, but an instructor ticket is essential, if for no other reason than to get a permit to work in a foreign country. Many have left the industry disappointed that their dream of spending their days primarily as teachers never materialized.

And to top it all, you have to read this from a person who started his own micro brewery.

A lot of times, we misunderstand novelty for passion. You have a sedentary desk job and travelling once in a while looks like life’s calling, but the question to ask is, would it still be your life’s calling if you had to do it 24 by 7 while earning a substantially lower income? Would you not be more happy earning a good salary, enjoying the material comforts that your day to day job provides and travelling once in a while to break the routine?

In most of these offshoot jobs, the number of slots where you can be comfortable with the income is limited and the aspirants for these slots are unlimited. Also, in a majority of these, you have to be at the pinnacle to earn really well. We all love to think that if we are talented, success naturally follows, but I call this specious. There is more to success than just talent, success is mostly a factor of being in the right place at the right time and luck, not to say that talent and other factors do not help, but it is for sure not only talent. Thinking fast and slow by Daniel Kahneman expounds a bit on this.

The melancholy that sets in with a day to day job is our own making. If you are really interested in spicing up your work, there are innumerable ways to do it, it is just that effecting change is boring while cribbing about it is romantic.

As programmers, a lot of times, we blindly work on product requirements without questioning the intention behind them. As it was in our case, the person who gave the Excel requirement did not have any idea as to the difference in the amount of work one would have to put in to create an Excel report versus a CSV, nor the technical debt of the two approaches. From his perspective, it was all the same.

When a requirement comes to you, take a step back and go through them, see if there are chances to simplify. If some of the requirements sound absurd to you, tell it to the stake holder, if some of them might take a long time to implement, talk to him about this. The person providing the requirement has absolutely no idea that some of these features might be technically difficult to pull off or take a long time to implement or there is a better alternative that might not meet the criteria strictly but will not take eons to implement. Bias towards action is good, but a trigger happy one hurts everyone.

Whenever architecting components, devote ample amount of time to failure scenarios. Let us take the case of a piece of code which interacts with an external, third party API. What are the questions you should be asking when designing this component? What happens if the API suddenly stops responding one day? Can I hold my main thread hostage to the API response time? What happens if the API takes eons to respond? In case there is an exception, am I logging enough data to debug? If there are performance issues, do I have enough diagnostic data? Diagnostic data might be in terms of graphing the API response time, no of times the code path was executed, etc. Do I need to send out an alert when something goes wrong? All these question revolve around failure handling. These questions should be second nature to you as a software engineer.

I have seen a tendency among developers to devote inordinate amount of time in making their code adhere to the latest programming fad, trying to use the best possible library etc, but not to failure scenarios. Logging data might not be as sexy as debating which design pattern to use, but once things break, logs are your only friend. Next time when you are furiously pounding on the key board, take a step back and ask these questions. In the future, the developer who maintains the code that you wrote today, will thank you for doing this.

A road is something that transcends caste, religious, and socioeconomic boundaries. A road does not discriminate between a rich man driving his BMW or a poor milk seller on his bicycle, a pandit riding his Luna or a moulvi plying his scooter. I do not for a second doubt the ingenuity of our so-called secular politicians to come up with a statement on the lines of Minorities have the first right to our roads; but until that happens, for all purposes, we can take rest in the fact that a road is a great unifier of all sections of the society.

How do roads benefit the poor?

Connectivity makes transportation efficient, thus reducing the cost; this is an indirect benefit for all. Let us take a specific case of how roads will make the life of an impoverished auto-rickshaw driver better. If an auto driver plies on potholed roads all day long, it takes a toll on his auto; this will directly reflect in the efficiency of the auto as well as the money he has to spend on maintenance, not to mention the lost opportunity to make money by ferrying more customers due to the reduced travel time. All in all, an auto driver has to gain a lot with smooth roads. Instead of working on these, our government is hell-bent on extending the economic black hole of NREGA to urban poor, and our media is a silent spectator to this theater of absurd.

Photo by Fancycrave.com from Pexels

The other day, people at the gym were opening up the machines and lubricating and tuning them. This guy was taking a keen interest in the job and helping them out. This sparked a wave of thoughts in my mind where I put myself in his shoes and thought how I could move up the value chain if I were him. I would learn how to tune and lubricate these machines properly. If the norm is to do this activity once a month, I would volunteer to do this every week. Everyone wins, the machines are smoother to be used by the members on a regular basis and I get a good opportunity to hone by skills. Once I am adept at doing this, I would go to the other gyms and offer my services to them. There are a lot many gyms in Indiranagar that I know of and probably some more that I do not know of. I am sure all these people are looking for someone who knows his way around the equipment because such people are usually scarce these days. Once I have a good number of gyms under my belt, I would probably bring more people from my native village and teach them the job so that I can build a team and expand my operations throughout Bangalore. In this way, I am not only bettering my life, but also providing employment opportunities to the people of my village. Once I have Bangalore under my belt, you know the drift, expand more etc.

This is a very hypothetical scenario, but a belief that I very much hold on to, if you try hard enough, there is always a way to move up the value chain.

Another point of note is consumer tech driving futuristic technology. Take google’s driverless cars for example, where is the money coming from? From consumer tech like search. Where did Elon Musk get his money from to bootstrap Tesla motors? Again consumer tech like paypal etc. As of today, I see consumer tech driving research and development more than defense and space which in a way is good for us as consumers because we get a taste of the shiny new thing without having to wait for decades for it to be made mainstream.

1. Go lang is strongly and statically typed. This means that a lot of mistakes that could potentially cause your code to blow up in production would be caught at compile time. Apart from this, if the language crosses a critical mass of adoption, someone will come up with an IDE that can possibly match Eclipse, IntelliJ, et al. Also, one of the principles behind the language’s design is to aid tooling which means that a lot of tools could possibly crop up which would help to make code more secure, performant etc.

2. Syntax of Go lang is not revolutionary. I consider this a virtue. I am strongly of the opinion that if a language has to gain mass adoption, it’s syntax should be very close to the prevalent languages. Go lang does not deviate much from the Cish syntax but has subtle improvements which improves programmer productivity.

What makes Java a good programming language for the enterprise? Syntax of Java is very close to C which means that you could possibly train a lot of the computer science graduates out there to code in Java. Try Lisp with an average Joe programmer and you will know what I am talking about. With the people supply problem solved let us move on to other factors. Enterprises gravitate towards stability, security and viewing programmers as replaceable components of a machine. Java gives enough constructs to prevent a reasonably sane headed person from shooting themselves in the foot. Static/strong typing, code analysis tools and IDEs go a long way in helping with this. Without tearing your hairs out, try working with a code base in a dynamic language designed by an architect and then handed to a team for coding and then passed on to a testing team and then shifted over to an offshore team for maintenance. And to top it all, you do not have Eclipse or IntelliJ to help you with this mess.

Go lang has all the pluses of Java minus the verbosity. Of course it has a lot of other features which you can read about in the website.

Do you have version control? Do you back up data(server data, db data, etc)? Can you build using a single command or prosaically, do you have build scripts? Can you deploy code to production in a single click/command? Do you have different staging environments like dev, qa, pre production etc? Do you have a script to bootstrap the database so that any one can run the app locally? Are your configuration files in version control? Do you use a system provisioning tool? Do you use a database versioning tool? Are your system and business metrics graphed? Do you have alerting systems in place when your business or system metrics go out of whack? Do you have a document listing all the ips/machines in use, external software services used along with their user name and password? Do you have unit and integration testing suites? Do you use a task/project management tool? Does your team swear by code review?

Case in point, Arvind Kejriwal and AAP. Folklores were weaved around AK’s simplicity and his party’s candidates, on a daily basis we were bombarded with their spartan lifestyle stories. How come it occurred to no one to ask the questions that really mattered? Is being a simpleton really enough to be an effective administrator? How well did these people perform in their previous jobs? What extraordinary achievements did they unlock in their past endeavors? Do they have the skills to govern and deliver? Instead of asking these pertinent questions we were wallowing in their hallowed simplicity.

A simpleton(aam admi) should not try to be a leader. Leadership is not for the faint hearted, it is a game of thrones. A leader is someone who tries to move a collective group of people towards a single goal and when there is a group of people there will always be mismatches, heart burns and politics. An effective leader should know how to deal with this and make the right calls most of the time, an aam admi is not suited for this role. If someone plays this role well, he is not an aam admi because not everyone is born a leader nor has the leadership skills.

Yes, leading a simplistic lifestyle trumps a hedonistic one governed by avarice any day but put the full stop there, it does not translate into one being an able administrator or being at the top of one’s job. I would any day prefer a Ferrari driving, Armani wearing, Gucci slipping chief minister who has a vision and track record than a slipper wearing one who has absolutely no idea how to govern and deliver.

I feel this move smacks of licence raj. First of all, government has absolutely no role in regulating non essential private businesses over what/how much they charge. This throttles innovation and hinders entrepreneurship. Government should foster free market rather than be the antithesis of it. If the general public decide that movie ticket prices are exorbitant, either they will stop watching movies at multiplexes or some enterprising chap will come up with a plan to provide them this service at a sweet price point. By introducing artificial constraints, government is strangling the entrepreneurial spirit and hurling us back to the hey days of socialism where having a phone connection was considered a privilege. More the constraints in a market, lesser the participants, fewer innovations, lesser economic activity, fewer jobs and capital movement.

Secondly, does it look like our government has less tasks on it’s plate that it wants to add one more, trivial, non impacting regulation to it’s todo list? What do you think is more important, providing basic education, health and security to it’s people or investing time and energy in regulating ticket prices in multiplexes? Would not the government be more wise in exerting it’s authority on bettering our public health system or libraries?

As soon as government starts meddling in the affairs of a private business, it is not those who excel at innovation that triumph but people who kick ass in playing politics and boot licking. With all it’s flaws, a free market economy trumps a regulated one any day.

Take a peek at the acquisitions carried out by Facebook. Can you spot a pattern? A common recurring theme around these companies is photo sharing, image recognition, mobile and analytics. As mobile devices/digital cameras become ubiquitous, internet connectivity becomes omnipresent, people are going to upload snaps/videos in ever increasing numbers. Photos/videos reveal a lot about you without you consciously intending to. A stream of photos that you upload on a regular basis is as good as a life journal that you are not explicitly writing. How about mining this information to show relevant ads?

Your photo reveals that you wear spectacles, show eye wear ads to you. Your photo stream shows that you regularly take overseas vacation, you are a hot target for the travel industry. Your photos suggest that you have an entry level car and you tell Facebook that you got a new job, time to show the next segment car ads in your wall? In most of the photos you wear denim and t-shirt, maybe Levi’s would be very interested in you.

Think about it, the amount of information a stream of photo gives out about you is endless. This is a gold mine for companies that want to tailor ads for you.

Path of least resistance is a tenant everyone in a product company should live by. Take for example a signup form. Given a sign up form with two fields versus one that takes gazillion of fields, which do you think will appeal more to your users? Now you may say my product needs a phone number field to send sms. Ask yourself this, do I need to take this field during sign up or can I ask the user to enter their phone number during the course of usage? When you sign up users, do you really need the confirm password field? Or, for that matter, do you really need users to sign up to use the product? A few days back, I was carrying out comparative analysis of a product on a couple of e-com sites. I had different sites open, one of them allowed me to sign up using gmail while the other required a custom sign up. I did not even bother with the one that asked for the custom sign up and went on with the one that allowed me to login through gmail.

Can a product compete in a deeply entrenched market with the only differentiator being it’s simplicity? I would say so. Take a look at the mobile instant messenger market.The new kids on the block have successfully waged a battle against the Goliaths and have emerged on the winning side. They have done this by being laser focused on simplicity.

When we are part of a product, use it day in and day out, we become so entwined with it that we can use it with our eyes closed. But our customers are not on the same boat, some usage patterns which might seem obvious to us might feel like navigating through a maze to our customers. It is extremely important that we keep reminding ourselves of this each and every day.

1. A lot of Indian e-com startups where founded by IITians and just this IIT tag was used by them to secure irrational funding. AAP supporters do not shy away from touting Arvind Kejriwal(Ak)’s IIT credential.

2. Majority of Indian e-com startups secured funding from foreign VCs. Recently, there have been allegations that this was a way for foreign players to gain foothold in the Indian retail scene which was insulated from FDI. AK is alleged to have got financial boost from Ford Foundation. Questions have been raised by rival political parties around this.

3. Indian e-com stood on the shoulders of urban middle class for it’s growth. AAP’s core supporters tend to be from this demography.

4. There have been claims of Indian e-com founders raking in exorbitant salaries/benefits. AAP’s SUV cavalcade bears some resemblance.

5. Indian e-com was the darling child of everyone a few years back. As of today, AAP is the poster child of Indian/international media.

6. People who called the bluff on Indian e-com a few years back were ridiculed. The same treatment is meted out today to anyone who does not believe that AK is the second coming of Jesus Christ.

7. There have been some nasty internal feuds in Indian e-com and AAP is not left behind in this.

8. The most poignant of all, Indian e-com propelled itself with freebies(free delivery, discounts, selling below MRP, buy today pay later, etc etc). AAP rode to power with free/low priced electricity and water(There might be more freebies, I have not read their manifesto).

Indian e-com companies which defied the basic laws of economics have been either forced to shutdown or secure more rounds of funding or to course correct their former splurging ways. In the case of AAP, it is the tax paying, law abiding citizen who will have to bear the burnt of their irrational populism. Indian e-com made a fool out of it’s investors, AAP is making a fool out of the gullible Indian citizens.

In small organisations without any sort of processes, most of the requirements usually come on the fly, they are briefly discussed through mails, the dev team gets into action, once they are done and testing is through, it gets pushed to production. Then starts the barrage of mails asking, How do we prevent fraud? I need these reports for the board, how do I get them? CS does not even know this feature is live, how did that happen? I need an admin panel to do so and so, why is it not there already? How well is this feature performing, how do I check that? Are we measuring so and so? etc etc.

I am not saying this is always the case, but most of the time this is how it plays out. Why does this happen? Usually there is some sort of disconnect between the various teams as everyone is busy doing their own shit due to lack of time/resources and abundance of problems to solve. Also, there is this constant pressure to throw things out as soon as possible in the name of existential threat/out pacing competition that details not in your face tend to be over looked. For example, a dev may not really think about an admin panel as during development he/she can play around with the database or some config file and get around the problem that an admin panel solves for a non technical person.

How do we fix this? The simplest and the least resistance way to fix this is a project management tool. I have expounded on this before, but let us look at it again in light of this problem. Avoid discussing features over email and do this through a project management tool so that everyone involved has a 360 degree view of the problem. Also, have a template for each task with sub tasks, for example each task can have the following sub tasks(not an exhaustive list and in no particular order): 1. Admin panel. 2. CS briefing. 3. Executive reports. 4. Code review. 5. Testing scenarios. 6. Analytic requirements. 7. Fraud.

I am sure you can add to this list based on your domain but the bottom line is have a template with the obvious listed out. How this helps is, when we have things on our face, we tend to actively think about them. Also, the unintended side effect of this is self documentation for coming generations and preventing blame game later on :). How many times have we heard this typical conversation, one guy says “I had told you this, why was this not done?” The other retorts, “We never discussed this, what crap are you talking man”(Exaggerated for dramatic effect).

Other ways of solving this problem is what big organisations do, having a strict spec review and sign off process, holding regular meetings where all the stake holders are present. I prefer a much more informal process of self documentation and processes which give you the feel of lack of it, your mileage may vary.

Let us take a hypothetical manager in an organization who has taken upon herself to deliver a feature. The feature deadline is tomorrow. She is furiously working on it today, and suddenly the customer support person calls him and says there is an issue that needs an immediate resolution. Now, this person has no other option than to discontinue the task she is currently working on, look at the problem, figure out who in her team can work on it, assign it to the appropriate person and let the customer support in charge know that so and so is looking into it.

Now with the issue at bay, she happily rolls up her sleeves and starts typing the next line of code when the product manager walks up to her and says - this feature that we released last week, the company board is asking for numbers, can we assign someone to work on it, I need the reports like yesterday. The manager furrows her eyebrows, does a mental calculation of all the tasks going on, which can be de-prioritized, who can be pulled out of their current work and pick this ad-hoc task or should I say no to the product manager right now and bargain for some additional time. With that out of the way, she hopes at least now she can get back to her task.

She is about to begin when a team member walks up to her and says we need to talk about this design trade-off of using a new database table for this feature versus accommodating it in the existing scheme of things. By the time this is over, it is already mid-afternoon and time to take stock of all the current tasks, clear roadblocks and plan for the future. There goes the poor manager’s day and no progress on the feature which she was supposed to deliver the next day.

Is the answer to this not doing any hands-on work and just delegating? No. Take on tasks that do not have a tight deadline, which is good to have but always keeps getting pushed out due to some higher priority user-facing or revenue-generating feature. For example, right now you do not have the means to version control your database changes, you are doing it through some ad hoc ant tasks. Database versioning is an excellent thing to have, but it is not needed like tomorrow and has no visible revenue/user impact, so it is ok if you plan to release it tomorrow but gets pushed out by a couple of days.

While choosing these sort of tasks, try to work on those that touch different components of the system, gives you a good mental model of the whole product, familiarizes you with the day to day issues faced by your team members, gives you a good understanding of the inner workings of the critical code paths in the product. If you do this, you have hit two birds with one stone, you are no longer just a delegator of tasks and to top it up, you have an excellent understanding of your product and can empathize with the team.

Photo by Bethany Legg on Unsplash

Our media and intellectuals have trapped us in a state of inaction. Daily, we hear that our politicians and government are useless, and there’s no hope. Even when a politician does something good, the media focuses on criticisms instead of praise. This creates apathy among educated people who don’t vote, which means they can’t influence policies. Consequently, decision-making gets hijacked by short-term thinking, harming the country. Those who vote for these policies often lack the education to see that short-term gains can lead to long-term pain.

Occasionally, a politician with a strong, clean track record emerges. He uses technology to improve lives, ensure electricity and good roads, and share a vision based on ideas, not just talk. Yet, we still ask, “What can this guy do?”

Most educated people want quick fixes. They favor revolutionary ideas that excite them rather than steady, reliable improvements. This leads them to support movements like Anna Hazare or Arvind Kejriwal. In contrast, poorer people often hold onto hope for change and improvement, while the wealthy educated classes express skepticism that anything will ever change.

Why is this? Many in today’s yuppie generation have never faced poverty. After college, they land high-paying jobs and enjoy comforts their parents worked hard for years to provide. They’ve never waited for water at public taps at dawn or skipped bus stops to save a few coins. The poor still face these challenges. When a leader with vision appears, they rally behind him, while the wealthy educated remain indifferent, romanticizing poverty from their comfortable lives.

Even if Narendra Modi comes to power—if is key—change won’t come overnight. It’s hard to fix something deeply flawed quickly. But a strong leader’s confidence and effective management can inspire hope. Their energy can motivate young people, making the future look promising.

I feel there is a better way to do appraisals. Let me outline it. Regular one on ones should be mandated by the organisation with whomever you are reporting to. During these one on ones, the manager/lead jots down what went well during the time, how did you contribute to this, what was your role, did you go out of your way in putting the task on a fast track, did you do something that not only benefits this particular project/task but the organisation as a whole, did you help anyone when they were stuck etc etc. These questions can be mapped to the vision/mission statement/OKRs/whatever of the org. Also, the manager asks each of the team member as to did anyone else in the team help them in any significant manner. This is peer feedback on a regular basis.

The advantage with this approach is, when it is appraisal time, you do not have to fill a lengthy form and scratch your head as to what you did during this period or fill a peer review form because your manager already has it. Due to the weekly interactions and note taking, the manager has a very good idea of each employee’s strength weakness etc and can outline it in a coherent manner to the upper management and make a clear case for hike/promotion/demotion/pink slip etc.

I feel this approach takes the pain out of employees and makes the appraisals more meaningful rather than some namesake process.

Let us answer the most important question, why a collaboration tool is needed when we have email? I am sorry to burst the bubble but email is not a team collaboration tool, email is an one to one collaboration tool. What are the shortcomings of email which prevents us from using it as a collaboration tool? 1. Gmail(de facto email :)) took a big leap forward with threaded replies but a task manager should have more levels of hierarchy. This is where asana shines for me, I love the infinite task hierarchy feature of asana, this is something that I really missed in trello. 2. Using email, it is a herculean effort to have a bird’s eye view of all the tasks currently in progress, about to end or scheduled to start. 3. A collaboration tool makes it really easy for you to track the progress of a task and see the wrenches in the system that prevent it from zooming ahead. This enables people to jump in and course correct things. 4. It is a single point of contact for you to figure out who is working on what and to whom you can assign something next. There is this really burning customer service(CS) issue, oh ok, Ram worked on this feature, let me see what he is upto today. Open the collaboration tool, click on Ram’s name and see when he is pushing the feature he is working on to QA. Wow, he is pushing it today itself, so, as soon as he is done, he can take a dig at this. Create a task for this CS issue and assign it to Ram. Also, make the CS agent the follower of this task so that he knows Ram will be on it soon and he can track the progress. 5. Someone higher up in the org wants to know whether the feature that was discussed a couple of weeks back is on track. You can be out of the loop and just ask the person to peek at the collaboration tool and he should have his answer. 6. Finally, the biggest reason for using a collaboration tool, documentation. How many times have you seen this scenario being played in organizations? There is a bug in some legacy feature, but you have no clue why the feature was designed like that in the first place. You ask your manager/lead as to why it was designed the way it is and he gives you a blank stare and says, let me dig up my email, we discussed about this. He comes back saying he is not able to retrieve the particular email thread. If only the team was smart enough to use a collaboration tool, this scenario would have turned out totally different. Using a collaboration tool you can aggregate all the discussions and decision points that went into a feature and you can return to it when in doubt.

Now that the reasons for using a collaboration tool are out of the way, let us come back to the question of making it a part of the workflow. As the article correctly points out, people use these tools for a week and then go back to email. Why does this happen? It is not because email is an awesome tool or that there is no better thing out there other than email. It is just the power of habit and to put it mildly, laziness. We have been so habituated to email that it has become second nature to use it. Also, individuals in the team do not see the reason for using a collaboration tool because an individual contributor of the team does not face the pain of task allocation etc which a collaboration tool primarily addresses. Since that person is not exposed to these problems, he does not see a merit in wasting his time on something new.

So, how do you make a collaboration tool part of your team? Education. Educate your team members as to the reason why this is being done. Do not just tell them that this is asana, going forward you have to use it. No, that is not the way to do things in a team. Tell them that these and these are the problems that you are facing and a collaboration tool will greatly alleviate these problems. Usually, when you are honest and people realize that you are trying to solve a real problem, they go out of the way to help you out. This softer approach has to go hand in hand with the draconian enforcement of the tool in the initial days. People always relapse and it is your duty as the lead/head to make sure that they start on it once again. When you see communication shifting back to email, let the team strongly know that this is not ok, this has to be done through the collaboration tool. A couple of these cycles later, you will have the whole team on board.

As said before, we have been on asana for about 6 or so months now and are very happy with it. Give it a shot, there is a free user tier which should suffice for most small organizations.

Instead of getting it’s feet wet with building UIs, which usually are crappy, government should concentrate on trying to build robust apis/backends and let the citizens exploit it to their hearts content. There is a huge demand for any sort of utility api, I can vouch for this after being in charge of FreeCharge for an year now. We are rummaging for robust apis for utility services, while the whole spectrum is filled with crappy government websites.

Take for example how amazing it would be if IRCTC was an api provider/consumer instead of the website it is today. Think of all the cool apps/websites people would have built around this api. How many travel sites would have mushroomed around this concept? Along with spiking curiosity and entrepreneurship in people, this would also have lead to job creation.

This does not apply only to governments, but in general to anyone. Open up the gates, let people clamor to build UIs over your services. There are two major advantages in doing this, one, you get out of the distribution problem, two, you broaden the reach of your services. A classic case in point, twitter in it’s early days.

Building robust user interfaces may sound an easy job, but being in the consumer web for the better part of my career, I can confidently say that it is not as easy as it sounds. Does the line “If you build it, they will come” ring any bell?

I am sure anyone who has been part of the tech culture has witnessed this phenomenon. There is nothing inherently bad about using/wanting to use the latest/newest tool/language/framework. In fact, it should be encouraged, it is a clear sign that your team is not happy with the status quo, they wants new challenges, they are ready to come out of the comfort zone and tread into new uncharted territories. But, as usual, it is not as black and white as we want it to be.

If you are starting from scratch, it makes some sense to go with the latest, because anything that is new is a product of someone’s frustration with the existing and trying to better it. Someone writing something new has the advantage that they can learn from the experience of current, better all the issues that they faced with the current and not repeat the same mistakes that the current did. But, at the same time, there is a flip side to it, an existing product usually has a strong community behind it, the product has matured through various iterations and solved all the warts that would have been there when it was the new kid on the block. So, when using an incumbent, you are standing on the shoulders of giants and taking advantage of the huge community behind it. On the other hand, something new has the advantage that they can be scrappy, respond to changes faster than an incumbent, the community in the initial days is very enthusiastic to hold your hand because they want their product to proliferate. So, choosing over an incumbent versus new is more of an art than science, there are no hard and fast rules here.

Coming back to the git versus svn story, if we were starting our product from scratch, I would have for sure gone with git, but we were working on an existing product where svn was deeply entrenched in the eco system. Moving to git just for the sake of being part of the latest wave would mean significant disruption in our whole process and a considerable investment of time. Along with this, there was the question of educating the team members on git who did not know about the git works flows etc. It boiled down to the eternal question of do we spend time in doing this just to satisfy our geek curiosity or can we spend that same time in trying to fix some real issues that we had with our code/infrastructure. I leaned in on the latter.

Fast forward to today, we have moved to git. Our team was growing, branching and merging were becoming issues due to the way svn branches work. We were facing a real problem which a better tool would alleviate. It was no longer a question of being part of the latest fad but was about making ourselves more efficient as a team and we took the plunge. As a leader of a tech team, you will always be in this situation, do not blindly start using something because it is new and shiny, adopt the new kid on the block only if it solves a real problem. Any sort of change is always disruptive irrespective of how small it is, do it only if it makes your team faster and meaner and adds value to your product. I keep reminding myself this all the time, your customers do not give a shit as to whether your product uses cobol or node.js behind the scene, all they care about is how it adds value to their life. Being nose deep in the tech community sometime makes us oblivious to this fact.

In any situation, being vocal about something has always helped me. For example, recently, when Pavi and me were in Thailand, we had booked our room in Phuket city while the happening night life was in Patong, which is an hour’s ride from the city. We try to use as much public transportation as possible in our sojourns, but our hotel manager informed us that we would not get a ride back from Patong to city in the night. The last bus was at around 7 in the evening. We would have to pay at least 600 to 700 Baht for a ride back. To give an idea, the bus ticket was 20 Baht. This was reaffirmed by the fare board put by Thai tourism in Patong beach, the same fare was listed to the city during night. Even, the Indian restaurant manager where we had our dinner confirmed this. After having shit loads of fun in Patong, it was time to go back. We tried a couple of tuk tuks and all were quoting the same price of 600 to 700 Baht but that did not stop us from trying and talking with the tuk tuk drivers for a lesser price. Finally, one tuk tuk guy agreed to take us back for 300 Baht and that driver became our friend too. On the next day too we took his tuk tuk back to city and tipped him generously on our last ride back.  If we had just accepted what others said and did not try to talk the price down, we would have been poorer by around 600 Baht in two days. Also, as a silver lining, we made a good friend and had a lovely conversation with the driver on our rides back.

I am not saying it always works, there are many instances in my life where being vocal did not bear any fruits. For example, in Zynga, I put very bluntly in my appraisal form that I was not very happy with the quality of work I was doing. I expected the HR to talk to me regarding this, but that did not happen. Even though, when I appraised them of my decision to move to FreeCharge, Zynga tried to talk me into moving to a team where I wanted to work, but it was too late by then, but still, being vocal did not achieve it’s goal here. But what the heck, I did not loose anything. I had the personal satisfaction of having tried my best, but somehow it did not work out. Now that I am responsible for FreeCharge, I encourage my team members to be vocal about their problems and talk to me in case of any issues however small it might look.

The bottom line is, if you see something is not right or you are not happy in a situation, talk to people, usually it works for the better, even if it does not, no pandas will be killed during the process :).

As someone who has worked from home for an extended period and also now being responsible for FreeCharge, I can objectively look at this from both sides. Let me first talk about my personal experience while working from home. I used to love it; the most significant factor that used to work for me was the time saved on the commute; the gained one hour in a day used to do wonders for me. Whenever I used to tell my friends that I work from home, I used to get the usual question—how do you manage to do it? I am not someone who needs external motivation to work; I would be writing code even if no one paid me for it; pay is the icing on the cake. I love building stuff, accruing knowledge, and bringing things to life, and my profession as a software engineer lets me do all these. Without digressing, I am trying to convey that, to work from home, you need to be highly self-motivated. If you are in your job only for the paycheck, then work from home will result in anarchy at your company. The usual reaction in big organizations when someone sends the WFH(work from home) mail is a snigger.

Even though everyone in your organization might be supremely dedicated, two more prerequisites are essential to making work from home successful in your organization. The first question to ask—are all my employees roughly on the same productivity and experience plane? Junior developers need a lot of mentoring and collaboration to create quality software irrespective of their motivation. It is a herculean task to bring this sort of collaboration and guidance when the guru and the acolyte are not physically present in the same location.

The second question is, is the entire team remote, or is it only a small portion of the team? If you are in a situation where only a sparse population of the team works from home, it will be challenging to pull this off. What usually happens in such a case is fences are created—on one side, you have the remote workers, and on the other, you have the office goers. Whey you have this asymmetry, information flow is one of the biggest concerns, as people who work in the office find it easy to proliferate information through word of mouth, which does not reach the employees who work from home. In such a situation, you have to engineer a cultural shift in the way people communicate and document in your organization. It is difficult to engender this as those who are physically present in the office do not see a point in many of the processes you will have to accommodate to benefit the remote workers. As you know, when people do not believe in something, it is next to impossible to get them to do it.

The above are difficult to pull off in an organization of the size of Yahoo. No wonder, while it works for 37Signals, it does not work for Yahoo. Mayer took the most sensible way out. When do you make special provisions in your organization to let people work from home? It would be only on extremely critical projects where a person/team works in a silo, detached from the rest of the organization. You are one hundred percent sure that this team/individual would not abuse the freedom that comes with work from home.

Image by Joshua Woroniecki from Pixabay

Being a perfectionist is all the rage these days. Perfectionism has a dark side to it, which gets glossed over.

When you are new to something, perfectionism acts as a hindrance. Many times, you know that you cannot dedicate wholly to something. In this scenario, perfectionism prevents you from starting—your subconscious has been trained to reject any activity whose outcome it foresees as not meeting the high standards you have set for yourself. When you face this situation, reason with your brain saying—I know that I might not ace this, but I want to do it for the fun of doing it. No rule states that you have to ace everything you do—the joy of doing something trumps the outcome.

This applies in work setting too. Take, for example, that you have to work on a rotten project and make it right. The rot is so thick that it permeates every aspect of the project. Being the perfectionist, you want to set everything right all at once, which is humanly impossible given the extent of the rot. Hence, you become depressed and grumpy and lose sight of the original goal. The right way to solve this is to remind yourself of the saying—little drops of water make the mighty ocean. List all the things wrong with the project, prioritize those that need to be fixed immediately and work on them. The other cruft in the project will irk you, but remind yourself that you will get to them one day and set them right.

You start something with zeal, you are almost in sight of the finish line, but you never seem to finish—why? You are in an infinite loop of polishing to meet the high standards you have set for yourself. The question to ask is, is the time you are investing in refining giving diminishing returns? Are the improvements making any difference other than giving you a sense of accomplishment? If the answer is yes, call it done.

Being a perfectionist is good, but do not take it to an extreme. Remember the saying—even the nectar of life, if taken in voluminous proportions, is poisonous.

Photo by Fabrizio Chiagano on Unsplash

I started to reason around the problem based on my mental model of the inner workings of the various components in our application. Jobs are freezing, and at the same time, tomcat stops serving out requests. Cannot be a tomcat issue as I am not getting a tomcat connection error, so it means that tomcat has enough threads to spare and its thread pool is not exhausted. The implication of this is new threads are being spawned but are not able to proceed. Hmm, ok, so the threads are getting blocked. Now how do I figure out where the threads are getting stuck. Can I hook something into the JVM and dump the state of all the threads running inside the container. Oh yes, JMX gives me that ability. Restarted tomcat with JMX configured, hooked into JMX and found out that the threads are getting blocked while trying to fetch database connections from the connection pooler. Our servers were not under any undue traffic spike, hence cannot be a scaling problem. That means some query is running rogue. How do I get the query that is running rogue? I can do two things - check the process list in the MySQL server or check the stack trace of the threads currently running and identify the point at which it is stuck. Took the thread approach, zeroed in on the code where the query was getting executed, fixed the query and things were back to normal.

If you are a full-stack developer, you can’t develop a deep understanding of every component in the ecosystem. Let us list down the things that you need to be on top if you are running a web application in Java. You have to know Java, the J2EE servlet spec, the application server that you are using, your MVC framework of choice, the components the MVC framework uses(maybe spring for dependency injection or hibernate for persistence), SQL and the peculiarities of the database, HTML, CSS JavaScript, JavaScript frameworks, CSS frameworks, the OS in which you are hosting your app, the administration of this OS, physical hosting, the list goes on and on. There are so many options available for each of these components. Also, these components have additional dependencies. Getting intimate with the intricacies and the idiosyncrasies of every element of the application is an arduous task better suited for immortals. This is where having a generic mental model helps.

Coming back to the problem of the crons stalling, even though I was not an expert in the internals of tomcat nor JMX nor DB connection pooling, I had an excellent general understanding of how these systems work in general, and I could fit the problem into it and reason through and fix it. Let us say that you are using an evented HTTP server and you see that the throughput is not up to mark. If you know how an evented server works in general, you can think of at least two loop holes in your system that might be causing this. Either you are doing blocking calls or running some CPU intensive tasks. This knowledge is transferable irrespective of whether you use Netty or Node.js or Tornado or some other new kid on the block.

I am not trying to discourage you from developing a deep understanding of particular frameworks or technologies, but to the contrary in addition to it try to abstract your understanding out of the specifics and into generals so that tomorrow when a new shiny technology shows up, you can reason around it based on your past experience.

Photo by Josh Riemer on Unsplash

Even when an event is all about appearance—say, a beauty pageant—organizers feel compelled to add a hollow intellectual layer. Remember the Q&A round in Miss World? Or the actress who stripped on screen and insisted the scene was vital to the plot? Let’s drop the moral baggage for a moment and really look at it. What’s the difference between someone using their body to gain attention and someone using their mind? Aren’t both natural gifts? Some people are born smarter than others—so why is it admirable to use intelligence but shameful to leverage beauty?

Two reasons come to mind. First, using beauty for personal gain is seen as selfish. It rarely contributes to the collective good. Ask yourself: what has Kim Kardashian done to improve the world? Now contrast that with Einstein, whose intellect helped humanity progress. Intelligence touches lives far beyond the self; beauty rarely does. Society moves forward through brainpower, not good looks.

Second, intelligence is something you can build. Beauty, in contrast, is largely a matter of genetics—you can’t work your way into symmetry. Intelligence rewards effort. It can be sharpened, trained, expanded. That’s why society places it on a pedestal—it’s accessible, at least in part, to everyone. When people push themselves to grow intellectually, it creates a ripple effect that lifts more than just the individual. That’s likely why our ancestors, consciously or not, shaped a culture that reveres the mind over the body. It was never just about fairness—it was about survival, progress, and building something that outlasts us.

In my opinion, these discussions miss the real essence of the issue. The question is not whether algorithms and data structures are a requirement for a programmer, it goes without saying it is. What we should debate is the depth of knowledge someone should have on these to be effective at their day to day work. Are we after someone who knows the different sorting algorithms and has a fair understanding of their trade-offs? At work, if this person faces a tricky problem, she knows where to start and then reaches out to Google to aid her? Or, do we want someone who can go to the whiteboard and code the algorithm from scratch and explain the math and computational complexity right in the interview? I can very confidently say that for most of the day to day work, you do not need to ponder on the mathematics behind algorithms and complex data structures. What is required is sound coding and design skills, tenacious curiosity, and a passion for excelling.

What do I mean when I say sound coding skills? The very basics; splitting lengthy code into modules, appropriate comments in the code, test cases, following code conventions and the spirit of the programming language. The above is the foundation of a great project and has a tremendous impact on the future of the project. If you do not follow these basic tenets of programming, scaling the project and modifying it in future becomes a project of its own.

Good design is what helps you scale your application as the business needs change and user base expands. In a web project, a good design would be to have business logic in a separate layer and not tying it up with controllers and views. I have personally seen this helping us at burrp! where we rolled out our mobile app in a month as our backend was flexible enough to support a new interface with minimal changes. If our design was not good, there was no way we could have developed a mobile app which had most of the functionalities of our website in such a short period.

Now coming to the most critical part, curiosity. Why is curiosity such an important trait? Without curiosity, we stagnate. We become complacent. We do not innovate. We do not try to become incrementally better. We think what we know is the ultimate truth and whatever language/framework/methodology that we use is the best. I have interacted with professional programmers who were not aware of PHP which powers a substantial number of websites.

If you are tenaciously curious, when faced with a problem, you start obsessing about it. You see it as a puzzle that has to be solved. For example, if you feel that a part of your code is slow, you dig deep into it and try to figure out what is happening. In this process, you come across people who have faced similar issues and solved the problem using a better algorithm or data structure.

I would say interviews focused heavily on data structures and algorithms are biased towards new college students and people who are just preparing for interviews. Most of these questions are in no way novel. Go to one of the interview preparation sites, spend a couple of days trying to solve the problems and suddenly you become an algorithm and data structures expert. Just because FANGs ask these kinds of questions in their interviews, everyone has adopted this interviewing style, and it has got to a point where interviews are more about showing off the interviewer’s data structure and algorithm skills rather than evaluating whether the person on the other side of the table is a good fit for the job. Also, an unhealthy side effect of this is that in a fury to concentrate on algorithms and data structure skills, everything else gets sidelined.

Unless you are working on some cutting-edge niche project, you do not need people who can whiteboard algorithm and data structure problems in interviews. Hiring people who are infinitely curious and can get incrementally better will give you a much better bang for your buck.

None of the movies/shows listed above are straight forward to grasp, they move around in tangles with plots with in plots. Take for example, Friends with benefits, you could detach your brain, keep it aside and still the movie would make sense to you(not to say I did not enjoy it :)). Movies like Looper,  you have to watch them with unparalleled concentration so that you do not miss something subtle the whole movie hinges on or some key plot which is the glue that holds the whole movie together or that one dialogue from the hero which is so mind numbing that you start thinking the director is the Buddha reincarnated. Sometimes, even after assuming you have got it, you read a blog where a person has a total different interpretation of the movie and you start doubting your own grasp of the movie. Was this what the director really trying to portray would be the conversation running in your brain for sometime. Is this person correct or am I in the right?

If you think deeply enough, the answer is crystal clear.  Why did you spend that weekend diving deep inside the code written by someone else, chasing that one bug which used to happen once in a while in your app? Or spend a couple of days trying to break into stripe’s servers? Or rewrite a piece of code the nth time thinking that finally you have the most legible and maintenable piece of code ever? A programmer’s brain snatches on to things which are not easy enough for it to grasp, which need reasoning and logic to untangle it. Many of these movies are not just about the movie, it is about the whole mind fuck experience, the digging in through internet trying to figure out that one scene in the movie which you did not understand, the endless debate with your friends as to what the director is really portraying in the movie and to some extent also the knowledge of being in that elite minority who really got what the movie was about. Last but not the least, these movies are like jigsaw puzzles where some of the pieces are missing and there is no one one to supply you with the missing pieces. It is upto you to craft these missing tiles, stitch them together and complete the puzzle. Show me a programmer who does not like a good puzzle to solve.

On a day to day basis, the status quo is getting challenged thanks to the internet and the surrounding technologies.

Traditional strangleholds are getting breached by young kids with unkempt hair, dreamy eyes, and a keyboard who are not part of the elite cabal. Internet and technology are challenging the status quo on a day to day basis and reversing the role of the producer and the consumer, something unprecedented till a couple of years back.

Let us take some examples and see how the internet and technology have brought on a paradigm shift.

India mainstream media till a few years back was an incestuous coterie whose narrative was guided by leftist agenda. Not to say there was no resistance to this, but it was minuscule with a minimal distribution channel. Mainstream media was a pack of mutually back patting buddies who used to invite each other for columns, talk shows, news anchoring, opinion pieces, etc. Outsiders who used to differ in their thoughts from this closed group were not welcome there.

Today, things have taken a radical turn; it is no longer possible for this invite-only group to keep the commoners at bay; they do not control the keys to this game anymore. If you have an opinion that you want to express, open a WordPress account, and start blogging, inseminate your thoughts through Twitter.  These days, social media sets the agenda for mainstream media. Until a few years back, mainstream media could bury inconvenient issues under the carpet and feast on trivialities, but it is no longer possible. Skewed news coverage and personal party affiliations get challenged on Twitter and Facebook daily.  Since the reach of social media is humongous, it is no longer possible for mainstream media to ignore social media’s reach and carry on with their doppelganger acts.

Now let us move over to consumerism.

A couple of years ago, if you wanted to take advantage of globalization and indulge in retail therapy, you had to be a resident of one of the A-Grade cities. Many goods were available only in shiny big cities that people in smaller towns could only dream of. Today, it is like the geography has shrunk; we can call it the reverse continental drift.

You want to buy a new pair of Diesel jeans and top it off with a UCB t-shirt and shiny new Nike sneakers, go online and shop. The online marketplaces are more than happy to deliver to even the remotest corners of the country. If you are okay with paying import duties, you can get things shipped from overseas in a jiffy.

Are you a shy guy and feel awkward buying condoms in the medical shop with hundreds of eyes prying on you? Order online. Many websites even promise discreet delivery to save you from that awkward moment.

As per me, the most profound impact of the internet has been in the knowledge space.

Knowledge is no longer confined inside elite libraries and institutions.  You do not have to spend megabucks on books and college degrees if your end goal is to seek knowledge. Almost everything is available free in the annals of the internet. You have access to quality education from renowned professors at the tip of your fingers; you no longer have to spend megabucks on an encyclopedia; Wikipedia is the new encyclopedia, and it is free.

If you are a fitness freak and want to try out one of the health programs but do not have the megabucks to spend on a personal trainer, surf the net for a variety of fitness programs. Think your shin splints are caused due to your poor running form, log into youtube, and watch running videos.  Do not have access to a quality swim coach, see Micheal Phelps in slow motion online and try to mimic him in the pool. You have a health/personal issue and do not know whom to ask, post it in one of the online forums, and some good samaritan at the other end of the globe will take pains to help you.

Thinking of traveling the world but do not know where to start, Wikitravel helps you out. Your local tour operator can no longer con you into expensive shitty tours; she no longer has a monopoly over this.

I can go on and on like this, but let me get to the title at this point.

Where does Gandhiji come into picture in the scheme of things?

Gandhiji stood for self-sufficiency and decentralization. That was his mantra, and this is what the internet is doing to the world. It is toppling autocrats from their cushy thrones,  kicking the middle man out and putting power and knowledge into the hands of the common man.

Anyone who knows to code and has an idea can take on the lethargic behemoth conglomerates of this world and come on top of them. The cost of trying out things has come down drastically that it is within everyone’s grasp. You do not need expensive hardware and servers to try out your ideas. Almost everything is available as a service these days; the cloud has put ideas within the grasp of anyone who has the will to do it; this is just the start, and things can only go up from here, and I do not see an end to this.

We have just tasted the icing on the cake, and the cake does not care who bakes it anymore, anyone hungry irrespective of whether she is from Palo Alto or Puttur has a fair shot at it.

Image by OpenClipart-Vectors from Pixabay

Views personal.

If you don’t see it, check your spam folder.

Thanks for subscribing!