Void

Make It Small

“Make it large” is the mantra these days, but when it comes to releasing software, think the opposite, make it small. The secret to a successful release is to break it into numerous small deployments; this serves a dual purpose, minimizes risk as well as gives you enough leeway to fix bugs before it negatively impacts end users.

Never wait for a feature to be complete for deployment. Break it into multiple smaller chunks and keep deploying these to production even though no one will be using it. Smaller deployments always win over one big bang deployment. Longer the code runs in production, more data points you have to analyze and take corrective steps.

animal-art-baby-1289845

If you can test with real users without them knowing, there is nothing like it. If you put your mind to it, you will always figure out a way to do this. Taking an example, let us say you are building a chat feature for your application. You can stealthily release it without informing your users and send dummy messages on their behalf. The user does not see any chat related functionalities, but behind the scene, you create dummy messages and send them to one another. Apparently, this is how Facebook tested their chat before launch; this allows you to check your feature realistically in ways it will be ultimately used.

Another trick is to log the action without actually carrying out the intended side effect. Say for example; you are introducing an API rate limiter. As a first step, start logging whenever an API call hits the rate limit threshold. During this phase do not block the calls, the idea is to emulate the result; this lets you test out the feature with real traffic and fix any bugs before it starts negatively affecting your API consumers. Once you are confident of your approach, you can roll out the feature where you block offending callers.

Phased rollouts is a life saver. Release features to a small set of users, keenly observe your application and gradually ramp up to 100%; this minimizes the impact of bugs and lets you catch them early.

Another lifesaver is feature flags. The idea behind a feature flag is simple; you put all your features behind an on-off flag which you can toggle if and when needed. The ability to turn off features during their early lives gives you a lot of flexibility and room to maneuver and improvise.

If you are working on refactoring or replacing an infrastructure component, never do a stop the world cut off. Always run both the systems in parallel and only once you are confident of the new approach, do the switch over. Say, for example, you are moving your data source from MySQL to Mongo, start by writing to both the datastores in parallel. Then change your application to work with Mongo while still keeping the option to fall back to MySQL if needed. Put this behind a flag which you can toggle. Only once you are confident of everything working with Mongo, pull the plug on MySQL.

The common thread that holds all these strategies together is to plan your deployments such that they are small and gradual; if something goes wrong, it is not catastrophic. You increase the scope gradually as you observe and gain confidence.

As they say – Risk is what’s left over when you think you’ve thought of everything. A bit of planning goes a long way in reducing risk as well as giving an excellent experience to your users.

Photo by Magda Ehlers from Pexels

Knowing Versus Doing

Over-engineering is ripe in the software industry; this mainly manifests in three ways:
1. Needlessly complicated design.
2. Lift and shift engineering – Picking up technology and practices from other places without understanding the context in which it was developed.
3. Being trendy – Using frameworks and languages that are currently in fashion irrespective of whether one needs it or not.

I have written about this before.

Please do not take this prose as an argument for being sloppy and building crappy products.

architecture-building-building-site-224924

I firmly believe one of the reasons why this occurs is that people confuse knowing with doing. You being aware of something should not force you into using or implementing it. You might have a personal inclination towards a language or a framework; you might even believe that it is the best in the world, but that does not imply you rewrite your currently well-working production application in the said technology. You come across something new, and shiny does not mean it has to be part of your application stack.

When we see something novel and trendy, our brain actively tries to figure out ways in which we can make it a part of our lives.

Another arena where this plays out is in doing things if and when needed. Because you read a blog post touting Redis as the next best thing to sliced bread should not make you go about slapping Redis as a cache in front of your data stores. Your data size might be tiny that it entirely fits into main memory. Achieving five nines of reliability is a daunting task which takes a lot of engineering and effort. As a geek, it is a fascinating problem to work on, but your application may not need it. Just because you are aware of what others are doing to achieve this does not imply you too embark on this path.

If you ponder over this, you will realize this behavior is not only restricted to work but plays out in our personal lives too. When we come across something new and exciting in any domain of life, we try to adopt it irrespective of whether we truly need it or not, and when we do it, we go way out of line in justifying its usefulness to ourselves.

Open Source and Revenue

This is the second part in a series on open source software. In the first part, we examined why equating open source with “just” free is fool’s errand. In this post, we will explore the different avenues for revenue from open source software.

lucas-favre-489526-unsplash

Get essays about software development, management, productivity, and more by subscribing to the blog

Join 222 other followers

The first one is pretty straight forward – charge for support, maintenance, consulting, and custom development. Software takes an effort to understand and maintain. Either you can do it in-house or outsource it to an external firm. Big enterprises have specific needs which require custom enhancements. They also need consistent support and suggestions. There are a lot of companies which have used this model to generate revenue from open source software like Redhat, Percona, etc.

The SAAS option for open source software has gained immense traction in the last decade or so especially since the advent of cloud. Instead of you taking the pain to host and maintain, the company behind the software deploys and manages it for a recurring fee. Most of the popular open source software is available under this model nowadays. WordPress, MongoDB, ElasticSearch are some prime examples of this strategy.

Another revenue strategy is the open core model. The core is open and free, but features which enterprises need like security, high availability and user management are part of the commercial offering. For example, the core database could be open, but clustering and high availability might be available only in the retail version. InfluxDB uses this model.

Then there is the licensing play. Software licensing is nuanced and comes with a lot of baggage and restrictions. The open source version of the software is released under a restrictive and commercial antagonistic license. If you want to use the software in a business setting, you have the option of buying a more permissive and commercial friendly license; this is very prevalent in software bundled in commercial products.

It is not uncommon for a company to use a mixture of the above strategies.

In the next part of the series, we will go through some recent developments in the open source world in an attempt to ward off the threat from big cloud providers like AWS.

Image Credit: lucas Favre

Open Source != Free

This is the first post in a series on open source software. You can read the second post here.

One of the most common conflations I see people making is mistaking open source software for free software; both are not the same. Being free is just icing on the cake, the more significant advantage is the freedom and flexibility that comes with open source software.

Let us say you are an enterprise with millions of dollars that has built its entire product on a closed source database. Your business is now profoundly entwined with the success of the database company. What happens if the database company goes kaput and shuts down? You now have to replace the database. Depending on the complexity of the product and the business, this might take significant effort and end up derailing your entire business. Open source software severely mitigates this problem.

beans-close-up-colors-1446267.jpg

 

Get essays about software development, management, productivity, and more by subscribing to the blog

Join 222 other followers

There is no concept of shutting down in open source world. Open source software development is inherently decentralized. In a lot of cases, committees govern open source software development. These committees have many stakeholders whose best interest is in keeping the software alive. Apart from this, many boutique firms provide development and maintenance services. All this leads to a robust eco-system that prevents a project from abruptly shutting down and taking you hostage.

Commercial closed source software reminds me of a famous line from the Eagles song Hotel California – You can check out any time you like, but you can never leave. Once you are locked into a piece of software, it is not easy to get out. During pricing discussion, there is an asymmetry. As a locked in customer, it is difficult for you to leave, there is no BATNA. Open source software does not have this problem.

Having access to source code is a huge advantage. When I was building Kwery, I used Apache Derby as the database. I started seeing a weird bug in Kwery which lead me to tinker with the Apache Derby source and finally unearthing a bug in the Derby database. A couple of mail exchanges on the Derby mailing list confirmed this. If I did not have access to the source code, there would be no way for me to figure this out.

I am not saying that open source software is a panacea to all problems and that you should completely shun commercial closed source software. Each has its place but equating opensource software with only free is folly.

You can read the second post here.

Image credit: Photo by Adailton Batista from Pexels

The Source

I think the minute you have a backup plan, you’ve admitted you’re not going to succeed.

Elon Musk said so. Chew on this for sometime before you read the rest.

ilyass-seddoug-667396-unsplash

I was not honest with the above. It was not Musk who made the statement but Elizabeth Holmes, at the peak of her popularity. It has been a hard fall for Elizabeth from then to now; she is now accused of fraud.

Did your opinion of the quote change with the source? Did you go from awe to retching?

I believe we give as much importance to the source of a quote as to the quote itself. We should be internalizing quotes and aphorisms by divorcing the source. A quote should be evaluated solely on its content, not who it came from. When we do not do this, the aura of the person shadows the import of the saying, reducing it to personality worship. The significance of the quote tends to get lost.

The act of viewing a quote objectively also acts like a shit umbrella against famous people getting away with baloney like the above from Holmes. If a successful person makes a statement, we take it at face value thinking if she says it, it must be true. We should guard ourselves against this attitude.

Internalizing a quote just on its content is not easy to do as we all love narrative fallacy, but it is worth trying. As with everything, we get better at it with practice.

Image credit: Ilyass SEDDOUG

Déjà Vu

You have been trying to solve a problem for quite some time; the solution appears to be elusive. As you grapple more with the problem, a seed of a solution germinates which sprouts into an answer. In hindsight, the resolution appears obvious. You kick yourself telling why did I not think of this sooner?

How many times has the above happened to you?

I believe almost everyone goes through these in life.

diego-ph-249471-unsplash

One simple hack to get better and faster at problem solving is to backtrace through your thinking. Once you successfully arrive at a solution, replay your unsuccessful attempts and figure out what you could have tweaked in your thinking process to arrive at the answer faster.

High performance teams do post-mortem analysis after a critical issue. Members create RCA(root cause analysis) document which contains what went wrong, what could have been done to prevent the untoward incident from occurring and what are the steps to be taken to avoid a relapse. We should be applying the same steps to our thought process when we do not arrive at solutions on time; think of this as an RCA of your thinking process.

This simple trick I believe helps us in getting better and faster at problem-solving.

Image credit: Diego PH

Now You See Me

In the modern software world, where micro-services are de rigueur, observability of systems is paramount. If you do not have a way to observe your application, you are as good as dead.

w-a-t-a-r-i-776085-unsplash

W A T A R I

The first step towards embracing observability is figuring out what to track. Broadly, we can categorize software observability into:
1. Infrastructure metrics.
2. Application metrics.
3. Business metrics.
4. Distributed tracing.
5. Logging.
6. Alerting.

Infrastructure metrics:
Infrastructure metrics boil down to capturing the pulse of the underlying infrastructure where the application is running. Some examples are CPU utilization, memory usage, disc space usage, network ingress, and egress. Infrastructure metrics should give a clear picture as to how well the application is utilizing the hardware it is running on. Infrastructure metrics also aid in capacity planning and scaling.

Application metrics:
Application metrics help in gauging the efficiency of the application; how fast or slow the application is responding and where are the bottlenecks. Some examples of application metrics are the API response time, the number of times a particular API is called, the processing time of a specific segment of code, calls to external services and their latency. Application metrics help in weeding out potential bottlenecks as well as in optimizing the application.

Infrastructure metrics give an overall picture whereas application metrics help in drilling down to the specifics. For example, if the infrastructure metric indicates more than 100% CPU utilization, application metrics help in zeroing in on the cause of this.

Business metrics:
Business metrics are the numbers which are crucial from a functionality point of view. For example, if the piece of code deals with user login and sign-up, some business metrics of interest would be the number of people who sign up, number of people who log in, number of people who log out, the modes of login like social login versus direct. Business metrics help in keeping a pulse on the functionality and diagnosing feature specific breakdowns.

Business metrics should not be confused with business reports. Business metrics serve a very different purpose; they are not to quantify numbers accurately but more to gauge the trend and detect anomalous behavior.

It helps to think of infrastructure, application and business metrics as a hierarchy where you zoom in from one to the other when keeping a tab on the health of the system as well as diagnosing problems. Keeping a check on all three ensures you have hale and hearty application.

Logging:
Logging enables to pinpoint specific errors. The big challenge with logs is making logs easily accessible to all in the organization. Business metrics help in tracking the overall trend and logging helps to zero in on the specific details.

Distributed Tracing:
Distributed tracing ties up all the microservices in the ecosystem and assists to trace a flow end to end, as it moves from one microservice to another. Microservices fail all the time; if distributed tracing is not in place, diagnosing issues which span microservices feels like searching for a needle in a haystack.

Alerts:
If you have infrastructure, application and business metrics in place, you can create alerts which should be triggered when they show abnormal behavior; this pre-empts potential downtimes and business loss. One golden rule for alerts is, if it is an alert, it should be actionable. If not, alerts lose their significance and meaning.

Both commercial, as well as open source software, are available to build observability. NewRelic is one of the primary contenders on the commercial side. StatsD, Prometheus and the ilk dominate the open source spectrum. For log management, Splunk is the clear leader in the commercial space. ELK stack takes the crown on the open source front. Zipkin is an open source reference implementation of distributed tracing. Most of the metrics tracking software have alerting capabilities these days.

If you already have microservices or are moving towards that paradigm, you should be investing heavily on observability. Microservices without observability is a fool’s errand.

Market Size

When I was working on Kwery, a lot of people used to ask me – Why are you not trying to build a team and raise money? The niche market that Kwery served was not amenable to VC money or building a team. Kwery was an experiment in creating a single person lifestyle business.

Market size is defined as – The number of individuals in a certain market who are potential buyers and/or sellers of a product or service.

carnival-carousel-circus-992763

When building any business, gauging the potential market size is the most critical step. The size of your team and the amount of money you want to raise should correlate with the market size. If the intention is to create a big business, you should be going after a big market. A lot of would-be startup founders are fixated on their idea and raising money that they do not analyze the market size or fool themselves into thinking that the market is vast. Peter Thiel, in his book Zero To One, beautifully explains the significance of market size.

The story that you build around your startup should also reflect the market size that you are after. In 2014, professor Aswath Damodaran estimated the value of Uber at ~6 Billion dollars with the story that Uber is an urban car service company. Bill Gurley, the lead investor in Uber, refuted this saying that Uber is not an urban car service company but a logistics company that is going to be everywhere, not just urban areas and is going to exploit global networking benefits, not local networking benefits. With this story, Bill Gurley was valuing Uber at ~53 Billion, a ~9X increase from Aswath Damodaran’s valuation. With a change in the story, the market size that Uber is after went up. This incident illustrates the importance of framing your startup story for big market size.

Some parting words on market size from Kunal Shah, founder of FreeCharge – Market > Product > Team.

Poor Man’s Anomaly Detection

You have a feature where if someone signs up on your product, you create a wallet for that person and top it up with complimentary money. Your organization swears by micro-services; hence sign-up logic is in one service and wallet creation and crediting is in another service. Once a user signs up, sign up service sends a message to the wallet service so that it can create the wallet and do the credit. To ensure the sanctity of the system, you have to make sure that the number of signups, wallets created and credits done match one another. Also, if these go out of sync, alerts need to be in place to take corrective action.

Since the two are disparate distributed systems, one way to achieve the above is to use an anomaly detector. There are off the shelf products for this as well as open source projects. If you do not have the time, need and resources to invest in deploying an anomaly detection system, having a reconciliation system is the way to go.

black-and-white-child-connected-265702

Reconciliation is deeply entrenched in the financial domain where it is a way of life. The technology world can borrow this and use it as a poor man’s anomaly detector. For the scenario that we started with, we run queries on the data repository of the sign-up and wallet systems at regular intervals. These queries fetch the count of sign-ups, wallet creations, and credits that occurred during the period. Once we have the numbers, all we have to do is ensure that they match. One can do this with a simple bash script; this is extremely simple to develop and deploy.

Reconciliation can play a role in all places where two-phase commit flows are involved. For example, most payment flows follow a two-phase commit process. You first deduct money from the user’s payment instrument and then fulfill the commitment. There is a good chance that post payment debit, your system dies not doing the fulfillment. Having a reconciliation system in place helps you to take corrective action in these scenarios.

Reconciliation is a simple way to achieve anomaly detection until you have the need and the resources to invest in a more robust distributed anomaly detector.

Startups and VC La La Land

With daily morning newspapers dedicating pages and front lines to startups and venture capital(VC) investments, they are no longer the niche they used to be. Whenever a startup investment is in the news, I hear questions along the lines of:

  1. Is the VC crazy to invest so much money in that startup?
  2. Why does that startup need massive amounts of money?

In my opinion, these questions stem from not having an understanding of how the VC business works.

Let us say that there is a contest with prize money of 1 billion dollars. You have to form teams, do some tasks; whoever wins, walks away with the prize money. There are a hundred individuals who are eager to participate but do not have the financial means and resources to recruit people to form teams and get to work. There is a wealthy benefactor who is ready to fund these hundred individuals with 10,000 dollars each. He strikes a deal with each of these hundred people saying, if you win, you give me 50% of the prize money; if you lose, you do not have to repay the 10,000 dollars I gave you. The benefactor is sure that one of them is going to bag the prize, she does not know who. The benefactor has invested 1 million dollars in total(10000 * 100) to ensure a return of half a billion dollars, i.e., 500x returns.

The above is a crude explanation of how the VC industry works. The individuals are the startup founders, and the wealthy benefactor is the VC. The only difference is, in the real world, there is no guaranteed success, i.e., there is no way for the VC to tell with cent percent certainty that someone is going to win for sure; it is all about probabilities, data, reading the trend and gut instincts. To bring a semblance of certainty to returns, VC spreads her bet on a wide array of startups in different domains; at least a handful of them is going to hit the jackpot.

dollar-499481_640.jpg

Why does this startup need so much money and are the VCs crazy to give them that?
Startups that VCs invest in are not chasing organic growth. Organic growth is steady, sustainable growth which compounds to big returns over decades. VCs are chasing exponential returns in a few years; not steady growth. The money that VCs give to startups helps them with this exponential growth.

Why so?
Double-digit returns are achievable by investing in public companies and debt. Why take the risk of investing in startups with absolutely no guarantee of returns when there are better ways to get there? For the chance that VCs are taking, the gains have to be comparable, i.e., exponential returns.

Why do startups need so much money?
To short-circuit the whole organic growth curve and achieve exponential growth so that investors get the returns in a relatively short period. The money is spent on scaling teams, building technology and more importantly in acquiring users through discounts and advertising. In short, the VC capital infusion empowers startups to compress multi-decades of growth to a decade or less.

Why is this startup spending money on discounts when they are not even profitable?
Big spending circles back to the concept of exponential growth and building market dominance for a winner take all effect. The idea is that once the startup dominates the market or has enough users, they will eventually figure out a way to make money, dominate the market and turn profitable. Case in point; Amazon, Google, and Facebook. When VCs invested in these firms, none of them were revenue positive, but look at where they are today. Of course, there is a risk of startups never figuring out how to make money, but VC investment is about calculated risks and spreading the bet.

You might have read of startups of previous successful founders raising astronomical capital without even having a product. VCs are not crazy to do this; they are betting and playing with probabilities and sentiments. The chance of the founder hitting the lottery again is high, hence the clamor to invest. VCs know for sure that not all the startups they invest in are going to be home runs, some are going to go dud and die, and they take this into account while investing. Only a very handful of the startups they invest in turn out to be successes and these handfuls generate the returns. A minority of the successes cover up for the majority of the failures. Returning to our lottery example, out of the hundred, the VC is betting on only one person to win, she knows the rest of them are going to fail, but she gets 500x returns with just one win and 99 failures.

Treat this as a dummies guide to startups and venture capital. I have watered down a lot ignoring VCs and funds who invest for more extended periods. Like any business, the VC industry is varied and diverse.