Creating Quality Software

Creating software is easy; creating quality software is hard.

In today’s world of umpteen open source libraries, cloud computing platforms, and StackOverflow, creating software is child’s play. You can look up things on the internet, glue together components, and voila – you have created something.

What is hard is creating quality software. Software that scales; software with very few bugs; software that is elegant and takes care of edge cases and boundary conditions; software that stands the test of time.

smiley-2979107_640

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

I believe the fundamental reason for this is the age-old problem of simultaneously holding opposing thoughts in mind and doing justice to them.

To create something, you need to be an optimist. You need to look at the bright side of things. Whereas, if you want to build quality software, you have to foresee all the probable future problems and develop interventions for them, i.e., you need to wear the pessimist cap for this.

Being an optimist and pessimist at the same time is hard, and I believe that is one of the reasons why quality software seems to be elusive.

Image by Gino Crescoli from Pixabay

“Intentionally” Leaky Abstractions

Software abstractions that developers create should leak enough to let non-developers design the right product experience.

Eons back in internet history, Joel Spolsky wrote a post on leaky abstractions. The gist of the writing is that even though we create software abstractions to make life easy, they somehow unintentionally leak through.

With technology getting a lot more complicated, I believe that the software abstractions that developers create should intentionally leak to some extent to let non-developers design the right product experience.

gas-2400340_640

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Taking a concrete example, let us say that you are creating a search utility which goes over a lot of data. Your algorithm has two steps, one that works on recent data and the other, older data. Recent search is blazing fast while searching through old data is slow. You do both in parallel, combine the results and display it to the user. You have two options here; you can abstract this two-pronged approach from the product teams and act as if the search is a single step process or you can leak this abstraction so that they can take this into account while designing the product.

What can the product team do with this information?
They have multiple options.

Show the faster results first without waiting for the slower ones. Where the older results are displayed, show a loading placeholder, and then populate the results as and when they become available.

Another option is to make this a two-step process by showing only the recent results and then asking the users explicitly whether they want to search through old data too.

They can do this only if your abstraction leaks through informing them of the internal search nuances.

You need to be judicious with what you expose, and what not, so that the product teams are not burdened with unnecessary details – striking the right balance is essential.

Image by Jesse Bridgewater from Pixabay

Competitive Advantage

Are you focussing enough on developer productivity? If not, read on why you should.

Technology is getting commoditized these days. In the yesteryears, you could have counted on your software frameworks and technologies, hardware prowess, access to capital and talent pool as significant competitive advantages. In a world, which is getting commoditized day by day, the access to these has been democratized, thus leveling the playing field. Today, it does not matter whether you are a single person startup or a large multinational corporation; everyone has access to almost the same resources.

erwan-hesry-102070-unsplash.jpg

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

Cloud providers ensure you have enough servers and hardware at your disposal to meet your growing demands. Open source frameworks and technologies provide the required technological backbone to build stuff. Umpteen services exist to recruit talent and develop products; co-working spaces take care of your physical office requirement at an amenable cost; social media gives you access to a broad marketing audience. In spite of these, if you are short on capital or want to splurge, there is the ever-growing VC industry to take care of your capital needs. The icing on the cake is internet and sites like Stackoverflow and Coursera which have levelled the playing field when it comes to access to knowledge.

In light of this, what constitutes your business competitive advantage? One short answer, developer productivity.

Developer productivity dictates how well you can effectively and efficiently leverage all the resources at your disposal to build a great product and trounce your rivals. Developer productivity is a trifecta of having the right engineering tools, structure and processes. When you excel in these three, developers can move fast, and business can conquer markets with ease.

Developer productivity is too vast a topic to be covered in one post. I will be writing more on this in the coming months; it is an area which is extremely close to my heart, and I firmly believe companies do not focus enough on this.

Photo by Erwan Hesry on Unsplash

Make It Small

“Make it large” is the mantra these days, but when it comes to releasing software, think the opposite, make it small. The secret to a successful release is to break it into numerous small deployments; this serves a dual purpose, minimizes risk as well as gives you enough leeway to fix bugs before it negatively impacts end users.

Never wait for a feature to be complete for deployment. Break it into multiple smaller chunks and keep deploying these to production even though no one will be using it. Smaller deployments always win over one big bang deployment. Longer the code runs in production, more data points you have to analyze and take corrective steps.

animal-art-baby-1289845

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

If you can test with real users without them knowing, there is nothing like it. If you put your mind to it, you will always figure out a way to do this. Taking an example, let us say you are building a chat feature for your application. You can stealthily release it without informing your users and send dummy messages on their behalf. The user does not see any chat related functionalities, but behind the scene, you create dummy messages and send them to one another. Apparently, this is how Facebook tested their chat before launch; this allows you to check your feature realistically in ways it will be ultimately used.

Another trick is to log the action without actually carrying out the intended side effect. Say for example; you are introducing an API rate limiter. As a first step, start logging whenever an API call hits the rate limit threshold. During this phase do not block the calls, the idea is to emulate the result; this lets you test out the feature with real traffic and fix any bugs before it starts negatively affecting your API consumers. Once you are confident of your approach, you can roll out the feature where you block offending callers.

Phased rollouts is a life saver. Release features to a small set of users, keenly observe your application and gradually ramp up to 100%; this minimizes the impact of bugs and lets you catch them early.

Another lifesaver is feature flags. The idea behind a feature flag is simple; you put all your features behind an on-off flag which you can toggle if and when needed. The ability to turn off features during their early lives gives you a lot of flexibility and room to maneuver and improvise.

If you are working on refactoring or replacing an infrastructure component, never do a stop the world cut off. Always run both the systems in parallel and only once you are confident of the new approach, do the switch over. Say, for example, you are moving your data source from MySQL to Mongo, start by writing to both the datastores in parallel. Then change your application to work with Mongo while still keeping the option to fall back to MySQL if needed. Put this behind a flag which you can toggle. Only once you are confident of everything working with Mongo, pull the plug on MySQL.

The common thread that holds all these strategies together is to plan your deployments such that they are small and gradual; if something goes wrong, it is not catastrophic. You increase the scope gradually as you observe and gain confidence.

As they say – Risk is what’s left over when you think you’ve thought of everything. A bit of planning goes a long way in reducing risk as well as giving an excellent experience to your users.

Photo by Magda Ehlers from Pexels

Knowing Versus Doing

Over-engineering is ripe in the software industry; this mainly manifests in three ways:
1. Needlessly complicated design.
2. Lift and shift engineering – Picking up technology and practices from other places without understanding the context in which it was developed.
3. Being trendy – Using frameworks and languages that are currently in fashion irrespective of whether one needs it or not.

I have written about this before.

Please do not take this prose as an argument for being sloppy and building crappy products.

architecture-building-building-site-224924

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

I firmly believe one of the reasons why this occurs is that people confuse knowing with doing. You being aware of something should not force you into using or implementing it. You might have a personal inclination towards a language or a framework; you might even believe that it is the best in the world, but that does not imply you rewrite your currently well-working production application in the said technology. You come across something new, and shiny does not mean it has to be part of your application stack.

When we see something novel and trendy, our brain actively tries to figure out ways in which we can make it a part of our lives.

Another arena where this plays out is in doing things if and when needed. Because you read a blog post touting Redis as the next best thing to sliced bread should not make you go about slapping Redis as a cache in front of your data stores. Your data size might be tiny that it entirely fits into main memory. Achieving five nines of reliability is a daunting task which takes a lot of engineering and effort. As a geek, it is a fascinating problem to work on, but your application may not need it. Just because you are aware of what others are doing to achieve this does not imply you too embark on this path.

If you ponder over this, you will realize this behavior is not only restricted to work but plays out in our personal lives too. When we come across something new and exciting in any domain of life, we try to adopt it irrespective of whether we truly need it or not, and when we do it, we go way out of line in justifying its usefulness to ourselves.

Open Source and Revenue

This is the second part in a series on open source software. In the first part, we examined why equating open source with “just” free is fool’s errand. In this post, we will explore the different avenues for revenue from open source software.

lucas-favre-489526-unsplash

Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

The first one is pretty straight forward – charge for support, maintenance, consulting, and custom development. Software takes an effort to understand and maintain. Either you can do it in-house or outsource it to an external firm. Big enterprises have specific needs which require custom enhancements. They also need consistent support and suggestions. There are a lot of companies which have used this model to generate revenue from open source software like Redhat, Percona, etc.

The SAAS option for open source software has gained immense traction in the last decade or so especially since the advent of cloud. Instead of you taking the pain to host and maintain, the company behind the software deploys and manages it for a recurring fee. Most of the popular open source software is available under this model nowadays. WordPress, MongoDB, ElasticSearch are some prime examples of this strategy.

Another revenue strategy is the open core model. The core is open and free, but features which enterprises need like security, high availability and user management are part of the commercial offering. For example, the core database could be open, but clustering and high availability might be available only in the retail version. InfluxDB uses this model.

Then there is the licensing play. Software licensing is nuanced and comes with a lot of baggage and restrictions. The open source version of the software is released under a restrictive and commercial antagonistic license. If you want to use the software in a business setting, you have the option of buying a more permissive and commercial friendly license; this is very prevalent in software bundled in commercial products.

It is not uncommon for a company to use a mixture of the above strategies.

In the next part of the series, we will go through some recent developments in the open source world in an attempt to ward off the threat from big cloud providers like AWS.

Image Credit: lucas Favre

Open Source != Free

This is the first post in a series on open source software. You can read the second post here.

One of the most common conflations I see people making is mistaking open source software for free software; both are not the same. Being free is just icing on the cake, the more significant advantage is the freedom and flexibility that comes with open source software.

Let us say you are an enterprise with millions of dollars that has built its entire product on a closed source database. Your business is now profoundly entwined with the success of the database company. What happens if the database company goes kaput and shuts down? You now have to replace the database. Depending on the complexity of the product and the business, this might take significant effort and end up derailing your entire business. Open source software severely mitigates this problem.

beans-close-up-colors-1446267.jpg

 

Get essays about software development, management, productivity, and more by subscribing to the blog

Join 19 other followers

There is no concept of shutting down in open source world. Open source software development is inherently decentralized. In a lot of cases, committees govern open source software development. These committees have many stakeholders whose best interest is in keeping the software alive. Apart from this, many boutique firms provide development and maintenance services. All this leads to a robust eco-system that prevents a project from abruptly shutting down and taking you hostage.

Commercial closed source software reminds me of a famous line from the Eagles song Hotel California – You can check out any time you like, but you can never leave. Once you are locked into a piece of software, it is not easy to get out. During pricing discussion, there is an asymmetry. As a locked in customer, it is difficult for you to leave, there is no BATNA. Open source software does not have this problem.

Having access to source code is a huge advantage. When I was building Kwery, I used Apache Derby as the database. I started seeing a weird bug in Kwery which lead me to tinker with the Apache Derby source and finally unearthing a bug in the Derby database. A couple of mail exchanges on the Derby mailing list confirmed this. If I did not have access to the source code, there would be no way for me to figure this out.

I am not saying that open source software is a panacea to all problems and that you should completely shun commercial closed source software. Each has its place but equating opensource software with only free is folly.

You can read the second post here.

Image credit: Photo by Adailton Batista from Pexels