Void

Ode To Queues

If you have a producer with an uneven rate of production and a consumer which cannot keep pace with the producer at its peak, use a queue.

If you have a workload which need not be addressed synchronously, use a queue.

If your customer-facing application is riddled with workloads which can be deferred, move these to a queue thus making the customer-facing application lean and mean.

duck-3217049_640

Think of a queue as a shock absorber.

There are workloads which need to be processed immediately with sub-millisecond latency and then there are ones where you have the luxury of taking time. It is advisable not to mix these in an application. The second kind of workload can be addressed by moving it to a queue and having a consumer process them.

For example, consider a scenario where you are consuming messages and persisting them in a data store. These messages are coming in at a variable rate and at its peak, the data store cannot handle the load. You have two options. Scale the data store to meet the peak load or slap a queue in between to absorb the shock. Queue solves this problem in a KISS manner.

Queues enable applications to be highly available while giving enough room to manoeuvre. As long as the queue is highly available, the chance of message loss is almost nil. Since a queue is durable, you need not perfect your consumer’s high availability, you get leeway to manage.

With applications embracing microservices paradigm, there is a lot of API back and forth. Not all API consumption has to be in real-time. Whatever can be deferred should use a queue as the transport mechanism.

Queue introduces a bit more complexity into an application but the advantage it brings to the table makes it a worthwhile investment.

Advertisements

Process Introduction

Whenever a new process is introduced, there is always going to be some discomfort. The cause can be categorized into:
1. Uneasiness due to newness.
2. There is a problem with the process itself.

idea-1876658_640

Category one is due to human nature. Deviation from an established routine causes queasiness and a yearning for the old way. It takes over-communication, repetition and sometimes “just giving it time” to tide over this initial phase. This is usually a short-lived phenomenon.

Category two is the troublesome one. When someone complains about a newly introduced process, it is extremely important to get to the source of this discomfort. Prod as to whether the reason for disapproval falls into category one or two.

A good process has to roughly follow the Libertarian Paternalism idea popularised by Behavioural Economist Richard Thaler. The process should be a nudge towards better behavior rather than a dictatorial dictum. A process whose intention is to police people does not end up well.

A new process introduces some amount of friction but this friction has to be local, not global. This friction should not slow down the task at a global level, instead, it should aid speed, agility, and stability.

Take the checklist process as an example. It nudges people towards being more aware and aids better behavior. It does introduce friction at the local level but on the whole, globally, the task speeds up with a much better result on an average.

It always helps to think along these lines to figure out whether a new process is worth its salt. Instead of introducing a new process and then reneging, put in the effort to evaluate the efficacy of a process beforehand.

Checklist

Checklists have been in vogue for quite some time now. Probably Atul Gawde’s book The Checklist Manifesto kickstarted this. I have been late to the party but once I arrived, I never left. A checklist is an amazing tool to organize personal as well as professional life.

checklist-1266989_640

 

Hospitals have figured out that by following simple checklists they can reduce the surgical mortality rate by a big percentage. The airline industry, which is an epitome of the highest possible standard in safety and training, relies on checklists for fail-safeness.

Who does not forget daily chores? Who has not kicked themselves for being late on bill payments? A to-do list solves this problem. There are tons of to-do apps out there, just pick one to start with. Apart from solving daily trivial problems, a checklist is a great way to keep track of long-term goals too, be it saving more, losing weight or adopting a healthier lifestyle.

Professional life requires one to regularly follow-up. A checklist is a great way to list down the follow-ups needed. On the flip side, no one likes doing follow-ups. If you want to be one of those rare people who respond without requiring to follow up, use a timebound to-do app to keep track.

Checklist brings efficiency to project management. Having a checklist of all the tasks to be accomplished before taking a project live leaves little room for ambiguity, misses and last minute scurrying.

A short-term or throw away project may not justify the time spent in automating disparate tasks related to the project. In this scenario, a checklist is a great substitute.

automation

All organizations have an onboarding process. Make a checklist out of it and hand it over to employees on the day of joining. All software projects have some common alerts and metrics needed. Checklist them. Once you do this, you do not have to bother about this for every new project. View checklist as a temporary substitute while you scout for software to automate these.

In short, a checklist can be used as a Swiss Army knife to organize and automate life. A checklist institutes repeatability eliminates ambiguity and improves efficiency.

PS: Link for the above comic from XKCD.

Software Security

Some disparate thoughts on security in no particular order.

Many security bugs can be avoided by making a clear distinction between authentication and authorization. When one logs into Facebook, one uses a username and password. Facebook lets you log in only once it is sure that you are the owner of the account by verifying your password. This is authentication. Once you log in, you cannot view all your friends’ photos. You can only view those photos which your friends have authorized you to view. This is authorization. There is a class of security bugs that arise because developers have not made this distinction.

security-department-1653345_640

A lot of security is knowing what not do. Security by obscurity and hand rolling security algorithms and protocols are the two things that immediately come to my mind. For example, while storing passwords, instead of coming up with an elaborate custom secure storage scheme, employ the industry standard bcrypt.

There is a thought process that you will do better security by having tons of access control. One of the manifestations of this is restricting SSH access to production boxes. Unless you have invested tons in tooling, this slows down teams drastically. In today’s world, where speed is paramount, this does not work. Under pressure to do things fast, teams find ingenious ways to circumvent these controls. Strict access control only works in organizations which are fine with taking things slowly but this usually stifles productivity and leaves a bevy of frustrated developers. The only way around this problem is to have the most necessary access control and take care of the rest through tooling. An example is how Netflix uses tools to enable developers to SSH into production boxes without compromising security.

Security implemented in a naive manner goes against the human nature of seeking to accomplish tasks in the least restrictive manner. If you do not invest in tooling, security always gets in the way of accomplishing things.

A less intrusive way of doing security is to configure systems with sane defaults. An example – when you provision a server, ensure that it is fortified by default. If you are using external tools, configure them with defaults. For example, if you are using Slack, configure it so that only people with your organization’s email address can sign up. Carry out a periodic audit of systems. This could be anything from periodically scanning SSH access logs to repository audits to ensure secrets and passwords are not leaked.

No writeup on security can be complete without touching upon compliance. There are tons – PCI, HIPAA, SOX etc. All these come with their own baggage. One simple way around this is to first understand what and all parts of your application have to be under the scope of compliance. For example, if you have an e-commerce application taking credit card information, you have to be PCI compliant. But this does not mean your entire application has to be under the scope of PCI audit. You can smartly bifurcate the application into parts that deal with payment and parts that do not. Once this is done, only the parts that deal with payment have to be under PCI scope.

A final note, security is a never-ending concern, there is nothing called enough security. Where you draw the line is up to you.

Here is a hilarious comic by XKCD on teaching a lesson to people who do not follow security practices.

exploits_of_a_mom

Blogs Versus Books

Long time back, in one of the social gatherings, someone asked me what do you do in your pastime? I said I read a lot. That person asked – What was the last book you read? I stumbled a bit and took a name. I realized that it had been quite a while since I read a book. Unbeknownst, I had gotten into the habit of reading blogs and articles online with social media and messaging boards acting as the source. There is always more than you can chew with articles touching a wide variety of subjects. Also, it feels a bit like going down a rabbit hole, one blog/tweet leads to another which leads to another and so on until you lose track of time.

I have been a voracious reader of books since my childhood. I got hooked onto online articles and blogs only during the later part of my life. This got me thinking about the difference between reading blogs versus books.

book-1659717_640

Even though most books revolve around a core central idea, the author takes the pain to reinforce this with varying thoughts and anecdotes. The author builds a structured case around the idea and presents a lot of scenarios leading to the core idea. Reading blogs and articles in most cases feels like reading a summary of a concept or just one facet of it.

When you read a book you are enjoying the journey whereas reading a blog feels more like focussing on the destination. Both have their own place but one should strike a balance between the two.

Naming Things

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

Even though the above might have been in jest, naming variables while writing code is a head-scratching experience. Should I make it short? Should I make it descriptive? If descriptive, how descriptive? These thoughts keep running in one’s head.

tag-309129_640

A simple strategy is to keep the descriptiveness of a variable’s name in line with the reach of that variable. If the variable is short-lived i.e within a small block, stick to a short name as the cognitive load of the variable is negligible. If the variable’s reach is much larger, as in if it spans a large number of lines, make it as descriptive as possible.

Goes without saying that names should adhere to the conventions that your team has adopted.

Self-infliction

I recently watched the movie Hichki. The Plot of the movie revolves around a group of kids from less privileged strata of society who get a chance to attend an elite school. During the course of time, these kids feel that the school and the more privileged students there do not give them the respect they deserve. They rebel against this by not studying, causing nuisance and failing grades.

suicide-1267709_640

Kids can be excused when they exhibit self-inflicting behavior but sadly, a lot of adults too manifest this. One common adult refrain is – The organization is not treating me well, so I am not putting my best into the job. Keeping aside morality and call of duty, who loses due to this behavior? It is you. When you do not put your best into anything, you do not improve. If you do not improve, you do not progress.

It is in your best interest to put 100% into your job irrespective of how your organization treats you. If you feel you are not treated well, talk to your higher up and see if you can change this. If not, move on. Slacking is not an answer. You are harming yourself by exhibiting this sort of behavior.

Kwery

When I was at FreeCharge, I did a lot of data engineering. One of the recurring patterns that I saw was:
1. A new product or feature gets launched.
2. The PM wants to schedule a report for her feature.
3. The report should run at a particular frequency.
4. The PM along with others from the C-team should get this report in their e-mail inbox.

We had hundreds of reports like this running at FreeCharge. We explored tools like Redash, Metabase etc; but all these tools were geared towards creating beautiful dashboards and visualizations, report generation and delivery through email was given a step motherly treatment. We hacked together a solution to solve this problem with bash/Python scripts, GIT, Jenkins etc. The query to generate the report was written by the business analyst team but they had to depend on the engineering team to schedule the report, convert the report into HTML and deliver it over email. One of the goals of the solution was to make the business team self-reliant in report generation. Since the solution was stitched together using disparate tools that were not really meant to solve this problem, it sort of worked but had its own chinks. This seeded the thoughts for Kwery in my mind.

logo

Kwery is a tool that solves the above niche problem really well. It was also an attempt at building a single person lifestyle business that could supplement my regular income. I followed a lot of lean startup principles while building Kwery. Built an MVP, deployed it at an organization that had the above pressing problem and then iterated from there.

With Kwery, initially, I had to take a call as to whether to make it a SAS product or an in-house deployment. I decided against the SAS route as I was not sure how many organizations would be comfortable giving complete access to their data sources to an external fledgling SAS tool. I wanted to make the onboarding process of Kwery as simple as possible. Hence, I shunned all external dependencies for Kwery, opted for an embedded database so that I could package Kwery as a single binary. The only dependency needed to run Kwery is Java 8 runtime.

I did deploy Kwery in a couple of places which use it even today. Kwery has received a lot of love from these places but I never hit the numbers that I had in mind when I started. The opportunity cost was bearing on me. I had to take a call between continuing working on Kwery or cut my losses and move on, I chose the latter.

Like the old saying – Every cloud has a silver lining, I have open sourced Kwery under MIT license. It is very easy to get started with Kwery. Give it a spin, open Github issues if you face any trouble. Pull requests are very much welcome.

Solving Problems

When faced with a problem, the way we should think is:
1. What is the quick and dirty solution?
2. What is the long-term solution?

hand-2208491_640

Most of the time, one tends to do only one conveniently ignoring the other. Our mind goes into overdrive and quickly implements a hacky solution and then we forget the problem only to find it re-occurring. Or, we languish to implement a long-term solution while the problem drags on.

One of the reasons why we might find it difficult to implement both is that they require a fundamentally different kind of thinking. The quick and dirty solution involves hustling and running around to get things done. Somehow by hook or crook, one wants to plug the hole. A long-term solution requires one to analyze the problem from all angles, think deeply about it and figure out a well-rounded solution. A vague comparison would be system 1 thinking versus system 2.

Both are equally important – neglecting either is a recipe for disaster.

Switching Languages

think-2177813_640

Many are apprehensive about switching programming languages. It is perfectly fine to have preferences – I am heavily biased towards statically typed languages with great tooling support, but being dogmatic is not something one should aim for.

What could be the downsides of switching programming languages? I am disregarding the psychological aversion to change and sticking to hard facts.

1. One will lose the fluency(syntax).
This is a non-issue, syntax is similar to muscle memory, one will get it back in a day or two. This is akin to swimming or driving after an extended break, one naturally gets it back.

2. One will forget the way of doing things.
Every language has a culture and a community accepted way of getting things done. Regaining this might not be as easy as syntax retrieval, but with some effort and thought, one should recoup.

3. One will not be up to date with the language.
Languages keep evolving, core ideas and philosophy remain the same. The standard library might become more expansive, VM might become faster, some earlier prescribed way of doing things might be an anathema now but the foundational principles remain intact.

4. There is no demand for this language.
As long as your fundamentals are good, this should not be a concern. There are roles which require deep language know how, but these are far and few. In fact, it is the opposite, more the languages in your kitty, more the opportunities.

The biggest upside to learning a new language is the exposure to new ideas and thought processes. Any new language immensely expands one’s horizon. For example, the way Java approaches concurrency is very different from GoLang’s take on concurrency. Having this sort of diverse exposure helps one build robust systems and mental models.

Programming languages should be viewed as a means to an end, not an end in itself. There are cases where programming languages make a difference, otherwise, there would not be so many around with new languages cropping up now and then, but you are doing a disservice to yourself by restricting to a few.