Month: November, 2018

Sherlock Versus Calvin Ball

We can classify software development into:
1. Maintaining and enhancing existing software.
2. Software development from scratch.

Given a choice between the two, developers usually gravitate towards from scratch development. Developing something from scratch is an intensive creative work where you have the freedom to shape the product the way you see fit. Hence, it is pretty obvious why people prefer this. I draw a parallel here with Calvin Ball. For those of you not familiar with Calvin ball, it is a game that Calvin invented where he makes rules on the fly during the game. From scratch development is akin to Calvin Ball, you can create and amend rules on the fly. If you chose a framework and in the course of development you see it does not fit the bill, you have the freedom to swap it with something else. You are operating under a lot of degrees of freedom.


Maintaining and enhancing existing software is more like solving a puzzle or playing a game with well laid out rules. Someone has already laid the foundation or in a lot of cases built the entire structure. You first have to expend time and effort in groking this and familiarising yourself with what is already there, only then you will be able to do something. A lot of times you need to get into the mind of the original developer and decipher things from her perspective. Working on code written by others is more like Sherlock Holme’s work. When you do changes and enhancements, you have to ensure what you are doing fits well into the existing framework. You are working in a constrained environment; you have to stick to the rules of the game. All this is as much or sometimes more challenging than developing software from scratch.


Debugging is an acquired skill which carries over to all areas of development. When you troubleshoot code written by others, you become more attuned to add enough debugging information in the code you write. You will start empathizing with the person who will maintain your system in the future and ensure that person has enough data points to debug when things go wrong. It might as well happen that that future person is you only. Injecting debugging information and future proofing your project is a fundamental behavioral change that maintenance induces in you.

There is nothing wrong in preferring to create something from scratch, but it is imperative to have the second skill set under your belt. The real world requires more of type two work than type one. If from scratch development is all you have done till now, it is high time you challenge yourself with category two work. You will feel a bit frustrated and handcuffed in the beginning, but the way to approach it is like solving a mystery. If you see it that way, it becomes a fun and entertaining experience.

PS: Calvin and Hobbes image taken from Wikipedia.


Concurrency Models

We can roughly classify concurrency models into:
1. Thread based concurrency.
2. Event based concurrency.

Imagine that you run a store with only one customer service representative. As soon as a customer walks in, the customer service representative greets the customer with a quick hello saying – “If you need any help, give me a shout, and I will help you out.” She then waits for the customer to seek help. She aims to complete the interaction as soon as possible and wait for the next communication. When a customer asks for help, she quickly answers the query and goes back to waiting. If a customer asks where is the washroom, she points in the right direction quickly and reverts to waiting. If a customer asks her for the price of a product, she quickly conveys the price and goes back to waiting. The point to note here is that there is only one customer service representative for the entire store servicing all customers. This model works exceptionally well when the representative is fast, and the answers to the queries are quick. Concurrency based on events works like this.

Now consider the situation where you have five customer service representatives in your store. As soon as a customer walks in, a representative is assigned exclusively to that customer. When another customer walks in, one more representative is picked from the pool and assigned to the customer. The critical point to note here is that there is a one to one relationship between the customer service representative and the customer. When one representative is servicing a customer, she does not bother about other customers; she is exclusive to that customer. Since our pool has five representatives, at most, we can serve only five customers at a time. What do we do when the sixth customer walks into the store? We can wait until one of the customers walks out or we can have a rule saying that a representative services a customer for a fixed period after which she will be assigned to another waiting customer. She is reassigned to the original customer once the time elapses. Concurrency based on threads works like this.

Coming back to the scenario wherein the sixth customer walks in. Now, we have to ask the sixth customer to wait until a representative is free. On the other hand, we have to wean away a representative from one of the existing customers and assign her to the new customer. When this happens, the customer who was initially being serviced by this representative has to wait. After the elapsed time, we have to assign the representative back to the original customer. When a lot of customers walk in, and you have a fixed no of representatives, quite a bit of coordination is needed to service all customers satisfactorily. In a computer, the CPU scheduler takes care of switching between tasks. Switching is a comparatively time-consuming operation and an overhead of the thread based concurrency model when compared to an event based one.

In the single representative scenario, what happens if one of the customers starts a long conversation with the representative? The representative will be stuck with the customer, and if other customers have queries, they will have to wait for the representative to finish the ongoing conversation. Also, what if one of the customers sends a representative on a long-running errand like fetching something from the depot a mile away? Until the representative returns, all other customers have to wait to get their queries resolved. One egregious customer can jeopardize all other customers and hold up the entire store operation.

Hence, when working with event based concurrency, it is essential not to:
1. Carry out CPU intensive tasks akin to having a long-running conversation with the representative.
2. Carry out blocking IO tasks similar to sending the representative to the depot.


NGINX and Redis are probably the most commonly used software that leverage event based concurrency. The workloads that these cater to are quick. Hence event based concurrency makes perfect sense here.

Taking the case of NGINX used as a reverse proxy, what does it do? Pick a client connection from the listen queue, do some operations on this and then forward it to the upstream server and then wait for the upstream to respond. While waiting for the upstream, NGINX can pick more client connections from the queue and repeat the above. When the upstream sends a response, it relies on this back to the client. Since all these are short-lived operations, this fits beautifully into an event based concurrency model. Good old Apache HTTP server creates a thread/process for each connection to do the same. The no of threads it has constraints apache. If the number of incoming requests is more than the number of threads in its pool, it has to deal with switching and coordination. NGINX does not have this overhead which makes it comparatively faster than Apache in real-world workloads. All of this is a bit simplistic and hand-wavy but should convey the idea.

Event based concurrency cannot leverage multiple CPU cores which all modern processors have. To do this, you create one event unit for each core usually called a worker. Also, most software that leverage event based concurrency adopt a hybrid model where they use event based concurrency for short-lived quick operations and off-load long-running tasks to a thread/process.

I have glossed over a lot of details and nuances to explain a complex topic like concurrency in simple terms. Treat this as a good starting guide to dig more into this fascinating world.

Startup Hiring

Hiring is an area which is usually given a lot of lip service but neglected in practice. Especially in a startup, where there is a shortage of workforce, hiring at all levels has a profound impact on the future of the organization. When compared to a big organization, the value proposition of a startup to its employees is concerning speed, freedom, and responsibility. But, when it comes to hiring, startups adopt the cookie cutter hiring strategy of big companies.

The regular interview process is biased toward extroverts and people who can speak well. In my opinion and observation, this does not correlate to someone who is good at what she does. I have been observing this since my college days. During campus placements, flamboyant and boisterous students made their way into companies while the quiet ones who were good at studies and who could get along with others were often overlooked. The same thing happens to a lesser extent with the regular interview process too.

Taleb says the following of doers.

For those who do things, it is harder to talk about what they do. Reality doesn’t care about talk. It is full of pauses and hesitations. Clear, non-hesitant speech is the domain of non-doers.

Interviewing is an intimidating experience, and not a lot of people excel in this, especially people who are good at what they do. Also, the kind of questions that are asked during an interview has hardly any resemblance to the day to day work. I have written about this before too. A big step in this direction is to craft an interview process that mimics your organization’s day to day tasks. The best way to do this is to have an assignment based interview process. You can create assignments that resemble tasks that you do on a daily basis and ask candidates to work their way through this. An approach of this sort takes the whole adversarial, and interrogative tone out of interviews and makes it an enjoyable experience.

An important aspect to take care while creating an interview process is to take subjectivity out of interviews. Who conducts the interview should not have a bearing on the outcome of the interview. Another thing to keep in mind is to subject all candidates of a particular level to the same questions. Following the above ensures that you are evaluating every candidate on the same yardstick. It is essential to do this to create a quantitative interview process. This sort of standardization also helps in creating a checklist of what you expect in answers to questions. If you have these covered, anyone can take anyone’s interview and evaluate candidates. Is it not ironic that startups claim to be data-driven, but when it comes to interviews, the whole data-driven approach takes a corner seat.


An example interview template would be to have an assignment that closely resembles a task related to work. Try to keep the assignment practical yet straightforward. A good candidate should not take more than 3 to 4 hours to complete the assignment. On submission; evaluate the assignment on code quality, conventions, comments, instructions to run, test cases, boundary condition handling, error handling, and documentation. Post that, have a phone screen with a bunch of questions that mostly have a one-word answer. Make this as practical and broad as possible. Try to touch all aspects of work from data structure choices to computational complexity to databases, web servers, etc. The idea here is that if someone is good at what they do, they should be able to answer these without any preparation. Then invite the candidate for face to face round where you ask the candidate to expand on the take-home assignment. Add a bit more complexity and see how the candidate fares. Finish off the entire process with a design round. I have presented a very rough template, tweak it, add more rounds and improvise to fit your needs; every organization is different.

Along with all the above, soft skills are critical. The sad fact is a lot of people do not know how to conduct interviews. I have heard of instances where interviewers were rude and outright obnoxious, not punctual, candidates not kept informed in advance of the interview schedule, candidates not being adequately attended during the face to face rounds, etc. All these play a significant role in shaping the image of the organization in the candidate’s mind.

In the talent market, big companies, as well as startups, are competing for the same piece of the pie. By adopting a hiring process similar to big companies, startups are not going to get an ace up their shoulder. What startups should be focussing on is coming up with a flexible hiring process that is unique to their organization. It is not easy to execute this in a big corporation but very much doable in a startup. Starting from how you engage with a candidate for the first time to the interview to the offer rollout process, everything is ripe for disruption. Differentiate yourself from others. An interview is a big opportunity for you to make a mark on the candidate, seize it right there.