Void

Idiomatic Code

What does writing idiomatic code mean?

Let us say you are using Python to populate a list with numbers.

One way to do this is


nos = []
for i in range(5):
    nos.append(i)

Another is


nos = [i for i in range(5)]

The second one is the idiomatic code. It leverages the Pythonic way of coding. Python is just an example here; every programming language has a philosophy and a method of doing things. Code that adheres to this is said to be idiomatic.

matrix-356024_640.jpg

The advantage of idiomatic code is that it takes less mental work and space to understand. The first one spans a couple of lines which translates to spending more grey cells to understand what is going on. The second one is concise and to the point. You end up expending less mental bandwidth to understand the second piece of code.

Also, idiomatic code is an inside speak between people in the know; like how a society functions with social norms and conventions, programming languages and communities achieve the same through idioms and conventions. You look at idiomatic code and you know instantly what this piece of code is trying to do, it is embedded in your subconscious like muscle memory.

Learning a programming language is not just about learning the syntax, it is more about learning the idioms and standard conventions of the language and the community behind it.

PS: You can populate a list in Python in a lot of different ways including using built-in library functions. Debating that is not the point of this post.

Advertisements

Sherlock Versus Calvin Ball

We can classify software development into:
1. Maintaining and enhancing existing software.
2. Software development from scratch.

Given a choice between the two, developers usually gravitate towards from scratch development. Developing something from scratch is an intensive creative work where you have the freedom to shape the product the way you see fit. Hence, it is pretty obvious why people prefer this. I draw a parallel here with Calvin Ball. For those of you not familiar with Calvin ball, it is a game that Calvin invented where he makes rules on the fly during the game. From scratch development is akin to Calvin Ball, you can create and amend rules on the fly. If you chose a framework and in the course of development you see it does not fit the bill, you have the freedom to swap it with something else. You are operating under a lot of degrees of freedom.

calvin_and_hobbes_original

Maintaining and enhancing existing software is more like solving a puzzle or playing a game with well laid out rules. Someone has already laid the foundation or in a lot of cases built the entire structure. You first have to expend time and effort in groking this and familiarising yourself with what is already there, only then you will be able to do something. A lot of times you need to get into the mind of the original developer and decipher things from her perspective. Working on code written by others is more like Sherlock Holme’s work. When you do changes and enhancements, you have to ensure what you are doing fits well into the existing framework. You are working in a constrained environment; you have to stick to the rules of the game. All this is as much or sometimes more challenging than developing software from scratch.

sherlock-3828991_640

Debugging is an acquired skill which carries over to all areas of development. When you troubleshoot code written by others, you become more attuned to add enough debugging information in the code you write. You will start empathizing with the person who will maintain your system in the future and ensure that person has enough data points to debug when things go wrong. It might as well happen that that future person is you only. Injecting debugging information and future proofing your project is a fundamental behavioral change that maintenance induces in you.

There is nothing wrong in preferring to create something from scratch, but it is imperative to have the second skill set under your belt. The real world requires more of type two work than type one. If from scratch development is all you have done till now, it is high time you challenge yourself with category two work. You will feel a bit frustrated and handcuffed in the beginning, but the way to approach it is like solving a mystery. If you see it that way, it becomes a fun and entertaining experience.

PS: Calvin and Hobbes image taken from Wikipedia.

Concurrency Models

We can roughly classify concurrency models into:
1. Thread based concurrency.
2. Event based concurrency.

Imagine that you run a store with only one customer service representative. As soon as a customer walks in, the customer service representative greets the customer with a quick hello saying – “If you need any help, give me a shout, and I will help you out.” She then waits for the customer to seek help. She aims to complete the interaction as soon as possible and wait for the next communication. When a customer asks for help, she quickly answers the query and goes back to waiting. If a customer asks where is the washroom, she points in the right direction quickly and reverts to waiting. If a customer asks her for the price of a product, she quickly conveys the price and goes back to waiting. The point to note here is that there is only one customer service representative for the entire store servicing all customers. This model works exceptionally well when the representative is fast, and the answers to the queries are quick. Concurrency based on events works like this.

Now consider the situation where you have five customer service representatives in your store. As soon as a customer walks in, a representative is assigned exclusively to that customer. When another customer walks in, one more representative is picked from the pool and assigned to the customer. The critical point to note here is that there is a one to one relationship between the customer service representative and the customer. When one representative is servicing a customer, she does not bother about other customers; she is exclusive to that customer. Since our pool has five representatives, at most, we can serve only five customers at a time. What do we do when the sixth customer walks into the store? We can wait until one of the customers walks out or we can have a rule saying that a representative services a customer for a fixed period after which she will be assigned to another waiting customer. She is reassigned to the original customer once the time elapses. Concurrency based on threads works like this.

Coming back to the scenario wherein the sixth customer walks in. Now, we have to ask the sixth customer to wait until a representative is free. On the other hand, we have to wean away a representative from one of the existing customers and assign her to the new customer. When this happens, the customer who was initially being serviced by this representative has to wait. After the elapsed time, we have to assign the representative back to the original customer. When a lot of customers walk in, and you have a fixed no of representatives, quite a bit of coordination is needed to service all customers satisfactorily. In a computer, the CPU scheduler takes care of switching between tasks. Switching is a comparatively time-consuming operation and an overhead of the thread based concurrency model when compared to an event based one.

In the single representative scenario, what happens if one of the customers starts a long conversation with the representative? The representative will be stuck with the customer, and if other customers have queries, they will have to wait for the representative to finish the ongoing conversation. Also, what if one of the customers sends a representative on a long-running errand like fetching something from the depot a mile away? Until the representative returns, all other customers have to wait to get their queries resolved. One egregious customer can jeopardize all other customers and hold up the entire store operation.

Hence, when working with event based concurrency, it is essential not to:
1. Carry out CPU intensive tasks akin to having a long-running conversation with the representative.
2. Carry out blocking IO tasks similar to sending the representative to the depot.

superhero-534120_640

NGINX and Redis are probably the most commonly used software that leverage event based concurrency. The workloads that these cater to are quick. Hence event based concurrency makes perfect sense here.

Taking the case of NGINX used as a reverse proxy, what does it do? Pick a client connection from the listen queue, do some operations on this and then forward it to the upstream server and then wait for the upstream to respond. While waiting for the upstream, NGINX can pick more client connections from the queue and repeat the above. When the upstream sends a response, it relies on this back to the client. Since all these are short-lived operations, this fits beautifully into an event based concurrency model. Good old Apache HTTP server creates a thread/process for each connection to do the same. The no of threads it has constraints apache. If the number of incoming requests is more than the number of threads in its pool, it has to deal with switching and coordination. NGINX does not have this overhead which makes it comparatively faster than Apache in real-world workloads. All of this is a bit simplistic and hand-wavy but should convey the idea.

Event based concurrency cannot leverage multiple CPU cores which all modern processors have. To do this, you create one event unit for each core usually called a worker. Also, most software that leverage event based concurrency adopt a hybrid model where they use event based concurrency for short-lived quick operations and off-load long-running tasks to a thread/process.

I have glossed over a lot of details and nuances to explain a complex topic like concurrency in simple terms. Treat this as a good starting guide to dig more into this fascinating world.

Startup Hiring

Hiring is an area which is usually given a lot of lip service but neglected in practice. Especially in a startup, where there is a shortage of workforce, hiring at all levels has a profound impact on the future of the organization. When compared to a big organization, the value proposition of a startup to its employees is concerning speed, freedom, and responsibility. But, when it comes to hiring, startups adopt the cookie cutter hiring strategy of big companies.

The regular interview process is biased toward extroverts and people who can speak well. In my opinion and observation, this does not correlate to someone who is good at what she does. I have been observing this since my college days. During campus placements, flamboyant and boisterous students made their way into companies while the quiet ones who were good at studies and who could get along with others were often overlooked. The same thing happens to a lesser extent with the regular interview process too.

Taleb says the following of doers.

For those who do things, it is harder to talk about what they do. Reality doesn’t care about talk. It is full of pauses and hesitations. Clear, non-hesitant speech is the domain of non-doers.

Interviewing is an intimidating experience, and not a lot of people excel in this, especially people who are good at what they do. Also, the kind of questions that are asked during an interview has hardly any resemblance to the day to day work. I have written about this before too. A big step in this direction is to craft an interview process that mimics your organization’s day to day tasks. The best way to do this is to have an assignment based interview process. You can create assignments that resemble tasks that you do on a daily basis and ask candidates to work their way through this. An approach of this sort takes the whole adversarial, and interrogative tone out of interviews and makes it an enjoyable experience.

An important aspect to take care while creating an interview process is to take subjectivity out of interviews. Who conducts the interview should not have a bearing on the outcome of the interview. Another thing to keep in mind is to subject all candidates of a particular level to the same questions. Following the above ensures that you are evaluating every candidate on the same yardstick. It is essential to do this to create a quantitative interview process. This sort of standardization also helps in creating a checklist of what you expect in answers to questions. If you have these covered, anyone can take anyone’s interview and evaluate candidates. Is it not ironic that startups claim to be data-driven, but when it comes to interviews, the whole data-driven approach takes a corner seat.

hiring-3531130_640

An example interview template would be to have an assignment that closely resembles a task related to work. Try to keep the assignment practical yet straightforward. A good candidate should not take more than 3 to 4 hours to complete the assignment. On submission; evaluate the assignment on code quality, conventions, comments, instructions to run, test cases, boundary condition handling, error handling, and documentation. Post that, have a phone screen with a bunch of questions that mostly have a one-word answer. Make this as practical and broad as possible. Try to touch all aspects of work from data structure choices to computational complexity to databases, web servers, etc. The idea here is that if someone is good at what they do, they should be able to answer these without any preparation. Then invite the candidate for face to face round where you ask the candidate to expand on the take-home assignment. Add a bit more complexity and see how the candidate fares. Finish off the entire process with a design round. I have presented a very rough template, tweak it, add more rounds and improvise to fit your needs; every organization is different.

Along with all the above, soft skills are critical. The sad fact is a lot of people do not know how to conduct interviews. I have heard of instances where interviewers were rude and outright obnoxious, not punctual, candidates not kept informed in advance of the interview schedule, candidates not being adequately attended during the face to face rounds, etc. All these play a significant role in shaping the image of the organization in the candidate’s mind.

In the talent market, big companies, as well as startups, are competing for the same piece of the pie. By adopting a hiring process similar to big companies, startups are not going to get an ace up their shoulder. What startups should be focussing on is coming up with a flexible hiring process that is unique to their organization. It is not easy to execute this in a big corporation but very much doable in a startup. Starting from how you engage with a candidate for the first time to the interview to the offer rollout process, everything is ripe for disruption. Differentiate yourself from others. An interview is a big opportunity for you to make a mark on the candidate, seize it right there.

Deviation From Expected

Someone sitting at a distance asks for the water bottle near me. I pick up the bottle and throw it at that person. Surprisingly, the cap is not screwed. Water splashes all over. When a bottle has its cap on, we usually expect it to be tightly screwed. When something deviates from the expected, unless there is an indication saying so, it creates trouble and confusion.

arrows-1834859_640

The same principle applies to systems and application design. For example, let us say that you have a development server where someone is running a production cron job. Since this is a development server, someone might take it down for experimentation. No one expects the non-availability of a development server to have untoward consequence.

Whenever you deviate from the expected, ensure you scream from the top of your voice so that no one misses it. Documentation, common conventions and putting in the right processes are some of the ways to mitigate this. The best is not to do it. Whatever you are doing, it always helps to ask, is this a deviation from the expected? If I am not part of the inner circle, would I expect it to be like this?

New Feature Efficacy

You have an established product, and you introduce a radically new/different feature. You are very enthusiastic, but the metrics show users are hardly using this.

There could be two reasons:
1. Users do not see a value.
2. Could be resistance to newness.

sparkler-839831_640

 

It is essential to zero in on the above to decide whether to put in more effort into the feature. If this is indeed resistance to newness, there might be multiple ways to nudge your users towards the feature gently.

A straightforward way to figure this out is to measure the stickiness of the feature. Of the small percentage of your users who do interact with the features, how many of them come back to it subsequently, i.e. once people are acquainted with the feature, do they come back to it later? If you see stickiness in this cohort, it is a good indicator that the feature is of value, you are doing a lousy job in educating your users and leading them to it. If not, it is time to cull the feature and invest that time and energy in something else.

Resolving Disagreements

When you disagree with something, either you do it because you think your idea is better or you want to keep your ego intact. Let us ignore the latter and focus on the former where the intention is to let the best idea win. When a group of people sit down and try to resolve disagreements, many a time, it goes nowhere. Sometimes you get this strange feeling of things going around in a circle. This is due to whataboutery and shifting goal posts. You start with an objective, as the discussion progresses, statements lead to counter statements and at the end, no one knows what they are trying to resolve.

antagonism-1940188_640

One simple hack to keep discussions on track is to write things down. Project a shared document where you note the objective and the point of contention. Whenever matters go awry, point people to the shared document. This helps everyone involved to stay focused and not to shift goal post as the discussion progresses.

Irrespective of how rational and mature one is, when someone disagrees with something that one believes to be true, one tends to become defensive and shift goal post without truly being aware of it. Writing things down makes one aware of this and helps course correct.

My View

I was looking at Jimi wallets online. Someone peeked at my laptop and asked what is it? I explained it is a rugged waterproof wallet. The other person’s immediate reaction was – Why would anyone need this? This person has never faced the fury of rain while cycling outside.

paint-2985569_640

Whenever I explain startups spending marketing dollars to acquire users even when they are not generating any profit, I get a dazed look from people coming from a traditional business background. It is difficult for them to grasp the concept of betting on explosive future growth at the expense of today.

Phil Knight, in his book Shoe Dog, writes a lot about how his bank was asking him to preserve capital when all he wanted to do was grow Nike at all costs during its fledgling years.

A lot of prolific US citizens opinionated that Trump had a naught chance at US presidency. The same goes for Brexit.

What is common in all these situations is a difficulty in viewing the world from a lens not tarred by our own experiences. Even if you want to do this, it is extremely difficult to implement because you do not know where to draw the line. Tomorrow, if someone tells you that she has invented the perpetual motion machine, what do you do? Do you dismiss it outright or be skeptical of this person’s claim?

In all these scenarios you have to do suspend your rational mind and view things from a radically incongruent perspective. It is easy to write this but extremely difficult to implement.

Ingratitude

Zoho’s domain was inaccessible for a while. This is an embarrassing event for a software organization.

Whenever I hear of events like this, I am reminded of a couple of pages in “The Black Swan“. Taleb calls it “A new kind of ingratitude”.

The idea presented by Taleb essentially boils down to a person who takes steps to prevent something catastrophic from happening. Since that person has taken steps to prevent the catastrophe, the catastrophe never occurs. Thus the person never gets his due and dies a silent hero.

 

This is a very fascinating thought that keeps repeating in all aspects of life. Whenever it floods, we make a big deal of politicians who fold their sleeves and get into action. What about that politician who took the necessary steps to prevent flooding?

Whenever there is a production issue at work and a team goes out of their way to put out the fire, that team is lauded. What about those teams that took steps to prevent something like this from occurring in the first place?

Software security is one big area that falls in this category. If you have a great security team, life would go on humming silently. You need to have the right tech leadership to recognize this otherwise it falls bang into ingratitude category.

This is a very obvious thought but Taleb has done a great job of giving structure to this idea. If you keep your eyes open, you see this happening around you all the time.

Micro Versus Macro Solutions

Imagine a person who walks from her home to office. Frequently she is late to work as she takes time to cover the distance. She wants to improve her pace. She goes to a walking expert to get tips on increasing her walking speed.

thought-2123970_640

One solution to the problem is to use some other means of transportation instead of walking. If you go to a walking expert, you are going to get tips on improving your walking speed. The expert is not going to ask you to forego walking and use a different mode of transportation. Also, if you are deeply attached to the idea of walking, you might not think of a solution beyond walking. Improving your walking speed is a micro solution whereas using some other means of transportation is a macro solution.

The above is a contrived example but something we come across in our professional and personal lives, both as solution givers as well as ones facing a problem. Programmers sometimes try to optimize the hell out of a piece of code while the right approach might be to chuck the code and use something else. Organisations try to nail down a process to the last mile while a sensible solution might be to completely do away with the process.

We lean towards micro solutions when we are either deeply entwined in a problem or are the domain expert in that particular area. In these situations, we tend to think within the bounds of a problem and not outside.

When you come up with a solution, bracket it as micro or macro. Being aware is the first step towards becoming better at anything. Also, outside view helps. Find someone who is not an expert in the domain or one who is not acutely aware of the problem. Run your solution through them. They might lead you to a macro solution or make you aware that what you have is a micro solution. Taking time and mind off a problem helps too like how Archimedes had his eureka moment.

Last but not the least, take a walk.