Void

Build versus buy

Consciously or unconsciously, as software engineers, we perennially take build versus buy decisions. It might be as trivial as copy pasting code from somewhere versus racking up our brains to write our own; using an already available library or writing one from scratch; using a time tested framework against designing one; building a piece of software internally as compared to buying one.

backdrop-21534_640

The way we account for the build versus buy decision varies. Some of the frivolous reasons for building in-house are NIH syndrome, hubris, and planning fallacy.  We generally tend to overemphasize our expertise, knowledge, and capability which naturally lead to building internally. Also, we underestimate the amount of work involved in creating software, only once we get our feet wet does the reality set in. A very valid reason for building internally is cost but when accounting for cost, we usually overlook the hidden cost of building software. Buying a software has an upfront monetary cost whereas by building internally we pay in the form of opportunity cost, talent cost, feature cost etc.

Build versus buy arguments are reminiscent of qualitative speak like “This is not our core expertise, we should be concentrating on solving our business problems”, “This is going to cost us a bomb, let us build in-house”, “We should have had this yesterday, building in-house will cost us another 6 months”, “Will that external product be able to handle our scale”, “Can we trust them with our data” etc. In most cases, build versus buy decisions are qualitative, it is not an easy exercise to quantify them.

When evaluating a product that is already out in the market versus building something similar, a cardinal mistake people commit is mapping features one to one. Even though having 100 different features looks rosy and attractive, usually we end up using only a select few. Instead of trying to match an external product feature to feature, scope out the features that you need or would probably use and then estimate the effort. Another is refinement. An external product will be refined and polished, but you may not need the same level of refinement. For example, you might not need a web interface for the product, a terminal interface would work fine for your use case.

When faced with the build versus buy decision, asking the following help:

  1. Is this my core expertise or is it something I can let others do for me?
  2. What is the cost of getting this done externally versus hiring people to build this?
  3. How much control do I need over this i.e can I live with some error, downtime or opaqueness?
  4. Will I really do a better job building this internally?
  5. Do I have the expertise needed to build this?
  6. Once I build this, will I be able to maintain and enhance?
  7. What is the opportunity cost of having this sometime in the future versus having it now?

Use the answers to the above as a beacon for the build versus buy decision.

Advertisements

Context

Sapiens, the book, gives an amazing perspective of the context in which today’s religions, society, social practises etc evolved and how in the current context, a lot of these are irrelevant. One of the core ideas presented in the book is that humanity, during the course of evolution, favoured social stability over individual liberty because trust was necessary for human advancement and the basis of this trust was a common belief in the same God, social practises, etc. Today, science and technology as well as robust social institutions and concepts like democracy, liberalism, capitalism etc form the basis of trust. In short, the context does not hold true today but we still continue with the practises. We can draw parallel with this to the way organisations blindly adopt technology, frameworks and processes from other places without understanding the context around which these evolved.

question-mark

During the course of my professional life, I have heard a lot along these lines; Netflix does micro services, let us also do that; Google and Facebook subject interview candidates to intensive data structure and algorithm questions, let us practise the same. Adopting something without understanding the context is a recipe for disaster. As a thought experiment, let us take micro services. Micro services evolved in tech organisations with complex products handled by multiple independent teams craving for control and autonomy without stepping on each others toes. Also, for micro services to succeed, you need to put in a lot of effort into alerting, monitoring, orchestration, devops etc. Without these, micro services is a bomb waiting to explode.

When borrowing technology or processes from other places, a lot of effort needs to be put into first understanding the context around which these evolved in the said place and also what is needed to make these work. Blind adoption usually leads to unmitigated disaster.

Look ma, no schema

Due to the plethora of NoSQL database choices available, schema less is a tantalising option these days . While starting, NoSQL databases sound attractive but reality sets in when the maintenance problems start creeping up later.

nosqlpng

If you are doing anything with data, you need to know the schema of that data. What this boils down to is whether this schema is explicit or implicit. In relational databases, schema is explicit and well documented whereas in NoSQL databases, schema is implicit i.e it is maintained in code. All the goodies that you get with relational databases like integrity checks, constraints etc should be taken care of in code.

Products go through multitude of changes, features get added and axed and in order to accommodate this, schema has to change accordingly. In relational databases, carrying out schema changes on tables with large amounts of data is a daunting problem which requires considerable planning and effort. This problem is not present in NoSQL databases, but again, you end up managing this in code. In order to handle multiple versions of the schema, code gets littered with if else statements. This gets even more messy as this is spread across wherever data is being handled. In relational databases, you handle schema changes once and for all, in NoSQL databases, you continue to do it in code long after the change.

Multiple people work on code at different points of time, people join and leave teams but code lives on. This is when the maintenance problem starts raising it’s ugly head. A big reason a lot of startups opt for NoSQL databases is due to the constant flux in these organisations. Requirements change frequently due to which a stable schema design becomes next to impossible. In this landscape, NoSQL databases look like God’s gift to mankind. If your company does not fall in this bucket and you find yourself tempted to use a NoSQL database, take a good hard look at the problem and ensure that it is worthy of a NoSQL database. There are problems which are better solved with a NoSQL database but these are far and few.

10 things you did not know about Vietnam

Sorry, could not help with the snarky title. We recently took a vacation to Vietnam and this is a collection of unconnected thoughts and observations about the country and our journey.

poster-nva-1970-poster-usa-hueys

During our travel, we visited Ho Chi Minh City, Da Nang, Hoi An, Hue, Hanoi and Halong Bay. In all these places, infrastructure was amazing, almost on par with western countries. In Hanoi and Ho Chi Minh city, there are abundant parks and public spaces. They are very well maintained too.

There are walking streets everywhere. Most of these are regular streets which get converted to pedestrian only movement during the night. You will spot umpteen eating and drinking options in these streets. These walking streets are not as scandalous as the ones in Thailand.

Food is a big part of the culture, street food is abundant. It is a paradise for non vegetarians, not so much if you are a vegetarian. Noodle soup(Pho) and banh mi are two of the most popular delicacies. Banh mi means bread in Vietnamese, usually it is sold stuffed with veggies and/or meat.  Banh mi is a relic of the French past like our Pav(Vada Pav). There is a version of noodle soup called hot pot which is cooked right on your table which is delicious. People watching seems to be a big thing, lots of cafes lined with chairs facing the street. I am in love with Vietnamese coffee, it is black coffee with condensed milk, tastes amazing.

Vietnamese seem to have a thing for “The North Face”. Fake North Face products abundant on the streets. Giving them company are Nike, Superdry and Under Armour. Adidas, Reebok and Puma are conspicuously absent.

Traffic is chaotic. Two wheelers zipping past everywhere breaking all rules. You can rent two wheelers in all the cities, which we did. There is neither license check nor passport deposit, it is a honour based system. Toyota is everywhere. We checked with a taxi driver as to why Toyota seems to be omnipresent. Since there are so many two wheelers, scratches and skirmishes are common it seems. Hence Vietnamese prefer Toyota which is easy to maintain and spare parts are economically priced.

Communication is a challenge. Google translate was a life saver. It was much easier to just type in translate and show to people.

Vietnam is blessed with abundant water, water bodies everywhere, in cities as well as countryside. From the 60th floor of Lotte tower in Hanoi, we could spot innumerable lakes.

Museums and historical places scream jingoism. Glorification of Vietnamese struggle against the French and Americans seem a bit in your face.

Vietnam seems to be on the path to becoming an economic powerhouse. Signs are everywhere, from the burgeoning constructions to commercial towers competing for the most number of floors title.

Vietnamese are a friendly lot, they try their best to make things happen for you. We have hardly had bad experiences with people apart from a parking attendant in Hanoi who lost his cool due to the language barrier.

All hotels including our room in the cruise had slippers as part of standard accessories which was new. Also, in some of the public toilets, you had to change your footwear to slippers present there, a nice hack to keep the toilets clean.

Indian cinema and tv shows seem to be very popular. Our van driver was glued to Telugu movie scenes on his phone and a taxi driver told us he is a big fan of Hindi movies.

Internal air transport is economical, couple of low cost carrier options.

Souvenirs and trinkets are sold everywhere. Propaganda posters are really cool. Vietnamese seem to have a soft corner for Tintin, various posters and fridge magnets of “Tintin in Vietnam” or “Tintin in Saigon”.

Vietnamese script is English with accents. A French person created the script for Vietnamese. Original Vietnamese script is long lost it seems.

Vietnamese seem to love kids, we had three toddlers in our group, they were greeted with chocolates and souvenirs by strangers almost everywhere.

From a cultural perspective, Vietnam and India share a lot in common. Vietnam has a rich cultural past, dating back centuries. In fact, at one point of time, Hinduism was the dominant religion in Vietnam.

While coming back, we took Malindo air. Malindo air was a pleasant surprise, friendly stewards, ample leg room and in flight meals.

When we told people that we are going to Vietnam, all asked us why and we did not have an answer. It was well worth it though.

Shoe Dog

nike

Phil Knight, founder of Nike, pens down the early days of Nike in this part memoir, part biography and part business book, “Shoe Dog”. The book starts with Phil Knight’s idea to sell Japanese made sneakers in USA, his business school thesis supporting the same, his journey to Japan to ink the deal and the subsequent world travel post which he dedicates his life to building Nike and the struggles thereafter.

In today’s interconnected world where communication is instant and automation is the norm, reading about running a business when these were non existent is fascinating. Nike is closer to today’s startup than to any old school business as the company prioritised growth and expansion over security and capital accumulation. His anecdotes regarding dearth of capital, constant tussle with the bank on rapid expansion, the bank asking him to scale down his ambition and focus on capital accumulation gives insights on the business mentality in the yesteryears.

Throughout the book, Phil Knight talks about his hands off management style. He styles his management on the below quote from Antoine de Saint-Exupery.

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

Profile of the Nike founding team is far fetched from the general imagery of traditional old school businessmen. They seem to be misfits who found a home in Nike, at least Phil Knight thinks so. No wonder the whole company rallied around doing iconoclastic things that many business would not even think of during those years.

All in all, the book is a great read. I was hoping to read more about Nike’s marketing strategy and how it evolved, but there is hardly any mention of this. Probably for another book I guess.

Reading Shoe Dog brought back fond memories of reading “Made in Japan” years back.

Unit test – purist versus practical

building-blocks-615239_640

Whenever you ask a question on unit testing in a forum, there is always that one person whose only job is to point out that what you are doing is not unit testing but integration testing. It is important to know this difference but it is more important to not lose sight of the goal. Also, you need to adopt a terminology that works for you and your team, rather than what purists think or say.

In absolute terms, if a test depends on anything that is not in your control, then it is not a unit test. For example, if a method that you are testing uses a public function or a function from an included library or a database or an external API, then it is not a unit test but an integration test. For a test to qualify as a unit test, you need to mock all these dependencies and get them under control, only then you can claim your test as unit test. Now that we have the purists happy, let us move to a more practical world view.

When a regular joe developer refers to a test as unit test, what she means is she is trying to test a functionality in a massive gnarly application that she thinks is a small independent unit. This unit might have some components that is not under her control. A regression test would test how these units behave when interconnected. Instead of debating whether what she is doing is unit testing or integration testing, a better discussion in my opinion is trying to figure out what is the intention of the test and what needs to be controlled and not. Figuring this out and working on achieving this will add more value than debating whether something is a unit test or not.

Fighting change

Ninja Fighter Sword

In my new workplace, I was given a brand new shiny MacBook Pro. My first reaction was to ask for a Ubuntu laptop. My brain justified by giving several reasons, it is developer friendly, softwares that you run on a server can be run as is in ubuntu, etc etc. I was almost about to voice this opinion, but system 2 took over from system 1. It started asking questions along the lines of is this really the reason you want a Ubuntu machine or are you just trying to avoid the unfamiliar. You have not used a Mac before and are you trying to run away from something new? The two debated for sometime and settled on giving the Mac a chance. So far, the experience has been wonderful and I am learning some cool things like using gestures for different actions etc.

Whenever something new and unfamiliar comes across us, natural instinct for most of us is to fight against it. Take a step back, analyse whether this is the primal part of the brain trying to fight against the unfamiliar or you really have a valid reason not to.

Altruism FTW

Have you observed the way Google maps asks for info about local joints and places? They word it in such a manner that it sounds like you are helping others to make an informed decision along the lines of “Give us more info to help others”. What they are doing in effect is appealing to the altruism in all of us to generate more info to make their product better.

I think this is a great way to ask for more data in this world of user generated content. Instead of asking to review a restaurant how about wording it as “Review this place so that others can discover great food”. Instead of asking people to rate your app how about saying “Help your friends discover the app on playstore, rate us”. It would be interesting to A/B test this and see the result.

A little extra effort

I was sauntering on Church Street and came across a used book store. It had been long since I had been to a physical book store, hence ventured in. I started browsing around. I have been wanting to read Shoe Dog for quite sometime, asked the proprietor did he have a used copy of the book. He answered in the negative and got back to whatever he was doing before. I continued my aimless browsing and got out.

Down the road, there was another used book store. Again got in and repeated the question. The person searched, said no, but he put a new copy of the book in my hand and said he can give me a 20% discount. I am sure he knew before hand that he did not have a used copy of the book, searching was just a ruse. I made the purchase.

We tend to neglect the impact of putting that little extra in. Sometimes all it takes is a little extra effort for a great outcome.

Selfie

Let us say that you want to execute a job periodically, what comes to your mind first? If you are familiar with Linux, I can hear your screaming cron. Well, no doubt about that, cron is great, but what if I told you there is another approach which you can take to execute periodic jobs? Our good old continuous integration server Jenkins can supplant cron as a tool to execute periodic jobs and it kicks ass in doing so.

What makes Jenkins such a gem for executing periodic jobs?

1. You get a great web front end which is comfortably accessible from the browser.

2. The front end gives you a complete history of the previous runs with detailed information like when did the last execution occur, how long it took, what was the output during this execution, historic trend of the execution time and other diagnostic information.

buildHistory

buildTimeTrend

3. You can leverage the Jenkins plugin eco system and do some nifty things. For example, you can use log parser plugin to parse the execution output and alert if a specific format is found in the output. The great part here is that your job need not have alerting logic baked in, your job concentrates on doing what it does best, let Jenkins take care of the rest.

4. You can configure regular Jenkins build rules like alerting on execution failure, preventing subsequent executions if the current one fails, etc.

5. You can chain multiple jobs and the chain is very obvious thanks to the great Jenkins UI.

All this is great, but one problem I faced with Jenkins is that you cannot have a Job call itself recursively with a delay in between, you have to schedule the job execution using cron expression. The difference is subtle, but there are implications of this limitation which I will expound with an example. Let us say that I have a job which ideally should run every 15 minutes, but sometimes this job execution takes more than 15 minutes to complete, in that case what happens is, job executions queue up and fire successively one after the other. The way I want this scenario to pan out is, once the execution finishes, it should wait for 15 minutes before the next execution starts. I could not find a way to do this in Jenkins and hence selfie was born.

Selfie is a Jenkins build trigger plugin which lets a project to trigger itself after a configured delay. The plugin appears as one of the build triggers while configuring a new project.

selfie

This is my first attempt at writing a Jenkins plugin, pull requests and code reviews are more than welcome.