Void

Context

Sapiens, the book, gives an amazing perspective of the context in which today’s religions, society, social practises etc evolved and how in the current context, a lot of these are irrelevant. One of the core ideas presented in the book is that humanity, during the course of evolution, favoured social stability over individual liberty because trust was necessary for human advancement and the basis of this trust was a common belief in the same God, social practises, etc. Today, science and technology as well as robust social institutions and concepts like democracy, liberalism, capitalism etc form the basis of trust. In short, the context does not hold true today but we still continue with the practises. We can draw parallel with this to the way organisations blindly adopt technology, frameworks and processes from other places without understanding the context around which these evolved.

question-mark

During the course of my professional life, I have heard a lot along these lines; Netflix does micro services, let us also do that; Google and Facebook subject interview candidates to intensive data structure and algorithm questions, let us practise the same. Adopting something without understanding the context is a recipe for disaster. As a thought experiment, let us take micro services. Micro services evolved in tech organisations with complex products handled by multiple independent teams craving for control and autonomy without stepping on each others toes. Also, for micro services to succeed, you need to put in a lot of effort into alerting, monitoring, orchestration, devops etc. Without these, micro services is a bomb waiting to explode.

When borrowing technology or processes from other places, a lot of effort needs to be put into first understanding the context around which these evolved in the said place and also what is needed to make these work. Blind adoption usually leads to unmitigated disaster.

Advertisements

Look ma, no schema

Due to the plethora of NoSQL databases available, schema-less is a tantalising option these days. While starting on a new project, NoSQL databases look attractive but reality sets in when the maintenance problems start creeping up later.

nosqlpng

 

If you are doing anything with data, you need to know the schema of that data. What this boils down to is whether this schema is explicit or implicit. In relational databases, the schema is explicit and well documented whereas in NoSQL databases, the schema is implicit i.e it is maintained in code. All the goodies that you get with relational databases like integrity checks, constraints etc should be taken care of in code.

Products go through a multitude of changes, features get added and axed and in order to accommodate this, the schema has to change accordingly. In relational databases, carrying out schema changes on tables with humungous data is a daunting problem which requires considerable planning and effort. This problem is not present in NoSQL databases, but again, you end up managing this in code. In order to handle multiple versions of the schema, code gets littered with if else statements. This gets even more messy as this is spread across the code base in multiple places wherever data is being handled. In relational databases, you handle schema changes once and for all, in NoSQL databases, you continue to do it in code long after the change.

Multiple people work on code at different points in time, people join and leave teams but code lives on. This is when the maintenance problem starts raising its ugly head.

A big reason a lot of startups opt for NoSQL databases is due to the constant flux in these organisations. Requirements change frequently due to which a stable schema design becomes next to impossible. In this landscape, NoSQL databases look like God’s gift to mankind. If your company does not fall in this bucket and you find yourself tempted to use a NoSQL database, take a good hard look at the problem and ensure that it is worthy of a NoSQL database. There are problems which are better solved with a NoSQL database but these are far and few.

10 things you did not know about Vietnam

Sorry, could not help with the snarky title. We recently took a vacation to Vietnam and this is a collection of unconnected thoughts and observations about the country and our journey.

poster-nva-1970-poster-usa-hueys

During our travel, we visited Ho Chi Minh City, Da Nang, Hoi An, Hue, Hanoi and Halong Bay. In all these places, infrastructure was amazing, almost on par with western countries. In Hanoi and Ho Chi Minh city, there are abundant parks and public spaces. They are very well maintained too.

There are walking streets everywhere. Most of these are regular streets which get converted to pedestrian only movement during the night. You will spot umpteen eating and drinking options in these streets. These walking streets are not as scandalous as the ones in Thailand.

Food is a big part of the culture, street food is abundant. It is a paradise for non vegetarians, not so much if you are a vegetarian. Noodle soup(Pho) and banh mi are two of the most popular delicacies. Banh mi means bread in Vietnamese, usually it is sold stuffed with veggies and/or meat.  Banh mi is a relic of the French past like our Pav(Vada Pav). There is a version of noodle soup called hot pot which is cooked right on your table which is delicious. People watching seems to be a big thing, lots of cafes lined with chairs facing the street. I am in love with Vietnamese coffee, it is black coffee with condensed milk, tastes amazing.

Vietnamese seem to have a thing for “The North Face”. Fake North Face products abundant on the streets. Giving them company are Nike, Superdry and Under Armour. Adidas, Reebok and Puma are conspicuously absent.

Traffic is chaotic. Two wheelers zipping past everywhere breaking all rules. You can rent two wheelers in all the cities, which we did. There is neither license check nor passport deposit, it is a honour based system. Toyota is everywhere. We checked with a taxi driver as to why Toyota seems to be omnipresent. Since there are so many two wheelers, scratches and skirmishes are common it seems. Hence Vietnamese prefer Toyota which is easy to maintain and spare parts are economically priced.

Communication is a challenge. Google translate was a life saver. It was much easier to just type in translate and show to people.

Vietnam is blessed with abundant water, water bodies everywhere, in cities as well as countryside. From the 60th floor of Lotte tower in Hanoi, we could spot innumerable lakes.

Museums and historical places scream jingoism. Glorification of Vietnamese struggle against the French and Americans seem a bit in your face.

Vietnam seems to be on the path to becoming an economic powerhouse. Signs are everywhere, from the burgeoning constructions to commercial towers competing for the most number of floors title.

Vietnamese are a friendly lot, they try their best to make things happen for you. We have hardly had bad experiences with people apart from a parking attendant in Hanoi who lost his cool due to the language barrier.

All hotels including our room in the cruise had slippers as part of standard accessories which was new. Also, in some of the public toilets, you had to change your footwear to slippers present there, a nice hack to keep the toilets clean.

Indian cinema and tv shows seem to be very popular. Our van driver was glued to Telugu movie scenes on his phone and a taxi driver told us he is a big fan of Hindi movies.

Internal air transport is economical, couple of low cost carrier options.

Souvenirs and trinkets are sold everywhere. Propaganda posters are really cool. Vietnamese seem to have a soft corner for Tintin, various posters and fridge magnets of “Tintin in Vietnam” or “Tintin in Saigon”.

Vietnamese script is English with accents. A French person created the script for Vietnamese. Original Vietnamese script is long lost it seems.

Vietnamese seem to love kids, we had three toddlers in our group, they were greeted with chocolates and souvenirs by strangers almost everywhere.

From a cultural perspective, Vietnam and India share a lot in common. Vietnam has a rich cultural past, dating back centuries. In fact, at one point of time, Hinduism was the dominant religion in Vietnam.

While coming back, we took Malindo air. Malindo air was a pleasant surprise, friendly stewards, ample leg room and in flight meals.

When we told people that we are going to Vietnam, all asked us why and we did not have an answer. It was well worth it though.

Shoe Dog

nike

Phil Knight, founder of Nike, pens down the early days of Nike in this part memoir, part biography, part business book; “Shoe Dog”. The book starts with Phil Knight’s idea to sell Japanese made sneakers in his home country USA, his business school thesis supporting the same, his journey to Japan to ink the deal and the subsequent travel around the world post which he dedicates his life to building Nike and the struggles thereafter.

In today’s interconnected world where communication is instant and automation is the norm, reading about running a business when these were non-existent is fascinating. Nike during it’s founding days was closer to today’s new age startup than to any old school business as the company prioritized growth over security and cash in the bank. His anecdotes regarding the dearth of capital, constant tussle with the bank on rapid expansion, his bank asking him to scale down his ambition and focus on capital accumulation; give insights on the business mentality of the yesteryears.

Throughout the book, Phil Knight talks about his hands off management style. He styles his management on the below quote from Antoine de Saint-Exupery.

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

Profile of the Nike founding team is far-fetched from the general imagery of traditional old-school businessmen. They seem to be misfits who found a home at Nike, at least Phil Knight thinks so. No wonder the whole company rallied around doing iconoclastic things that many businesses would not even think of during those days.

All in all, the book is a great read. I was hoping to read more about Nike’s marketing strategy and how it evolved, but there is hardly any mention of this. Probably for another book I guess.

Reading Shoe Dog brought back fond memories of reading “Made in Japan” years back.

Unit test – purist versus practical

building-blocks-615239_640

Whenever you ask a question on unit testing in a forum, there is always that one person whose only job is to point out what you are doing is not unit testing but integration testing. It is important to know the difference but it is more important to not lose sight of the goal, ensuring a reliable and a bug-free application. Also, you need to adopt a terminology that works for you and your team, rather than what purists think or say.

In absolute terms, if a test depends on anything that is not in your control, it is not a unit test. For example, if a method that you are testing uses a public function, a method from an included library, database or an external API, it is not a unit test but an integration test. For a test to qualify as a unit test, you need to mock all these dependencies and get them under your control, only then you can claim your test as a unit test. Now that we have the purists happy, let us move to a more practical worldview.

When a regular joe developer refers to a test as a unit test, what she means is, she is trying to test a functionality in a massive gnarly application that she thinks is a small independent unit. This unit might have some components that are not under her control. Instead of debating whether she is unit testing or integration testing, a better discussion is trying to figure out what is the intention of the test, what needs to be controlled/mocked and not. Helping her to figure this out and achieving this will add more value than debating whether a test is a unit test or integration test.

Fighting change

Ninja Fighter Sword

In my new workplace, I was given a brand new shiny MacBook Pro. My first reaction was to ask for an Ubuntu laptop. My brain justified by giving several reasons, it is developer friendly, software that you run on a server can be run as is in Ubuntu, etc. I was almost about to voice this opinion, but system 2 took over from system 1. It started asking questions along the lines of is this really the reason you want an Ubuntu machine or are you just trying to avoid the unfamiliar? You have not used a Mac before and are you trying to run away from something new? The two debated for some time and settled on giving the Mac a chance. So far, the experience has been wonderful and I am learning some cool things like using gestures for different actions etc.

Whenever something new and unfamiliar comes across, the natural instinct for most of us is to fight it. Take a step back, analyse whether this is the primal part of the brain trying to fight against the unfamiliar or you really have a valid reason not to.

Altruism FTW

Have you observed the way Google maps asks for info about local joints and places? They word it in such a manner that it sounds like you are helping others to make an informed decision along the lines of “Give us more info to help others”. What they are doing in effect is appealing to the altruism in all of us to generate more info to make their product better.

I think this is a great way to ask for more data in this world of user generated content. Instead of asking to review a restaurant how about wording it as “Review this place so that others can discover great food”. Instead of asking people to rate your app how about saying “Help your friends discover the app on playstore, rate us”. It would be interesting to A/B test this and see the result.

A little extra effort

I was sauntering on Church Street and came across a used book store. It had been long since I had been to a physical book store, hence ventured in. I started browsing around. I have been wanting to read Shoe Dog for quite sometime, asked the proprietor did he have a used copy of the book. He answered in the negative and got back to whatever he was doing before. I continued my aimless browsing and got out.

Down the road, there was another used book store. Again got in and repeated the question. The person searched, said no, but he put a new copy of the book in my hand and said he can give me a 20% discount. I am sure he knew before hand that he did not have a used copy of the book, searching was just a ruse. I made the purchase.

We tend to neglect the impact of putting that little extra in. Sometimes all it takes is a little extra effort for a great outcome.

Selfie

Let us say that you want to execute a job periodically, what comes to your mind first? If you are familiar with Linux, I can hear your screaming cron. Well, no doubt about that, cron is great, but what if I told you there is another approach which you can take to execute periodic jobs? Our good old continuous integration server Jenkins can supplant cron as a tool to execute periodic jobs and it kicks ass in doing so.

What makes Jenkins such a gem for executing periodic jobs?

1. You get a great web front end which is comfortably accessible from the browser.

2. The front end gives you a complete history of the previous runs with detailed information like when did the last execution occur, how long it took, what was the output during this execution, historic trend of the execution time and other diagnostic information.

buildHistory

buildTimeTrend

3. You can leverage the Jenkins plugin eco system and do some nifty things. For example, you can use log parser plugin to parse the execution output and alert if a specific format is found in the output. The great part here is that your job need not have alerting logic baked in, your job concentrates on doing what it does best, let Jenkins take care of the rest.

4. You can configure regular Jenkins build rules like alerting on execution failure, preventing subsequent executions if the current one fails, etc.

5. You can chain multiple jobs and the chain is very obvious thanks to the great Jenkins UI.

All this is great, but one problem I faced with Jenkins is that you cannot have a Job call itself recursively with a delay in between, you have to schedule the job execution using cron expression. The difference is subtle, but there are implications of this limitation which I will expound with an example. Let us say that I have a job which ideally should run every 15 minutes, but sometimes this job execution takes more than 15 minutes to complete, in that case what happens is, job executions queue up and fire successively one after the other. The way I want this scenario to pan out is, once the execution finishes, it should wait for 15 minutes before the next execution starts. I could not find a way to do this in Jenkins and hence selfie was born.

Selfie is a Jenkins build trigger plugin which lets a project to trigger itself after a configured delay. The plugin appears as one of the build triggers while configuring a new project.

selfie

This is my first attempt at writing a Jenkins plugin, pull requests and code reviews are more than welcome.

SQS versus Kinesis

A lot of people are confused between SQS and Kinesis. In some ways, both act like a queue, but there is a huge difference between the two.

SQS is a queue, adheres to FIFO and promises at least once delivery.

Kinesis is a distributed message stream handler. A simplistic and hand wavy way to think of Kinesis is like one huge log file, items that you write to the stream as lines in this log file. When you want to process the stream, you get a pointer to the log file. When you read a line from this log file, the pointer moves to the next line. Kinesis is stateless, as in, it does not maintain the pointer for you, it is upto your reading process to maintain this. What this means is that, say you are reading off a Kinesis stream and your process goes down, when you bring the reader process up again, it will start processing from the start, not from the last line before crash. There is no concept of popping items out of Kinesis, data is always there(expires after 7 days), you manipulate the pointer to this data. Hence, if you want to re process the stream, you can replay i.e you can start from the beginning and do the whole thing over and over again. AWS provides a client library for Kinesis which maintains the state for you. This client library uses dynamodb to persist the state.

This should give you a fair idea of when to use Kinesis and when to opt in for SQS.