Void

SQS versus Kinesis

There is some confusion around SQS versus Kinesis, both are QAAS™(queue as a service) provided by AWS(this statement is not entirely true, you will know why as you read on). This is an attempt at defogging this confusion.

SQS is a queue in the old fashion sense, it promises at least once delivery. You create a queue, enqueue items, dequeue items. One point to note is that, dequeueing an item does not delete that item from the queue, you have to explicitly delete the item post dequeue, this sort of goes against the intuitive understanding of dequeue, at least for me, but, you can configure that once you dequeue an item, that item should not be available for dequeueing again for a specified period of time.

Kinesis is a queue but it is not a queue and it is not a paradox, it is a high throughput stream handler. A conceptual way to think of Kinesis is like one huge log file, items that you enqueue as lines in this log file. You have a pointer to this log file, when you read one line from this log file(dequeue), the pointer points to the next line. Kinesis is stateless, as in, it does not maintain the pointer for you, it is upto you to maintain this. What this means is that, say you are reading off a Kinesis stream and your process goes down, when you bring it up again, it will start over the processing from where it started originally, not from the last line before crash. There is no concept of taking items out of Kinesis, the data is always there(retention period of a day), you manipulate the pointer to this data. Hence, if you want to re process the stream, you can replay. AWS provides a client library for Kinesis which maintains the state for you. This client library uses dynamodb to persist the state.

This should give you a fair idea of when to use Kinesis and when to opt in for SQS.

Release early, release often

Releasing early and often can make the difference between life and death for new age internet companies. Most of the successful dotcoms like Amazon, Google, Etsy etc do hundreds of deployments per day. If you are a small organization, your magnitude and frequency of deployments might not rival these big organizations, but it is always a good idea to release early and often.

If you plan to carry out multiple deployments a day, it is critical that you do not have downtime during these deployments. When your shiny new code is getting deployed, your end users should be able to access the site. This is usually done by routing all your traffic through a load balancer. The end user does not directly hit your application server, he/she hits the load balancer and it is the load balancer’s responsibility to route traffic to your application servers. In addition to this being the recommended architecture for server side software, it gives you the agility to deploy code without having to worry about down time. You take a server out of the load balancer, deploy your code on it, add it back to the load balancer and do the same with the other servers that are part of the group. Two of the most popular load balancers out there, Nginx and HAProxy, allow you to do this dynamically(while the load balancer is up and running you can add and remove back end servers) and I am sure that other load balancers let you too. If you are running on AWS, Elastic Load Balancer lets you do this.

Also, your deployments should be as simple as possible, even a monkey should be able to deploy your code to production. More the complicated it is to deploy software, less the enthusiasm of developers to do it. Using a continuous integration tool like Jenkins helps to make this as painless as possible.

Enable a kill switch for all new features. Your app should have the ability to turn on and off features in seconds. This gives you the power to turn off a feature if you detect any problems with it in the early days.

Also, gradually releasing features is a good idea. This lets you whet out performance and other issues on a small scale before it becomes a site wide issue. Release to 1% of your users and then slowly ramp up to 100% keeping a close eye on the feature all the time.

If you are working on a feature, you do not have to wait for feature completion to release. As and when you finish off logical steps, keep releasing. Your end users might not see the feature yet but this helps you to get away from the situation of one big bang release and everything going down. These incremental releases help to detect bugs early and build confidence for the final release.

Set alerts for all your key performance and business metrics. Post deployment, if any of these metrics go awry, you get an alert and you can set things right. In addition to alerting, having the ability to graph these is tremendous. Post a deployment, you can check your dashboard to see whether the deployment has had an adverse affect on your response times, key business metrics, etc. This adds to your confidence to do multiple deployments without having to worry about your new code degrading performance or adversely affecting business.

These are some of the simple tips that help you deploy code to production on a regular basis. I have not touched on more advanced topics like automated testing, integration testing, automated provisioning etc.

The Expectation Test

I got a phone lying down on road. Since I could not unlock the phone, I waited for the owner of the device to call. A couple of hours later, he did call and in his opening sentence started pleading to return it back. Even though he was the rightful owner of the phone, he expected me to never return it back to him.

I was stuck in a traffic jam caused due to BWSSB(Bangalore Water Supply and Sewerage Board) closing a very small section of the road to fix a sewer. This was on an afternoon on a main road in Bangalore and no wonder, there was a traffic pile up for close to 3 kilometers. I cursed myself for taking that road instead of some other alternate route. I did mentally spew venom at the people digging the road, but it never occurred to me to hold the authorities accountable for this and expecting better from them.

The curious case in both the incidents is the expectation of the people involved in them. Even though the right thing to do when you find something that is not yours is to return it back to the owner, people nowadays expect that not to happen. It is the responsibility of the government to execute it’s task in such a manner that it causes least annoyance to the citizens but we no longer expect that from our administration. If things do happen the right way, we are mesmerized and think of ourselves as lucky, it does not occur to us that this is how it should be. This is not a good sign because when expectations hit rock bottom, it becomes the norm and from there it is a vicious downward spiral.

The same is true for organizations as well. Holding people to higher expectation and making them accountable for it is the key to success. If in an organization, you are happy that your team members are not taking impromptu leaves, there is something really rotten. There is nothing stupendous about people not taking unplanned leaves, this should be the norm and if someone following this norm is an aberration in your organization, then this a warning signal to you that there is something deeply wrong which needs immediate fixing.

A test for any organization to see whether it is on the path to greatness would be the expectation test. Mentally go through your expectations from employees/company and see whether these are really higher expectations or something that should have been the norm. If these are expectations that breach the bar, well and good, if not, do a reality check and fix them.

We are a startup

Being part of a startup is not an excuse for you to shoot first and ask questions later. “We are a startup” is the most common phrase people spout when you ask them about their sloppy processes or development practices. Being a startup is not an excuse for you to have zero process or letting your developers be trigger happy. It is also not an excuse for condoning sloppy employee behavior like taking leave without adequate notice or working from home on a short notice.

I have heard of stories in startups where everyone writes components in a language of their own choosing. If today’s fad is Node, then the developer uses Node and if tomorrow hacker news has a post touting the next greatest language, the developer switches to that. We are a startup, we do not have a QA and pre production environment for testing, we test code directly in production. We are only a couple of people, hence we do not need version control. Since I am a single guy who handles servers, configuration files are not in version control, my brain is my GIT repository.

A start up is an opportunity for you to move fast without being burdened by the bureaucracies/politics of big organizations. Say for example, if you find an employee kicking ass, as a manager in a start up you can give him a raise by having a quick chat with the CEO. In a big organization, you have to make a case for that, present it to a committee, justify why someone should be given an out of turn raise etc etc. It is also an opportunity for you to trust people and call them out when you see something odd without processizing it. For example, if you see someone working from home on a very regular basis and it does not work for you, instead of floating a blanket rule saying work from home is prohibited there by inconveniencing other employees who use it judiciously, you talk to the person and see what is the problem that person is facing and try to solve it with him. This is very difficult to pull off in a big organization that has thousands of employees but highly doable in a startup.

A startup is an opportunity for you to create an atmosphere based on trust rather than draconian processes. Let us take technology choice for example. Big organizations have lots of rules and regulations on what technology can be used, what cannot be used, how to buy licenses etc etc. This is made with the central assertion that individuals are not responsible enough, so let us decide by committee. In a startup, you do not need to introduce a process for this, you can trust your employees to make the right technology choices as long as they do not go bonkers with it. If something goes wrong with a technology choice, you can talk to that person and figure out where he went wrong instead of mandating a policy for all employees from then on. Individual responsibility trumps rules any day, it is difficult to implement and follow this in a large company but can be easily pulled off in a small organization.

Next time when you use the phrase “We are a startup”, think whether you are trying to mask your inefficiencies in the guise of being a start up.

Let us set up an office

FreeCharge recently moved into a bigger office. This brought back memories of the first Bangalore office set up. Setting up an office might not seem a daunting task, but trust me it is. A previous company that I was part of had opted out of setting up it’s own office and worked out of an incubation center, I always used to wonder why not move into a proper office space? Now, after personally going through the hassles of setting up and maintaining an office, I appreciate their decision. If you are a small company, try to avoid moving into an office space of your own as much as possible, do it only if you should and must. There are other options available like co-working spaces, incubation centers etc. In spite of this, if you want to get into the adventure of setting up an office, read on.

First step of the process is prospecting potential office spaces. Office spaces in software parks are expensive and come with long leases, but they are the most organized. If you are reading this, most probably you do not fall into the category who can afford an office space in a software park. Some of the things to keep in mind while prospecting:
1. Keep enough leeway for growth, for at least an year.
2. Is there enough parking space? If there is a shortage of parking in the building and the building is located in a busy place, your employees are going to curse you. You do not want people to be grumpy in the morning before they start their work. On the other hand, if the office space is in a residential area, this should not be a problem, usually you find enough space on the roads to park.
3. Does the building have power back up? You have to keep this in mind until Modi gives us 24 by 7 electricity :).
4. Do all the major internet service providers(ISP) serve the area? Confirm this before you sign on the dotted line as some areas have restrictions against digging etc due to which ISPs refuse internet connectivity.

Once you zero in on the place, the second step of the process starts. Things to keep in mind:
1. Interiors and furniture.
2. Networking. Do you go for wi-fi or ethernet? If ethernet, you have to plan your layout in advance. If wi-fi, you might not be able to get away with a home router as the signal strength may not be strong enough throughout the office or the router might not be able to accommodate all the people in the office. In all probability, you need an industrial strength router/switch and repeaters.
3. Drinking water, do you buy a filter or go for water cans?
4. ACs, fans etc. Do keep in mind that you do not have to buy these, you can as well rent them.
5. Cleaning and maintenance, best to outsource this to an external agency. Keep phone numbers of electricians, plumbers etc handy, you will need them all the time.
6. Office keys, you might have to have a security guard 24 by 7 because you will have people coming in and leaving at different times and coordinating the key becomes a headache.

Also, it is best to hire an office manager who can co-ordinate all these tasks for you. These tasks eat into your precious time and I am sure your time is not well spent on these mundane yet extremely important tasks. If you are working out of an incubation center or co-working space you get most of the above as part of the package. As I said before, try to push off moving into an office of your own as much as possible, do it only if you must.

Nothing is sacrosanct

There is an interesting bug opened against Kafka. For those of you too lazy to click on the link and read through the description, I am reproducing it here in full.

It appears that validation of configuration properties is performed in the ConsumerConfig and ProducerConfig constructors. This is generally bad practice as it couples object construction and validation. It also makes it difficult to mock these objects in unit tests.

Ideally validation of the configuration properties should be separated from object construction and initiated by those that rely/use these config objects.

http://misko.hevery.com/code-reviewers-guide/flaw-constructor-does-real-work/

It links to an article by Misko Hevery on writing testable code. If you have not read posts by Misko Hevery, I urge you to. For an open source project like Kafka, it might make sense(Jay Kreps, the person behind Kafka does not agree with the bug as visible in the comment) to follow all the guidelines of writing testable code but for your project it might not. If you are a small company with a two or three person team, do not blindly follow practices because someone on the internet says so.

Follow rules and guidelines only if it helps you to make your code more secure, performant, easier to maintain etc, do not ape guidelines without understanding why they are laid out in the first place. To go back in history, checked exceptions were all the rage in the Java land a couple of years ago, but these days, after frameworks like Spring sprang up, people look down upon checked exceptions. Same with TDD. TDD was expounded as the next best thing after sliced bread, but now programmers are raising their doubts about TDD.

A lot of times, mocking objects, interfaces etc takes more work than writing the actual functionality. In many projects, there might not be an ROI in writing/maintaining this elaborate test framework/infrastructure. It is true that injecting dependencies into an object makes it easier to test, but it also comes with the downside of having to take care of injecting dependencies every time you create that object. If you are injecting dependencies by hand, object creation becomes an elaborate exercise each time, else you have to delegate this to some framework like Spring, Guice etc, now your project is bloated. Maybe you should side step dependency injection and create the object with the dependencies inside it.

The situations and background under which these rules/guidelines crop up might be radically different than the one in your organisation/team. Taking the call of what to follow and what not to is more of an art than science. Your instincts, past experiences etc help you in formulating them.

At what cost?

I happened to read the comic by zenpencil on Jim Henson yesterday while reminiscing on the comic by the same artist on Bill Waterson’s advice. Both the comics have the same essence of escaping corporate drudgery and following your dreams. Also yesterday, I watched Inside Llewyn Davis. Inside Llewyn Davis is a melancholic movie that depicts the daily struggles of a musician who has given up his job as a seaman to become a musician. Serendipitous right?

Most of the time, I get a feeling of disenchantment from people about their occupation. To put it bluntly, it is fashionable to be cynical about one’s job and complain about it. A lot of people feel that they want to do something different but most of them do not know what is this different path they want to take. And to fuel this fire, you have hundreds of books and blog posts which extol you to give up your job and follow your passion.

You go on a scuba diving holiday and all of a sudden you want to be a professional scuba diver. You do a couple of hikes and a la, you aspire to be a travel writer. You have started to cycle to office and you have this strong urge to start a bicycle touring company. You purchased this new DSLR and now your single aim in life is to be a wildlife photographer. An article about a so and so who gave up his cushy corporate job to start a local bike store or became a wildlife photographer reinforces these thoughts.

Even though these thoughts are romantic and warm the cockles of our heart, the reality is a bit different. When a hobby becomes a job, the fun aspect of the hobby goes out of the window and the boring part kicks in. When you do something repetitively, the novelty wears off. Some paragraphs from this article on becoming a travel writer:

But while free trips, global travel and your name in print sound glamorous, there are down sides. It’s hard work, hugely competitive and – unless you are the second Bryson – you won’t earn much. Roving overseas with a notebook, a deadline and a pack of other journalists can also take the fun out of travelling altogether. Not put off? Read on to find out how you can get this dream job.

Below is a paragraph from an article by a scuba diving instructor:

The job duties of an instructor aren’t what most newcomers expect, either. And to many it comes as a sad surprise. The reality of being an instructor at least full-time is that teaching is only a small portion of what you’ll do. Mainland-based instructors often work 40 hours a week at a dive store counseling customers or repairing equipment, then teach one or two classes a week on top of that. That means 60-hour weeks are commonplace. Think it’s easier at a resort? The norm for resort-based instructors is six days a week, and during busy periods seven days isn’t uncommon. Here, too, teaching is only a minor part of the job, but an instructor ticket is essential, if for no other reason than to get a permit to work in a foreign country. Many have left the industry disappointed that their dream of spending their days primarily as teachers never materialized.

And to top it all, you have to read this from a person who started his own micro brewery.

A lot of times, we misunderstand novelty for passion. You have a sedentary desk job and travelling once in a while looks like life’s calling, but the question to ask is, would it still be your life’s calling if you had to do it 24 by 7 while earning a substantially lower income? Would you not be more happy earning a good salary, enjoying the material comforts that your day to day job provides and travelling once in a while to break the routine?

In most of these offshoot jobs, the number of slots where you can be comfortable with the income is limited and the aspirants for these slots are unlimited. Also, in a majority of these, you have to be at the pinnacle to earn really well. We all love to think that if we are talented, success naturally follows, but I call this specious. There is more to success than just talent, success is mostly a factor of being in the right place at the right time and luck, not to say that talent and other factors do not help, but it is for sure not only talent. Thinking fast and slow by Daniel Kahneman expounds a bit on this.

The melancholy that sets in with a day to day job is our own making. If you are really interested in spicing up your work, there are innumerable ways to do it, it is just that effecting change is boring while cribbing about it is romantic.

Requirements

We had an admin interface from which people could download an Excel report. One day, we got a mail saying that the report format is Excel version so and so and it does not work with new Excel versions. The scramble began to find out which version of Excel was used, which version our app produced, how we can upgrade the library the app uses to create Excel documents to the latest version etc. For people not in the know, it is a pain to programatically work with any Microsoft office format in Java, you invariably feel like pulling your hairs out. In the middle of all this madness, I asked a simple question, instead of creating the document in excel format, why do not we create it as CSV? The report consumers were neutral, they were like we do not care as long as we can open it in excel.

As programmers, a lot of times, we blindly work on product requirements without questioning the intention behind them. As it was in our case, the person who gave the Excel requirement did not have any idea as to the difference in the amount of work one would have to put in to create an Excel report versus a CSV, nor the technical debt of the two approaches. From his perspective, it was all the same.

When a requirement comes to you, take a step back and go through them, see if there are chances to simplify. If some of the requirements sound absurd to you, tell it to the stake holder, if some of them might take a long time to implement, talk to him about this. The person providing the requirement has absolutely no idea that some of these features might be technically difficult to pull off or take a long time to implement or there is a better alternative that might not meet the criteria strictly but will not take eons to implement. Bias towards action is good, but a trigger happy one hurts everyone.

Designing for failure

In the world of software, failure is a certainty. Servers go kaput, databases go down, processes go out of memory, things break all the time. You can categorize software as good or bad based on how they behave in these adverse scenarios. I am not trying to imply that software has to be resilient to all these, on the other hand, I believe that it is perfectly fine to crap out when shit hits the fan. The question is how do you handle this crapping out.

Whenever architecting components, devote ample amount of time to failure scenarios. Let us take the case of a piece of code which interacts with an external, third party API. What are the questions you should be asking when designing this component? What happens if the API suddenly stops responding one day? Can I hold my main thread hostage to the API response time? What happens if the API takes eons to respond? In case there is an exception, am I logging enough data to debug? If there are performance issues, do I have enough diagnostic data? Diagnostic data might be in terms of graphing the API response time, no of times the code path was executed, etc. Do I need to send out an alert when something goes wrong? All these question revolve around failure handling. These questions should be second nature to you as a software engineer.

I have seen a tendency among developers to devote inordinate amount of time in making their code adhere to the latest programming fad, trying to use the best possible library etc, but not to failure scenarios. Logging data might not be as sexy as debating which design pattern to use, but once things break, logs are your only friend. Next time when you are furiously pounding on the key board, take a step back and ask these questions. In the future, the developer who maintains the code that you wrote today, will thank you for doing this.

Poor cannot eat roads

Rahul Gandhi allegedly made this statement. It is sad that an armchair, untrained economist like me understands the significance of good roads while someone who is poised to lead India does not. Check out this vice documentary on truckers in West Africa to get the connection between roads and economy. Jim Rogers, author of the excellent book Investment Biker also alludes to how Africa is rendered poor due to bad road connectivity. In spite of so much evidence suggesting good road connectivity being essential for a healthy vibrant economy, Rahul Gandhi makes this asinine statement and our so called independent media is busy discussing his dimples and his charm working up pubescent teenage girls.

A road is something that transcends socio economic, caste and religious boundaries. A road does not discriminate between a rich man driving his BMW or a poor milk seller riding his bicycle or a pandit riding his Luna or a moulvi riding his scooter. I do not for a second doubt the ingenuity of our so called secular politicians to come up with statement on the lines of Minorities have the first right to our roads, but until that happens, for all purposes, we can take rest in the fact that a road is a great unifier. 

How do good roads benefit the poor? Good connectivity makes the transportation of goods efficient there by negating the cost that would be introduced due to transportation inefficiencies. This is an indirect benefit that is enjoyed by everyone not just the poor, but let us take a specific case of how good roads will make the life of a poor auto driver better. If an auto driver drives his auto on pot holed roads all day long, imagine the toll it takes on his auto. This will directly reflect in the efficiency of the auto as well as the money he has to spend on maintenance, not to mention the umpteenth lost opportunity to make more money by ferrying more customers due to the reduced travel time. All in all, an auto driver has to gain a lot with smooth roads. Instead of working on these, our government is hell bent on extending the economic black hole of NREGA to urban poor and our media is a silent spectator to this theater of absurd.

 

Follow

Get every new post delivered to your Inbox.

Join 465 other followers