Mental model of systems

One beautiful Sunday evening our quartz jobs running inside tomcat server started to freeze. At the same moment, tomcat went kaput. I sshed into the server and started poking around the logs. No error in logs. Hmm, ok. Checked the system health, again stats looked hale and hearty. Now, what do I do?


Get articles on coding, software and product development, managing software teams, scaling organisations and enhancing productivity by subscribing to my blog

I started to reason around the problem based on my mental model of the inner workings of the various components in our application. Jobs are freezing, and at the same time, tomcat stops serving out requests. Cannot be a tomcat issue as I am not getting a tomcat connection error, so it means that tomcat has enough threads to spare and its thread pool is not exhausted. The implication of this is new threads are being spawned but are not able to proceed. Hmm, ok, so the threads are getting blocked. Now how do I figure out where the threads are getting stuck. Can I hook something into the JVM and dump the state of all the threads running inside the container. Oh yes, JMX gives me that ability. Restarted tomcat with JMX configured, hooked into JMX and found out that the threads are getting blocked while trying to fetch database connections from the connection pooler. Our servers were not under any undue traffic spike, hence cannot be a scaling problem. That means some query is running rogue. How do I get the query that is running rogue? I can do two things – check the process list in the MySQL server or check the stack trace of the threads currently running and identify the point at which it is stuck. Took the thread approach, zeroed in on the code where the query was getting executed, fixed the query and things were back to normal.

If you are a full-stack developer, you can’t develop a deep understanding of every component in the ecosystem. Let us list down the things that you need to be on top if you are running a web application in Java. You have to know Java, the J2EE servlet spec, the application server that you are using, your MVC framework of choice, the components the MVC framework uses(maybe spring for dependency injection or hibernate for persistence), SQL and the peculiarities of the database, HTML, CSS JavaScript, JavaScript frameworks, CSS frameworks, the OS in which you are hosting your app, the administration of this OS, physical hosting, the list goes on and on. There are so many options available for each of these components. Also, these components have additional dependencies. Getting intimate with the intricacies and the idiosyncrasies of every element of the application is an arduous task better suited for immortals. This is where having a generic mental model helps.

Coming back to the problem of the crons stalling, even though I was not an expert in the internals of tomcat nor JMX nor DB connection pooling, I had an excellent general understanding of how these systems work in general, and I could fit the problem into it and reason through and fix it. Let us say that you are using an evented HTTP server and you see that the throughput is not up to mark. If you know how an evented server works in general, you can think of at least two loop holes in your system that might be causing this. Either you are doing blocking calls or running some CPU intensive tasks. This knowledge is transferable irrespective of whether you use Netty or Node.js or Tornado or some other new kid on the block.

I am not trying to discourage you from developing a deep understanding of particular frameworks or technologies, but to the contrary in addition to it try to abstract your understanding out of the specifics and into generals so that tomorrow when a new shiny technology shows up, you can reason around it based on your past experience.

Photo by Josh Riemer on Unsplash

The social taboo – body and mind

I just finished reading this and my mind was contemplating over the society’s taboo of using your body to advance yourself in life. I do not think Sasha Grey’s parents adore their daughter’s achievements in the adult film industry but I am sure Nate Silver’s parents are proud of his mathematical acumen. The world today is filled with females who have advanced themselves riding on their endowed bodies making sex tapes, featuring in voyeuristic reality shows, stripping naked for photo shoots and in general acting stupid, but the society in general is not very supportive of someone whose claim to fame is only that they have a smoking hot ass.

Even if an event is all about looks and having the body of a Greek goddess, the people associated with the show try to shove in a meaningless intellectual layer to it. Does the Q&A round in miss world ring a bell or an actress who stripped naked on screen trying to rationalize the scene as part of the drama without which the film would have lost it’s whole meaning? Let us for once keep aside all our prejudices and all the social mores that we have been fed since our birth and analyze the situation. What is the difference between someone using their body to gain popularity versus someone using their brain? Are both not something that we are naturally endowed with? Are not some of the humans naturally more smarter than others? Why is it acceptable for these superiors among us to leverage this to their advantage but if you use your natural beauty and sexuality to your advantage, you are frowned upon?

I thought about this for sometime and there are two major reasons that I could come up with. For one, using your beauty to make a name for yourself is mostly a selfish act. This does not result in the society’s advancement as a whole. Let me ask this rhetorical question, what is Kim Kardashian’s contribution in making this world a better place? On the flip side, has Einstein’s genius helped us to make progress as a human race? The no of lives that you affect with your intelligence is infinitely more than the no of lives that you can hope to touch with just your looks.  The society in general makes strides due to human intelligence but human beauty does not make this world a better place.

The other side to this is, beauty is not something that you can work on and gain as you go about your life. It is mostly a binary property, either you have it or not. It is not something that you can toil towards to. Since the human race sees grey cells as something to worship people with, it becomes a goal for each and everyone of us to better ourselves in this area and due to this humanity gains as a whole. It is one vicious circle and I am thankful to our forefathers who had the wisdom to think about this and inculcate this in the human spirit.