I think we Indians don't like each other

It is 12:42 am (most would say middle of the night) and the revelry to drown the Ganesha is still in full-swing. I think there are a few kids shouting "Ganpati bappa morya" while dancing to "Baby doll" and "Lungi dance". Having a house next to a "river side" isn't something as glorious as it sounds at times such as these.

It got me thinking that, maybe the uneducated don't care. But I guess that is true for all us Indians. We are a Billion, I guess you will find one everywhere, probably even in space. So ya, since we are that many, we probably resent each other. I am not sure how many generations it will take for us to change this about ourselves.

When I was in the US, we would say "hi" to probably every other skin-color regardless of whether we knew the person or not. But when we saw someone who remotely looks like an Indian, we would shy away from an eye contact. And even probably utter a few quips. So I guess in India, when there are more of us, it is only natural to avoid each other and not think of the other person.

This is also the reason probably we don't help a fellow Indian who is lying helpless on the street. We read in the papers that there is a poor kid needing some treatment, and we probably turn the page thinking someone else will help.

So who ever reads this and probably wants to change our populous nation a bit:

- Please don't light firecrackers after 10 pm. There could be a kid sleeping, who will wake up with a scream 'cause you felt you wanted to "enjoy the moment".
- Please turn down the music after 10pm.
- If you see someone else doing the same request him/her to be compassionate.
- You find someone who would need help, call 108. It is free. It is only a hope that the ambulance will reach in time, but atleast you did your part.

Its 1:07 am and I called the cops 15 minutes ago, the music has gotten louder.

PS: Probably it worked. 15 minutes later, its awfully quite now. Can hear my thoughts again! 

We released Hercules!

This is an internal code name for a product that may not be available to our clients for another few months, but the release of Hercules is symbolic of the very DNA of Sokrati. It symbolizes the relentless hours and countless weekends put in to deliver, the indescribable camaraderie from everyone within the company and finally the one thing that brought all of us together. In the due course we discovered a new blood-line and we are an even more energized company! 

I would like to go over the events of the last few months to highlight upon the subtle changes that were done which made us accomplish an Herculean task! 

A Common Goal

It all started with a common goal. Instead of listing down 2-4 goals to achieve in a quarter we listed down the one goal in that quarter that will bolster us up. This one goal should be so impacting that the whole company will benefit from the accomplishment. This one goal should provide the power for self-alignment. And finally bring us all together into striving to fulfill it.  

Sokrati is all about experimentation. Unlike other companies that may follow a traditional route, we at Sokrati have always tried to inculcate anything new that will push us to "Pedal to the mettle". The one common goal was something new that Ashish read in "The Five Dysfuntions of a Team". This book was suggested by Bhavin and I am guessing he saw something wrong in the way things are run in Sokrati :)

This common goal was arrived upon after adequate deliberation and was calibrated as the one thing that will provide us a solid grounding to the future. 

Transparency 

Having a goal is just not enough. Does everyone in the company know about the goal? Is the goal well-received? It is one thing to come up with the goal and it is a whole other thing to be able to bring transparency to it. It is that one goal for the quarter and everyone within the company needs to know why we need it and how we are tracking it. 

Every wall of Sokrati now was plastered with the common goal and had weekly updates on how the goal was shaping up. 

Coming Together

Within a few weeks of the launch of the goal and after adequate amount of confusion was dispelled, one could clearly see alignment to the common goal. No one wanted to be left out or be distant to the goal. The entire engineering moved to one floor so that they could help each other, business team started engaging more frequently with engineering so as to give appropriate and timely feedback to the product, recruiting team was pushing harder to close any hiring requirements. In short every team had magically come together to become one giant team all pumping towards launching Hercules!  

Celebration

Yes and we did it! A product that would have taken 75 man months to finish was accomplished within 3 months of working together. A feat such as this has no easy words to describe. The founders pitched-in to take the entire company out for dinner. The evening started with a little presentation about the quarterly goal and was handed over to the teams to go over the details. It was so heart-warming to see every team go over how Hercules has changed all our lives. It was not a mere release of a product, it was infact the blood-line of the entire company. 

Another tremendous team effort! The Quarterly Common Goal has now become a tradition.

Raising Sanjana & Sokrati

My wife's and my joyous moments were at 4:46pm on the 7th of May 2012. After all the anxious excitement in the labor room and after the exhausting outcome, my little baby crawled to her mommy with a little weep, bringing tears to everyone present. I wasn't so sure then if the tears were out of joy or out of exhaustion. The gentle reminder from everyone "Your life is going to change" kept ringing. You may have done the hardest things in life, climbed the tallest peaks, would have accomplished million lines of code or have achieved million visitors on your site but when you hold a little baby in your hand all the confidence you mustered over the years would suddenly drain you leaving you so belittled. That's when it hits you "Maybe you are on ground zero. Your life has changed".

After the paternity leave when I head back to work with a 100 donuts to celebrate the joyous entry & return, the first thing that comes to my mind is "What has changed?". I am back with my folks who are in the other part of my life and I am back to the daunting challenges of making Sokrati successful. After the pleasantries and settling into my desk, the day just brisked fast. By the end of the work day which ended at around 08:30 pm Sokrati had no answers to that question. I reach home with my wife eagerly awaiting to greet me after my first day as father and while seeing our baby sleep I recite to her the events of the day. The day ended at around 1 am with still no answers lingering anywhere close. 

I think it is all up-to the way you perceive life. It is just another challenge and I have now (6 months later) begun to feel that with a right head and right principles you will need to change nothing. How is taking charge of a challenging enterprise any less of a change than raising a baby? I guess over the years I have begun to perceive myself as a parent to everyone at Sokrati. So in short I bring a tremendous parenting experience to Sanjana. There are so many similarities in both that it is mind-boggling.  

Lead by Example

Sanjana brought in a sense of realization & retrospection. I had begun to lessen my usage of foul words, begun to consume alcohol in moderation & had begun to set myself on the ideal path that I would expect my baby to aspire to. I know she will make mistakes and learn from her own as well, but I would try my best to lessen them for her.

How am I to expect that no production outage goes undetected at any time of the day, if I am not doing it myself? How am I to transpire enthusiasm in the company, if I am not energized myself? How am I expecting 100% client satisfaction from others, if I myself am unable to achieve the same? 

Building Trust

As a parent I am almost a manager on the outset. Trust cannot be built by the stick, it has to be built by arduous efforts of ensuring an excellent rapport. If I become unapproachable to my child's most trivial needs & demands I am sure she will become reticent towards me. If I expect that she disclose to me the faintest of her worries I got to create a bond that makes her believe so. One has to believe "No question is stupid, only answers are". 

The sooner a manager realizes that the fault is in his/her management style will he/she take corrective measures to ensure that the team operates smoothly. One has to have a very patient ear and a hawkish eye to understand the problems and not brush them away. There is no book or literature that states how to do that. It is only an exercise within.

Establishing a Good Life/Career

As a parent the first thing that comes to your mind is "I need to resolve to facilitate a good life for my child, so that they have a bright future and career". I am sure when the time comes I am going to ensure that she is in the right path of education, friendship & knowledge. I will make sure that she gets my undivided attention when she is beginning to explore life. 

It beats me as to how one can aim to achieve a successful company without facilitating successful opportunities of growth & career to everyone involved. A founder/entrepreneur has to take the necessary steps to ensure that everyone feels accomplished at the end. This would mean fixing processes, or setting new ones and establishing ladders to achieve growth. 

To sum it up

Yes if it ever did then possibly your life has changed for the better. So "Raising Sanjana & Sokrati", in the end it is either succeed in both or fail in both.

Engineering @ Sokrati: The Basics

Like every start-up the engineering at Sokrati has evolved over the years. Many inclusions are ideas within and ideas adapted through monitoring others'. No process definition can be complete & straight-forward. Life at Sokrati started just like any other start-up: bootstrapping, being scrappy, ensuring survival while delivering quality, followed by growth and now we are a midst growth and expansion.  

By the close of 2011, we realized the importance of re-hauling the current adhoc process in engineering. Why is it that we are constantly fixing code or constantly playing a catch-up with scale or are constantly involved with addressing client issues? We were managing over a million transactions per day and couldn't sustain any down-time. Software that used to run on one mighty server now is housed upon 126 commodity servers and the count is only growing. Our Systems Engineering team comprising of 2 manages 63 servers per person! And yes the system never sleeps! As a matter of fact if you are responsible for managing the advertising budget of prominent Internet players, it cannot sleep. 

Now Sokrati has hit a very fast paced symbiotic eco-system. We now deliver high-quality software, with super high-quality support & maintenance along with feature roll-outs that happen every 2 weeks and yet keeping our cost on a minimal. Here is an attempt in detailing how the entire engineering unit works towards pursuing and achieving quality. If you are thinking of adopting a unique process or gathering upon ideas I hope this would be of help. 

The product planning happens 2 months before the development team is involved. During the product planning the Product Manager plays a crucial role in gathering the requirements for the features to come and works with UX in crystallizing rough mockups and pencil sketches. The fully functional mockups generated by UX is then run by the clients, business owners & dev leads to achieve a common ground. These features are lined up to be delivered to the clients over the course of next 3 months. The features are enumerated in the form of user-stories and with the help of the dev team, the Product Manager details stories that should be delivered in 2 week stints. 

We follow the agile model @ Sokrati and hence a week before the sprints for the 3 month development effort start the sprint planning takes place. The sprints are 2 weeks long with a goal to deliver the feature at the end of the sprint, fully tested & deployed to production. The sprint planning week is owned by the lead of the dev team. He/she takes care of doing the necessary research work and laying out the tech spec minimizing unknowns so as to adhere to the commitments. Once the sprint starts every effort is made to not bring any requirements or change any existing ones unless it is business critical. 

The team comprises of atleast 3 members whose roles are pretty much diverse and ever changing true to our culture. There needs to be atleast 1 developer, 1 QA & 1 Operational Rotation. At the start of a new cycle since there is no operational rotation the sprint starts with 2 Devs and 1 QA. The devs plan the implementation while the QA plans breaking the code. The QA is a rotational as well who is also a developer. But in this role he plays the devil's advocate. Once the sprint ends the QA who has signed off the release goes on to becoming the operational hand-off for the next 2 weeks. Where he/she trains the operations team to manage the feature being rolled out. He/she also ensures complete documentation to facilitate support and bug fixing that arises. The sprint is planned such that the devs have enough time to fix the bugs being pointed by the QA. All efforts are taken to roll atleast 1 feature every 2 weeks! 

A week before the end of current sprint starts the sprint planning for next sprint. Here again the dev lead is involved in articulating the deliverables for the next release.  By the close of sprint planning, since active QA current sprint is happening, the lead also gets to know about the bugs that arose in the current sprint. If it is blocking the release of the feature the lead may choose to prioritize the bug in the next sprint cutting down upon a feature. These become feedback notes to the dev and the dev-lead as well, to fix any loop-holes in their tech spec or development effort. The next sprint now starts with a new QA rotational and a possible new developer. 

This iteration happens over the next 3 months. By the close of the third month the new features and requirements are formulated by the PM for the ensuing 3 months. As one can see the development effort is very fast paced. Due to this there may be certain compromises in quality of code or good to have design patterns. After the 3 months of development effort the team takes a 1 month cool-off period where the goal is to retrospect and fix the past. This is also a buffer period to plan for contingencies and to freshen up as the next 3 months are going to start pretty soon! In this period we also aim at shrinking the code by 30%. Thus addressing any design compromises that had taken place. Yes we delete code in this phase.

Hope it helps. This system works best if the development teams are small and have all the required skill-sets to minimize external dependencies. If your team needs to roll-out UI features I am sure it would include doing some back-end work as well. In such a case don't create 2 teams, instead have one team (albeit small) of the required skills to ensure delivery of the entire feature. 

In my next blog I will go over the various roles and various team structures to demonstrate case-studies. If you would like me to substantiate on anything please let me know.   

A few moments with Kanwal Rekhi

A week ago we got a call from Inventus telling us that Kanwal Rekhi would be coming to our office in Pune. He would be spending a few hours in our office and would like to talk to us all. The agenda was that Kanwal would first talk about his journey as an entrepreneur and would open it up for questions. At first it was excitement which slowly turned into a worry. What if we end up not asking any questions at all? Well as the day progressed it didn't take long to realize the fears were unnecessary. I will re-cap the rest of the day in the order of the questions asked and Kanwal's answers. I have paraphrased almost all of the responses. My intent here is to convey the message and not to quote.

Should the strategy of an entrepreneur be eyeing an exit? 
An effective entrepreneur eyes at building a company that is enviable. Exit is one of a lot of outcomes. It could be an acquisition or an IPO or a literal buy-out. So don't worry about structuring your company for an exit. But rather build a product/company that creates a dent. 


What do you feel about opening geographically distributed offices? Building a sales office local to the market makes sense but what about engineering. What are the challenges of managing engineering through remote offices?
The reasons mostly to open distributed engineering offices is to tap into local talent. Bangalore currently seems too crowded. While it has a lot of talent it also has attrition. Pune seems to be a good place to gather talent, albeit harder than Bangalore. As a certain baseball player said "Coney Island is so crowded that no one ever goes". But if you have to then you must preserve your culture. I would take a couple of my best engineers who have been with the company for sometime and start the new offices. They will set forth the required necessary infrastructure and preserve culture.

It was now time to go to the auditorium where the rest of the team had assembled. We had recently moved into the new building and the elevator wasn't fully functional yet. To prevent Kanwal from climbing 5 floors we had provisioned for a semi-functional elevator whose floor readings weren't fully calibrated. The elevator had to be stopped manually. Seeing this Kanwal instantly chose to climb the stairs instead. He seemed fit as a fiddle! While climbing the stairs he mentioned that he works out 45 minutes everyday!! And we youngsters were panting while Kanwal was reciting stories on his way!

Kanwal's journey as an entrepreneur.
It was 1967 and the times when Kanwal graduated from IIT Bombay were not very fruitful to competent engineers in India. There were mostly govt jobs which didn't seem so exciting to him. He finished his Bachelors in Electrical Engineering and moved to the United States. The arrival wasn't very welcoming in the USA. The scene was completely different for Indians then. Many of the Indians were considered non-smart. When Kanwal aced his exams the professor would say "maybe you were cheating", to which he would say "I am smart! I am from IIT!". The next time the professor stood behind to catch the plagiarism. Kanwal aced again to which the professor said "If you were cheating, you did it really well!".

After graduating Kanwal found a decent job and rose very well as an individual contributor. He mentored many to become great managers while staying as an executor. All his managers would say "If you did so well in this role why would we ever pull you out". He was making good money (USD 75K which now would be USD 350K). But an entrepreneurial bug always kept him unsettled. It was about the time in 1980s when Apple, IBM and other companies where penetrating the PC market. A one Mr Johnson's success into this industry kept him inspired. He finally got his guts together and decided to quit his full-time job. He goes home and tells his wife about his decision to which she asks "We have such a nice house, two cars and kids, why would you do this? Is it me?". His answer instantly was "It isn't you, it is Mr Johnson!".

Kanwal started in the field of networking setting up LAN and gradually into WAN. His company Excelan was into smart ethernet cards which adopted TCP/IP and was one of the champions behind the success of TCP/IP and Internet.

It may now seem easier for entrepreneurs to establish geographically distributed offices and tap into the US economy, but then it wasn't. Many of the Indian entrepreneurs who struggled then have now paved the foundation for now Indians to establish businesses.

He now yielded to the floor for questions.

What do you look for during investing in a company?
He looks for the team. Because if the team is good and deserving then whatever be the odds they will definitely be able to overcome the hurdles. He believes in the team and not just the idea.

You have such a busy schedule. How do you manage and what keeps you going? You now have made many to retire. 
He believes in giving back. The concept of venture capitals provides a good ecosystem for growth. India only recently has opened up and one can now see the tremendous growth achieved. A venture capital borrows the money from LPs assuring them returns. This in-turn goes into funding companies who can effect a change. This brings jobs and improves the economy of any nation while bringing in accolades for many. Nothing comes for free or is charity. When is there any time to rest?

What do you think of brain-drain?
He thinks this is inappropriate. Only a few thousands go oversees to pursue other interests. In a nation of  billions if only those few have brains then India should be worried!

How would you measure the effectiveness of innovation?
Innovation needn't be about something new, it is about doing something differently and pushing yourself ahead. There are many companies who prove this point effectively e.g. Hotmail, Google, Facebook. And competition drives innovation. Unless there is someone breathing down your throat, there is no urge to push yourself ahead.

A truly inspirational guy! 

Is Amazon EC2 really worth every penny?

At Sokrati we are very dedicated users of Amazon EC2. We swear by it and have a complete faith in their offering. Amazon EC2 has never let us down and has been sticking to their word of 99% uptime.

Many of my skills of Systems Administration I picked up during my 3 years stint with Amazon. I wonder whose idea it was but the complete decentralization prompts every Software Engineer to think through their hosts and procedures. It is at Amazon I learnt the importance of load-balancing, fault tolerance, datacenter redundancies, backup and those myriads of system maintenance that folks in other companies just take it for granted.

All of the infrastructure provided by Amazon EC2 is very much everything a company needs to run its systems with hardly any downtime. But are we using it right? Is it worth the money we are spending on it? When was the last time you checked the EC2 billing? Is there an alternative?

Are you using Amazon EC2 right?

EC2 provides a very convenient way to bring up boxes and shut them down when you are done using it. They have a very good API to achieve the same. But do we end up shutting them down? 

In a true cloud-sense machines are to be used when work is to be done and then shutdown after completion. Amazon bills you by the hours of usage. Though this paradigm seems very simple to understand and seems efficient in reality it aint so practical. When you bring up a box for usage you will have to configure it before it becomes usable. There are a couple of ways of doing this:

• You use Amazon AMIs: You create an image of the host with all the configuration you intend to have. When the need arises you bring them up, get the work down and then shut them out. The only drawback here is that the images have to be kept up-to-date otherwise the stale software puts your system in jeopardy. I donno about you guys but this is a constant pain for us and makes it prohibitive to use on an "on-demand" basis. 
• You use a central system configuration tool such as Puppet or Cfengine. Here after the host is brought up it talks to a system management server and gets itself in sync and then becomes operational. 

Apart from the above you need a way to bring up a box and to assign it work. This needs a major shift in the way your software is designed. We were able to solve this in Sokrati using a Distributed Scheduler. How many of us use EC2 in an effective way so that there are no idle cycles. We mostly end up bringing up boxes and keeping them up and running for time immemorial. 

When was the last time you checked an EC2 billing statement. I am not sure about you but it is so scary that I stopped looking at it. In this constant urge to keep pumping more software and more muscle we lose track of how much we are paying for the same. If you look at the bill it is divided into the following components:

• Elastic Block Storage: This is inevitable if you want data to be persistent after a crash
• S3: Inevitable as AMIs are stored on S3. You may also be using it for data backup
• Data Transfer: The bandwidth that you are consuming. Again split across External & Internal. Internal is free but external is not. I am not sure about how they charge inter-datacenter traffic.
• Load-balancer: A necessity for balancing traffic and fault tolerance.
• Finally the host itself. Small, Medium, Large, Extra Large

And then there is the concept of Reserved Instances which puts you in a contract for a certain number of months. 

Alternatives?

We recently began to dabble with Hetzner with a rhetoric question "do I really care where my boxes are located?". This is a hosting service provider located in Germany. And I am glad to say I am surprised. They have an offering of virtual machines, dedicated hosts and collocation services. There are many necessary solutions that Amazon provides where Hetzner doesn't even come close. Here I list them down and also provide workable solutions. 

Host Comparison

The Hetzner hosts are far superior and far cheaper than that of Amazon EC2. It surprises me how they are able to even provide them. I spoke to a couple of Indian datacenters asking them to match the price and they either don't call me back or in some instances they outrightly have told me that their price is unbeatable. 

The comparison is not necessarily apples to apples but I am picking up the most commonly used instance of Amazon EC2 (a medium 32 bit host) and a very large host of Hetzner to show you how way off Amazon EC2 is. Secondly I am also assuming at this time that you run your EC2 instance permanently, similar to the way you would run a dedicated Hetzner Host. 

Type	RAM (GB)	Hard Disk (GB)	CPU	#CPUs	Per Month (USD)	Setup Cost (USD)	Discounted Yearly Cost (USD)	Discounted Yearly Cost (INR)
EC2 Medium	1.8	335	Intel(R) Xeon(R) CPU	2	140	0	1680	83,999.15
EC2 Medium (1 yr reserved)	1.8	335	Intel(R) Xeon(R) CPU	2	80	138 (per yr)	1098	56,238.48
Hetzner EX5	24	2 * 700	Intel(R) Core(TM) i7 CPU  920  @ 2.67GHz	7	78.25	65 (1time)	813	41,641.00

• EC2 hosts are virtual and Hetzner hosts are dedicated physical machines
• Hetzner costs have 19%VAT included. And if you are billing unit is out of Germany you avail a 19% reduction i.e. you don't have to pay VAT.
• Discounted Yearly Cost includes the Setup fee + 12 * monthly fee. And for Hetzner 19% VAT is excluded.

You can clearly see how much better Hetzner is!

EC2 has Images!

Yep EC2 works on AMIs and is on Virtual Hosts and hence images is a great resource. Hetzner has no such thing. Infact Hetzner boxes are plain vanila boxes booted up with a specific flavor of the OS you want and that's it. 

You can easily use Puppet (a product of Puppet Labs) to configure your hosts as per your requirements. 

EC2 provides easy scaling

Hetzner boxes have to be requested as opposed to EC2 hosts. It is manually provisioned in their rack space and then allotted to you. It usually takes 1-2 days if the requested configuration is available. And if you caught them over a weekend then 3 days. 

To handle this I have started requesting 3 more than required. The billing starts after the first month. So you get a month after the host has been launched to decide whether you need it or not. Post 1 month the setup fee and the monthly billing gets effective.

EC2 has Load-balancing

Hetzner doen't provide load-balancing software or hardware and they aren't going to offer it. We have bypassed this by using HAProxy and Amazon EC2's load-balancer. Since Hetzner is in Germany we launched two hosts of Amazon EC2 in Ireland on which we installed HAProxy. HAProxy on these hosts know about the Hetzner hosts and provide load-balancing on them. These two HAProxies are behind the Amazon's Load Balancer in the cloud which provides fault-tolerance. Both the HAProxy are peers and operate in the same fashion. This adds an extra hop but is a very workable solution as opposed to doing your own load-balancer fault-tolerance. 

 Amazon EC2 LBs cannot manage hosts that are outside their datacenter. And hence this workaround. 

EC2 has EBS  

Honestly with the quality and quantity of the hard-drives that Hetzner is providing I doubt if we need more disk space. The Hetzner drives are RAID so they have an easy backup and are crash resistent. I am sure if you call Hetzner folks and ask for an extra drive on your host they will be able to provide you one for an extra cost. 

EC2 is in Singapore too

Hetzner is a Germany based provider. For India Amazon EC2 Singapore is the closest datacenter. Latencies are slightly more from India. 

• Ping time India -> EC2 Singapore Host: 157 ms
• Ping time India -> Hetzner Germany Host: 199 ms

EC2 has S3

Yep and chances are you may still want to use it! Hetzner has a backup solution but I am not sure how good it is. 

Conclusion

Going by the lines of my forceful high-school teacher, I conclude by saying please make an informed decision regarding your service provider. We are a startup and bleeding money is in short killing yourself. And no I don't work for Hetzner, I am a hardcore Sokratian!

If you'd like any specific information about either please leave a note and I will try to answer them. 

A Very Simple Concept for Crowd Control

As you get off Indore aiport you will neither hear any whistling nor will you see any cops. Granted the airport is small but the tendency to grow a crowd during departures and arrivals is very much possible. When I was being dropped off at the airport the cab driver was stopped to pickup an entry ticket into the airport. Owing to natural tendencies I quipped at it instantly, "Now they will charge you to even enter an aiport?". To this my fellow passengers corrected me saying that if the driver doesn't exit within 4 minutes he will have to pay Rs 100 as a parking fee.

It is such a brilliant idea for crowd control. The driver wouldn't wait even for a second after he dropped our bags. If you observe the rut that is created at the Pune airport, you will empathize how irritating it is to navigate the car around.

This is the first time that a government agency has ever surprised me. I haven't seen such a system even in the US.

Kudos!