Archive for January, 2008


More Concurrency and Scalability Stuff

January 23, 2008

One of the things I spoke about during my scalability piece was about state models. In a nutshell, you need a means to share state between multiple nodes of your application in your scaled-out environment, whether it be a database or a clustered memory model, a la Terracotta. What I failed to mention was that a clear and thought-out locking strategy also needs to be put in place.

In a concurrent multi-threaded world, using locks to access and modify shared data is common practice. But what happens when you have multiple instances of your application sharing state? Most locks are not effective across JVMs. Neither are the new java concurrency classes such as Semaphores, which are more elegant and configurable constructs than the old synchronized keyword. Terracotta provides its own locking constructs to access shared data. With databases, you have to ensure locks are implemented accurately, to achieve transactional integrity. Testing this out is also tricky and should be handled carefully. Either way, a close look should be taken at your locking model.

Our locking model, fortunately, never came under the microscope when we scaled out our application. Very early on in the development cycle, we hit upon the realization that despite heavy concurrent access to our state store, most accesses would be reads, not writes. This, in turn, led us to write a simple “Optimistic Concurrency Model” as our locking strategy. Essentially, every read from our state store would return a cloned copy of the data. No locks used. The thread was now free to modify the cloned data, without worrying about affecting the flow of any other concurrent thread. If it attempted to write back its new state to the state store, the following rules would be applied:

  1. It would acquire an exclusive lock to write back into the collection. No other reads or writes to that specific record could happen concurrently.
  2. If the data object had been updated by another thread in the interim between this one reading it and attempting to write, the write would fail. The thread would then get a cloned copy of the updated state and process the data again, with a high likelihood that a new code path is executed (unless the updated state has no effect to this thread’s execution).

This improved performance tremendously in a single-node scenario within our application and in a multi-threaded scenario, required no new code to re-engineer the locking strategy.

Now, why did this strike me after my talk and not before? Because, I came across this in the latest edition of The Java Specialists’ Newsletter:

Since Java 5, the language includes support for atomics. Instead of synchronizing access to our fields, we can use atomic references or atomic primitives. Atomics use the Compare-And-Swap approach, supported by hardware (the CMPXCHG instruction on Intel). For example, if you want to do a ++i operation with AtomicInteger, you would call the AtomicInteger.addAndGet(int delta) method. This would use an optimistic algorithm that assumes we will not have a conflict. If we do have a conflict, we simply try again. The addAndGet() method does something like this:

  1. get the current value and store it in a local variable current
  2. store the new value (current + delta) in a local variable next
  3. call compareAndSet with the current value and the next value
    1. inside the compareAndSet, if the current value matches the value in the AtomicInteger, then return true; otherwise false
  4. if compareAndSet returns true, we return next; otherwise start from 1.

We can thus have thread safe code without explicit locking.

You feel vindicated when the language platform adopts a similar approach to solve a similar problem.


Day 0 and a new team

January 9, 2008

Next week is planning week. It’s fun because you get to hang out with people you haven’t had a chance to talk to, you get a good overview of the quarter that’s just gone by, and the high-level view of the quarter coming up. It is also fun for two other reasons.

Day 0 is like attending a technical conference for one day. You get to attend technical sessions conducted by your peers on stuff you probably don’t know, and if you do, you can still learn something new or help someone else out. This time around, there are enough sessions to use up 2 rooms in parallel for an entire day, and a third room for a good portion of the day. For the first time, I am involved in driving two of the sessions, one on Java Concurrency and the other on Scalability. They are both subjects on which I have actively worked on in the past few months.

Working with concurrent code is something that truly excites me. It is devilishly hard to write concurrent code and harder still to test reliably. Race conditions occur perennially and are often impossible to reproduce on demand. Detailed logs are the name of the game and one has to sift through them with a magnifying glass to identify and fix problems. Though I struggle just as much as the next dev, it really is one of the few times I feel at home. The session itself won’t (or at least shouldn’t) be focussing on the difficulty of debugging concurrent code. It should be a short presentation on some of the new tools available for concurrency in Java since Java 5, followed by a hands-on session where participants will solve a couple of simple exercises by using these new features.

Scalability is another beast altogether. On numerous occasions, it has left me a broken, defeated man by the time I finish working on a story or fixing a particular problem. Interestingly, while it is a problem solved everyday and everywhere (or how can enterprise and web applications flourish everywhere?) and most people spout all the buzzwords and theory related to scaling, I am yet to meet a developer who didn’t cave in to the enormity of the task during implementation. The problems start with getting a test-bed similar to the eventual production environment, scaling the application successfully and then testing out the deployment to reach meaningful numbers. If debugging a concurrent application is hard, debugging a scaled-out application is darn-near impossible. There are tools out there to help you scale, but finding the appropriate ones and making them work for you isn’t easy. I am happy that I acquired a lot more practical knowledge in a field where I only knew theory (and not a lot of it!). This session will therefore be a presentation on some tools and processes that we researched and/or used in scaling out our Java application. I expect a lot of discussion and debate from the attendees, or we may be in for a very short session.

Another fun part of planning week is finding out your new team for the next quarter. Regardless of how well your previous team gelled, it is exciting to know that one or more of your team can change and you get to learn from people you haven’t worked with before. One of the last discussions that usually happens at planning with the new team is the coding conventions that we agree to follow as a team. My personal preferences are:

  • Not to break up one line of code into multiple lines: makes it less readable to me
  • To include a space after every comma in the parameter list
  • To put the { character on the same line, eg. for {
  • To put a space in front of the {
  • To not put a space in front of the ( when declaring or invoking a method
  • To not use {} when its not needed:
             for (int i=0; i<20; i++)
                 if (i%2 == 0)
                     return i;
                     return i*-1;
  • To have spaces after the semicolons in the for-statement, but nowhere else inside the parentheses.
  • To have spaces either side of the comparison operator and any logical operators in the if-statement, but nowhere else inside the parentheses.

I am sure I could think of a few more, but these will suffice for now. They are always a bone for contention because everyone has their pet peeves. There are also quite a few people who have told me that we could have Subversion format the code to an agreed-on template while checking in and then have Eclipse format it to our tastes when we update the code. This way, all the checked-in code is in a uniform format but the developers can read it in the format they are most comfortable. Should probably try it some day.


A weird game

January 8, 2008

Here is a weird game I played as white against a much higher-rated opponent. My usual disclaimer, that this wasn’t checked against the computer, applies as always.

1.d4 Nf6 2.c4 g6 3.Nc3 Bg7 4.e4 d6 5.Be2 O-O 6.Nf3 e5 7.O-O Nc6 8.d5 Ne7


We have reached a standard King’s Indian position. Black’s plan is to expand on the kingside, with f5 (after Nh5 or Ne8) and g5, and basically try and steamroll its way to the White king. White will try and play on the queenside with b4 and Nf3-e1-d3, preparing for the c5 break. White’s play may seem removed from the action while black generates an attack against its king, but the basic idea is to chop down the black army from the rear.

9.b4 a5?


This move doesn’t make sense. Instead of continuing with his plan for kingside expansion, black is wasting valuable time on the queenside.

10.Qb3 axb4 11.Qxb4 Nd7 12.Ne1 f5 13.Nd3 Nf6 14.f3 b6?


Once again, black suspends his activities on the kingside to move a pawn on the queenside. All this does is give me more targets to attack.

15.Be3 f4 16.Bf2 g5 17.a4 Ng6 18.a5 bxa5 19.Rxa5 Rxa5 20.Qxa5 h5 21.Nb5


With the time wasted by black, it is no surprise that white wins the race by breaking through first on the queenside.

21…c5 22.Qxd8 Rxd8 23.Ra1 Bf8 24.Ra8 Kf7


The king comes over to support the rook, and unpin the bishop in the process.

25.Nb2 Ke8 26.Na4 Bb7 27.Ra7 Rd7 28.Nb6 Rh7


Black has to be content with moving his pieces around, waiting for white to bring his pieces into the attack.

29.Bd1 Kd8 30.Ba4 Ne8 31.Be1 Rg7 32.Ba5 Nh8


Quite a funny-looking position. Both sides are struggling to find optimal squares for their pieces. White is still better as he is the only one playing for an attack.

33.Nc3 Ng6 34.Na8+ Kc8


Here I decide to give up my rook for two pieces.

35.Bxe8 Kb8 36.Rxb7+ Kxb7 37.Nc7


Black cannot return the favor by capturing twice on c7 because of his hanging knight on g6.

37…Be7 38.N3b5 g4 39.Ne6 Rg8 40.Bf7 Ra8 41.Bc3 Nh4 42.Bxh5 gxf3+=


Black slowly seems to be untangling himself. His rook and knight are on good, active squares again, and while his bishop looks very passive, it does a useful job of holding together the pawn structure. But I still prefer my position here. At the appropriate time, if I can sac a piece for a couple of his pawns and get my pawns rolling, it won’t be easy to stop them.

43.gxf3 Ra2 44.h3 Ra4


From this point onwards, for the rest of the game, the black knight stays trapped at h4. It has no squares it can move to, and if I can kick the black bishop off the diagonal, I can then round up the knight.

45.Nxc5! dxc5 46.Bxe5 Rxc4 47.Bxf4


I have achieved my goal. My connected passers on the d, e and f files are very strong, plus my dark-squared bishop is on the right diagonal to stop his c-pawn from queening.

47…Rb4 48.d6!+-


He cannot capture my knight as after dxe7, he cannot stop the pawn from queening.

48… Bd8 49.Nc7!


Again, he cannot trade his bishop for the knight, because after dxc7 followed by Bg4, black cannot stop the c-pawn from queening.

49… Rd4 50.Ne6 Rd1+ 51.Kf2 Bb6 52.Kg3


And here, black resigned. He is losing his knight and he faces an uphill battle trying to stop the avalanche of pawns.


SWOT analysis

January 7, 2008

With my wife studying for her MBA, I often get asked if I am picking up an MBA by osmosis. While that’s probably exaggerating it a little bit, I do pick up some interesting bits and bobs, such as SWOT analysis. In a nutshell, it’s a business planning tool where you compile a list of strengths, weaknesses, opportunities and threats into a 2×2 matrix, and use that to build on your strengths, work on your weaknesses, seize the opportunities in front of you and defend against the threats. Furthermore, as a career building exercise, the students were split into teams and told to write up a SWOT analysis of themselves. Each person’s analysis was then discussed within the group, with the other team members adding to the lists, as necessary.

It seemed like an interesting exercise to undertake. I worked on the list and then added more items after specific suggestions from the missus. While I won’t publish the final result here (In a commercial world where everybody does their darndest to shield their weaknesses from the world, it wouldn’t be the wisest move), I would recommend it as a useful means for introspection.

It was pleasing to note that out of the 9 weaknesses we listed, I am, in some form or the other, already working on 4 of them. What was less pleasing was that I listed less than half the items on my “strengths” list. Since I was trying to be brutally honest and making no attempt at modesty when I compiled the list (conclusion: I wasn’t aware of my perceived strengths), I haven’t been doing a good enough job of playing to my strengths.

Let’s hope that I remember to visit the lists periodically and to make constant evaluations on the different items on that list.