Wednesday, November 16, 2011

JVM performance optimizations at Twitter's scale

Complete presentation about application optimization based on twitter experiments; it was Java-centric, but I tried to generalize the presented principles.

and for those interested... I still do not like Java!

Takeaway: Twitter went from 60 second of Garbage Collection (GC) per hour to 5 seconds per 3.5 days, mainly because of the internal slab allocator.

Presentation slides here.

Twitter's biggest enemy: LatencyWhat are the main latency contributors (from worst to less):
  1. Garbage collector (GC)
  2. in process locking and thread scheduling
  3. I/O
  4. application inefficient algorithms
but only the first one can be addressed by JVM tuning (which happens to be the expertise of the speaker)

Areas of performance tuning:
  • Memory tuning (footprint, allocation rate, garbage collector rate)
  • Lock contention
Memory footprint tuning
  • Maybe you just use too much data!
  • Maybe your data representation is fat! (bad data model)
  • Maybe you have a memory leak (not covered in remaining presentation)
Global advice for performance tuning:
Profile your applications, especially when 3rd parties are involved abd not well known (ex: primitive SCALA array types eating lots of memory; another ex: Guava's concurrent map eating a lot of resource - especially if you do not need concurrency).

Even more important advice:
Writing a new memory manager is not a good idea unless you already optimized your data and your JVM... if you start doing this, it is a strong smell that you are doing something wrong!

Fighting latency
Performance tradeoff: Memory vs Time (convenient and valid, but oversimplified view); the performance triangle (press memory usage down, push throughput up and keep latency low) is a better model, but still too simple for Twitter optimization team. In the end they use a (optimization increase) = C x T x R where C=compactness (good data model), T= throughput and R=responsiveness.

The rest of the presentation was about tuning and optimizing Java JVM Garbage Collector (GC) - if you are interested, let me know (I will not document all of this in this post - lucky you!).

Side note:
At Twitter, they seem to also have a big gap between developer's "done" and "production ready" done.

1 comment:

Note: Only a member of this blog may post a comment.