JVM performance optimizations at Twitter's scale

Complete presentation about application optimization based on twitter experiments; it was Java-centric, but I tried to generalize the presented principles.

and for those interested... I still do not like Java!




Takeaway: Twitter went from 60 second of Garbage Collection (GC) per hour to 5 seconds per 3.5 days, mainly because of the internal slab allocator.

Presentation slides here.

Twitter's biggest enemy: LatencyWhat are the main latency contributors (from worst to less):

1. Garbage collector (GC)
2. in process locking and thread scheduling
3. I/O
4. application inefficient algorithms

but only the first one can be addressed by JVM tuning (which happens to be the expertise of the speaker)

Areas of performance tuning:

• Memory tuning (footprint, allocation rate, garbage collector rate)
• Lock contention

Memory footprint tuning

• Maybe you just use too much data!
• Maybe your data representation is fat! (bad data model)
• Maybe you have a memory leak (not covered in remaining presentation)

Global advice for performance tuning:

Profile your applications, especially when 3rd parties are involved abd not well known (ex: primitive SCALA array types eating lots of memory; another ex: Guava's concurrent map eating a lot of resource - especially if you do not need concurrency).

Even more important advice:

Writing a new memory manager is not a good idea unless you already optimized your data and your JVM... if you start doing this, it is a strong smell that you are doing something wrong!

Fighting latency

Performance tradeoff: Memory vs Time (convenient and valid, but oversimplified view); the performance triangle (press memory usage down, push throughput up and keep latency low) is a better model, but still too simple for Twitter optimization team. In the end they use a (optimization increase) = C x T x R where C=compactness (good data model), T= throughput and R=responsiveness.

The rest of the presentation was about tuning and optimizing Java JVM Garbage Collector (GC) - if you are interested, let me know (I will not document all of this in this post - lucky you!).

Side note:

At Twitter, they seem to also have a big gap between developer's "done" and "production ready" done.