Java Is Fast. Your Code Might Not Be. | Jonathan Vogel
Fixing common Java anti-patterns took an app from 1,198ms to 239ms. Eight patterns that compile fine and pass code review but silently drain CPU and b...
Engineering Blog Post
Summary
Netflix engineers optimized their Ranker service's serendipity scoring feature by replacing inefficient nested loops with batched matrix multiplication. This optimization significantly reduced CPU usage and cluster footprint by improving memory layout and leveraging the JDK Vector API.
Highlights
- Identified serendipity scoring as a significant CPU hotspot, consuming 7.5% of total CPU per node.
- Replaced O(M×N) nested loops for cosine similarity with more efficient batched matrix multiplication operations.
- Improved cache locality and reduced garbage collection pressure by restructuring memory layout with flat buffers and ThreadLocal reuse.
- Leveraged the JDK Vector API to optimize the computation of vector embeddings at scale.
auto-generated
Context
Audience
Software Engineers, Machine Learning Engineers, and Backend Developers
DomainMachine Learning Engineering
Formatlong-form article
Accessfree online
Source Overview
Netflix's Ranker service optimized its serendipity scoring feature, which consumed 7.5% of CPU by converting inefficient nested loops into batched matrix multiplication operations.
The team restructured memory layout using flat buffers and ThreadLocal reuse to improve cache locality and reduce garbage collection pressure, then evaluated various compute kernels including BLAS to achieve lower CPU usage per request. The optimization demonstrates that algorithmic improvements require careful attention to implementation details like memory layout and allocation strategy to yield real performance gains at scale.
Topics
Related
Vector EmbeddingsCosine SimilarityJDK Vector APIMatrix MultiplicationPerformance OptimizationNetflix Engineering
