Exa, a company building an extensive semantic web index, announced a 47% increase in embedding throughput to support indexing tens of billions of web documents. This advancement is crucial as embedding high-quality documents into vector space representations underpins Exa’s multi-tower semantic machine learning models used to deliver high-quality data for natural language queries.
The embedding process represents documents and queries with vectors in a shared semantic space, facilitating complex search tasks beyond simple keyword matching. To scale this process economically over billions of documents, Exa optimized its infrastructure on AWS AI accelerator hardware—specifically on Neuron-based Inferentia2 and Trainium2 chips—and leveraged in-house software improvements.
Initially doubling their embedding load due to a re-embedding backfill of approximately 30 billion priority documents, Exa revisited hardware and software configurations. Running workloads on Kubernetes clusters configured to pack Neuron cores efficiently, the team improved throughput through successive optimizations such as weight layout adjustment and precision autocasting, which balanced performance gains with accuracy maintenance at cosine similarity thresholds above 0.999.
Further performance gains arose from utilizing the Neuron Kernel Interface (NKI), including the implementation of a drop-in attention kernel and custom memory layout changes. Notably, Exa developed a specialized fused SwiGLU MLP kernel that reduced high-bandwidth memory stalls, improving chip utilization. Additional refinements targeted FLOPS reduction by customizing attention kernels to the model’s exact architectural shapes, decreasing computation waste.
An innovative aspect of Exa’s approach was employing an agentic development workflow to coordinate testing and implementation. Agents conducted correctness verification, profiling, and performance benchmarking using Flyte ML ops workflows, balancing tasks across lightweight parameter tuning agents and advanced code-generation agents for kernel development.
Recognizing variability in cloud resource availability, Exa designed a scheduler that optimizes workload distribution across multiple AWS regions and instance types, dynamically selecting the most cost-effective hardware for embedding tasks. This strategic flexibility helps avoid compute shortages and improves long-term operational efficiency.
While these hardware-specific optimizations have yielded significant throughput and cost benefits, the approach is constrained by dependencies on specific accelerator architectures and regional cloud resource availability. Further, maintaining the balance between computational savings and model accuracy requires continuous monitoring and validation.
Source & further reading
Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips ↗Source published 2026-11-18. Claims and limitations should be checked against the original announcement or research.