Keeping AI products affordable: batching, quantisation, caching and the JVM services around the model

Production · Published 2026-10-24

Aerial view of a residential district with terraced houses, curving streets and parked cars — Property search is one of the consumer markets where AI now meets ordinary users. Photo: rawpixel, CC0.

A product built on a language or vision model pays twice for memory: once for the accelerator memory that holds the model and its working state, and again for the ordinary servers that run everything around it. With memory prices up sharply since late 2025 (see the AI memory squeeze), both bills matter. This guide covers the techniques that reduce the first and the JVM settings that reduce the second, then looks at what consumer products in a few sectors actually do with AI.

On the model side: use less accelerator memory per answer

Batch what does not need an answer now

Serving many requests together keeps an accelerator busy and amortises the memory that holds the model weights. Teams that call a hosted model can get the same effect commercially: OpenAI's Batch API offers a 50% cost discount compared to its synchronous APIs, in exchange for a completion window of up to 24 hours. Classification of new listings, nightly summaries, embedding a catalogue and re-scoring old content rarely need a live answer and are good candidates.

Quantise the weights

The Hugging Face Transformers documentation describes quantisation as storing weights in a lower precision to reduce the memory needed to load and use a model, while trying to preserve accuracy: from full-precision fp32 to half precision (fp16 or bf16), and further to integer formats such as int8 or int4. Each halving of the bits per weight halves the memory the weights take. Quality must be checked on your own evaluation set, since some tasks degrade sooner than others.

Manage the KV cache

During generation a transformer keeps keys and values for every token of every running request, the KV cache, and that memory grows with context length and concurrency. The vLLM paper (Kwon et al., 2023) notes that this memory can be significantly wasted by fragmentation and redundant duplication, and introduces PagedAttention, which allocates the cache in pages in the manner of operating-system virtual memory. The authors report that vLLM improves throughput of popular models by 2 to 4 times at the same latency compared with the systems they evaluated. For a team, the practical levers are a serving engine that pages the cache, a bounded maximum context, and a limit on concurrent sequences.

Cache prompts and results

Long, repeated prefixes (system instructions, tool definitions, a document asked about several times) can be cached by the provider. Anthropic's prompt caching documentation prices cache reads at 0.1 times the base input token price for most models, with lower multipliers for some, and cache writes with a 5-minute lifetime at 1.25 times. Put the stable part of the prompt first and the variable part last, or nothing can be reused.

Result caching is older and cheaper still: if the same question about the same object comes back, store the answer keyed by a hash of the normalised input and the model version. A valuation for an unchanged listing, or a summary of an unchanged document, should not be generated twice.

Use the smallest model that passes

Route each task to the smallest model that meets its quality bar, and reserve the large one for the requests that need it. Extraction, classification and routing are often handled well by small models; a short evaluation set per task makes the choice measurable instead of a matter of taste.

Around the model: the Java services

Gateways, retrieval services and pre- and post-processing are frequently written in Java or Kotlin. They do not hold model weights, but they run in many replicas, and each replica's memory limit is multiplied by the replica count.

Gateways: mostly waiting

A gateway that forwards requests to a model spends most of its time waiting for tokens. That is the workload virtual threads were designed for: one thread per request without a large pool of platform threads, each with its own stack. Stream the model's output to the client as it arrives instead of assembling the whole response in the heap, and cap request and response sizes. The heap of such a service is usually small and short-lived; a modest heap with G1, or Serial in a one-CPU container, is often enough. Measure with GC logs before deciding.

Retrieval: know what one vector costs

A retrieval service holding embeddings in memory has an easy budget to compute. A 1,024-dimension vector of 32-bit floats takes 4 KiB (1,024 × 4 bytes) before any index overhead, and a float[] on the heap adds an object header. Large, long-lived arrays of this kind are cheaper outside the heap, where the garbage collector does not copy or scan them: memory-mapped files or segments from the Foreign Function & Memory API (JEP 454, final in JDK 22). If you go off-heap, count it in the container budget; NMT will not see memory mapped by a native library.

Pre- and post-processing: stream, do not buffer

Image resizing, document parsing and upload handling are where heaps spike. Read uploads as streams and write them to object storage rather than holding whole files in byte arrays; large arrays become humongous objects in G1 (see tuning G1). Bound the number of concurrent jobs per replica with a semaphore, so memory scales with the limit you chose rather than with traffic.

Container sizing

Size every one of these services from a measured budget: heap from the live set, plus Metaspace, code cache, threads, direct buffers and native libraries. The JVM in containers gives the method. Separate services with different shapes (a streaming gateway and a batch embedding job) into different deployments, so neither is sized for the other's peak.

Consumer AI in practice: a few examples

The products below are described from their own public pages; none of these pages says what language their back-end services are written in.

Real estate. Zillow announced on 27 June 2019 that its Zestimate home valuation uses neural networks and computer vision, trained on the home photos on Zillow, to distinguish between high- and low-end finishes. The same release gave an error rate below 2% for homes listed for sale, meaning half of all Zestimates fell within 2% of the eventual sale price.
Real estate, startup. ting.ad is an AI-driven real-estate app. Its App Store and Google Play listings describe an assistant, Ting AI, with which users talk through a project (buying, selling or renting); the assistant builds a personal map of the neighbourhoods and criteria involved, which can be shared with family and friends. The app adds voice and video calls, including live group calls with local agents, after which matches appear on that map. On its website, ting.ad asks buyers to describe the ordinary week they want, with criteria such as commute, schools, nature and nightlife, and returns properties whose neighbourhoods match it.
Fashion retail. Zalando announced on 1 October 2024 that its assistant, powered by Zalando's own models and OpenAI's large language models, offers logged-in customers personalised fashion advice in local languages across its 25 markets.
Music. Spotify introduced DJ on 22 February 2023, combining its personalisation technology, OpenAI technology for commentary written with its music editors, and a synthetic voice from its Sonantic acquisition.
Travel. Booking.com launched its AI Trip Planner on 27 June 2023, built on its existing machine learning models and partially powered by OpenAI's ChatGPT API.

The common pattern is a large model at one step of the journey, surrounded by ordinary services: search, listings, user accounts, media handling. Photos and videos submitted for an estimate, catalogue data for a stylist, listening history for a playlist: each must be received, stored, transformed and sent to the model. That surrounding layer is where JVM sizing decides a large part of the bill.

Checklist

Move every request that can wait into a batch.
Evaluate a quantised model and a smaller model per task before scaling hardware.
Bound context length and concurrent sequences; use a serving engine that pages the KV cache.
Order prompts for prefix caching; cache results keyed by input hash and model version.
Stream model output and uploads; never hold a whole file or response in the heap.
Size each JVM service from NMT and GC-log measurements, and count off-heap vectors in the budget.