OutOfMemoryError: unable to create native thread, and the limits behind it

The name says OutOfMemoryError, the heap graph is flat, and restarting the service fixes it for a day or two. "Unable to create native thread" is rarely about the Java heap. It means the JVM asked the operating system for a new thread and the operating system said no. The work is to find out which limit refused, and why the process needed so many threads in the first place.
What the message tells you
Every java.lang.Thread that is not a virtual thread is backed by one operating-system thread. On Linux the JVM creates it with pthread_create. When that call fails, Thread.start() throws an OutOfMemoryError whose text, in recent JDKs, is defined in the HotSpot source (os.hpp) as:
java.lang.OutOfMemoryError: unable to create native thread:
possibly out of memory or process/resource limits reached
JDK 8 and earlier used the shorter wording "unable to create new native thread". Both mean the same thing. The second half of the modern message is the honest part: the failure can come from memory, or from one of several counters that cap the number of tasks a user, a service or a container may run.
The JVM usually leaves one more clue. Just before the exception, HotSpot logs a warning through unified logging, which goes to standard output by default:
[warning][os,thread] Failed to start thread "worker-17" - pthread_create failed (EAGAIN)
for attributes: stacksize: 1024k, ...
(Abbreviated; the thread name and the list of attributes depend on the thread and the JDK release.)
The error code in brackets is the most useful word in the whole incident. EAGAIN points to a count limit (or, less often, to the kernel failing to find memory for a new task). ENOMEM points to memory or address-space mappings. Search your container or service logs for pthread_create failed before anything else.
The limits that can refuse a thread
Linux does not distinguish much between processes and threads: both are tasks, and most limits count tasks. Check them from the inside out.
The per-user limit: ulimit -u
RLIMIT_NPROC, shown by ulimit -u, is documented in getrlimit(2) as a limit on the number of processes, "or, more precisely on Linux, threads", for the real user ID. The count covers every task that user owns on the host, not just your JVM. Two services running under the same account share one budget, and a batch job under that account can starve the web server. For a running process, the effective value is in /proc/<pid>/limits on the "Max processes" line, which is more reliable than ulimit in your own shell.
The cgroup limit: pids.max
The cgroup pids controller caps the number of tasks in a cgroup and its descendants. In cgroup v2, pids.max holds the limit, pids.current the current count, and pids.events counts how many times the limit was hit. Container runtimes set it through options such as Docker's --pids-limit, and Kubernetes nodes can apply a per-pod limit. Inside a container, read /sys/fs/cgroup/pids.max and /sys/fs/cgroup/pids.current; a non-zero max count in pids.events is direct proof. See the JVM in containers for the memory side of the same cgroup.
systemd's TasksMax
A service started by systemd lives in its own cgroup, and systemd fills in pids.max from the unit's TasksMax= setting. When the unit does not set it, the default comes from DefaultTasksMax=, which the systemd documentation describes as 15% of the smallest of kernel.pid_max, kernel.threads-max and the root cgroup's pids.max. With the kernel's default pid_max that works out at 4,915 tasks, a number a busy JVM with generous pools can reach. systemctl show <unit> -p TasksMax -p TasksCurrent shows both values.
System-wide limits
/proc/sys/kernel/threads-max caps all threads on the machine, and /proc/sys/kernel/pid_max caps the IDs available to them. Hitting either affects every process at once, so it tends to show up as failures everywhere, not only in Java.
Memory and mappings
Each new thread needs a stack. Its size comes from -Xss (the same setting as -XX:ThreadStackSize); the java reference documents a default of 1024 KB on Linux/x64 and 2048 KB on Linux/AArch64. The stack is reserved as virtual memory and only committed as it is used, so a thousand threads do not cost a gigabyte of RAM. They do cost address-space mappings, and the kernel's vm.max_map_count limit on mappings per process can be reached by an application with many threads and many memory-mapped files. Overcommit settings and a 32-bit JVM are the other classic memory causes, the latter now mostly of historical interest.
Counting threads, from outside and inside
Before changing any limit, measure how many threads the process has and whether that number grows. From the shell:
ps -o nlwp= -p <pid> # number of threads (lightweight processes)
ls /proc/<pid>/task | wc -l # the same, from /proc
grep -i threads /proc/<pid>/status
From the JVM itself, jcmd <pid> Thread.print lists every platform thread with its name and stack. If Native Memory Tracking is enabled, jcmd <pid> VM.native_memory summary includes a Thread category with the thread count and the memory reserved and committed for stacks; Native Memory Tracking explains how to read it and how to take a baseline and compare it later. JMX exposes the same figures through ThreadMXBean (live, peak and total started), and plotting the live count against time tells you at once whether you face a slow leak or a sudden burst.
Finding the code that creates them
A thread count that climbs steadily and never falls back is a leak, and raising limits only delays the next failure. The thread names are the fastest way to the culprit:
jcmd <pid> Thread.print | grep '^"' | sed -E 's/^"([^"]*)".*/\1/; s/[0-9]+/N/g' | sort | uniq -c | sort -rn | head
This groups threads by name with every number replaced by N. A line such as 3200 pool-N-thread-N is the signature of Executors.newFixedThreadPool or newCachedThreadPool called again and again with the default thread factory: each call creates a new pool, numbered pool-1, pool-2 and so on, and nobody calls shutdown(). Other frequent patterns:
- an executor or HTTP client created per request or per message instead of once per application;
- an unbounded cached thread pool in front of a slow dependency, which adds a thread for every request waiting on it;
java.util.Timeror scheduled executors created in a loop, each with its own background thread;- libraries that start their own threads for every client instance, such as connection pools or messaging clients instantiated many times.
Generic names like Thread-42 say little. Giving every pool a ThreadFactory that sets a meaningful prefix costs one line and makes the next diagnosis a matter of seconds. When names are not enough, the stack of each leaked thread in the dump, read as described in reading thread dumps, usually shows the task it runs and therefore the component that submitted it. A Flight Recorder recording also logs jdk.ThreadStart events, with the stack trace of the code that started each thread when stack traces are enabled for that event.
Fixing it, in the right order
- Stop the leak. Create executors once, share them, shut them down when their owner closes. Bound pool sizes and put a bounded queue in front of them so overload becomes rejected work instead of new threads.
- Consider virtual threads for blocking work. On JDK 21 and later, virtual threads run on a small set of carrier threads, so ten thousand concurrent blocking calls no longer mean ten thousand operating-system threads. They need their own care, covered in virtual threads in production.
- Then size the limits deliberately. If the application genuinely needs more threads, raise
TasksMax, the container's PID limit orLimitNPROCfor that service, not the global defaults, and keep the limit finite so a future leak still fails before it takes the host down. - Only then touch -Xss. Smaller stacks free mappings and address space but do nothing for a count limit, and a stack that is too small replaces one error with
StackOverflowError.
Last, alert on the thread count itself. A JVM that normally runs 300 threads and has reached 2,000 is telling you about the problem days before pthread_create fails.
Sources
- OpenJDK HotSpot source, os.hpp (error message) and os_linux.cpp (pthread_create warning)
- getrlimit(2), Linux man-pages project (RLIMIT_NPROC)
- Linux kernel documentation: Control Group v2 (PID controller) and sysctl/kernel (threads-max, pid_max)
- systemd-system.conf(5) (DefaultTasksMax) and the
javacommand reference for JDK 21 (-Xss defaults)
Threads & CPUReading thread dumps
Take a series, read states and lock lines, and let the JVM point out deadlocks for you.
MemoryNative Memory Tracking
Heap is only part of the process: read the NMT summary, diff against a baseline, and know what it cannot see.
ProductionThe JVM in containers
How limits are detected, what MaxRAMPercentage really sets, and a budget that keeps the OOM killer away.