Skip to main content

Memory and out-of-memory conditions in Aiven for Metrics

Understand the memory limits and out-of-memory conditions that apply to your Aiven for Metrics service.

Service memory limits​

The practical memory limit will always be less than the service physical memory limit.

All services are subject to operating overhead:

  • A small amount of memory is required by the operating system kernel to manage system resources, including networking functions and disk cache.
  • Aiven's cloud data platform requires memory to monitor availability, provide metrics, logging and manage backups.

A server or node's usable memory can be calculated as:

usable memory = RAM - overhead

Where:

  • overhead is 350 MiB (≈ 0.34 GiB).

Services may utilize optional components, service integrations, connection pooling, or plug-ins, which are not included in overhead calculations.

If a service is overcommitted, the operating system, management layer, backups or availability monitoring, may fail status checks or operations due to resource contention. In severe instances, the node may fail completely with an out-of-memory condition.

Out of memory conditions​

Many processes request more memory from the kernel than they will ever use or need. In these cases, the kernel overallocates memory. This allows it to satisfy multiple processes requesting more memory than is available, which is not used or is freed by the time any other process actually needs it.

However, if enough processes start using all their allocated memory simultaneously there may not be enough physical memory available and an Out Of Memory (OOM) condition occurs.

warning

This situation is critical and must be resolved immediately.

The solution that the Linux kernel employs is to invoke the Out of Memory Killer (or OOM Killer). This reviews all running processes and kills one or more of them to free up system memory and keep the system running.

The OOM Killer selects process to kill based on an oom_score; a calculation that balances how much memory the process is using with how long the process has been running.

Processes that have been running for a long time are less likely to be killed. Subprocesses are summed with parent processes in terms of memory usage, so a process which forks many subprocesses, but itself does not use a lot of memory, may still be killed.

In most instances, the hosted data service, or a child process, will have the highest memory footprint and be a prime candidate for termination when the OOM Killer inspects the running processes.

Aiven's cloud data platform leverages kernel namespaces or containers to isolate processes from each other. Isolation has several benefits, including:

  • A smaller footprint for security‑related concerns
  • A smaller blast radius for failure
  • Greater control of system resources

Left unchecked, the OOM Killer may opt to kill the primary service. This is undesirable as unclean termination of the primary service can lead to data loss, inconsistency, or corrupted backups.

Further, if Aiven's management platform detects that the primary service is unavailable for , the service will be marked as down and a failover will occur.

To mitigate this scenario, namespaces are used, some with additional memory limits, in combination with an oom_score_adjust on the primary process, to coax the OOM Killer into selection of less critical processes.

This will still result in a service restart, but in a more controlled process, where the database is shut down, rather than killed; exposure to data loss is limited and recovery is faster when the service restarts, often avoiding failover.

warning

Out of Memory conditions can still lead to unexpected behavior, including data unavailable or data loss conditions.

Avoid running low on memory​

The OOM killer only runs when the system is critically low on memory. To prevent it from running, either reduce your memory usage or increase the available memory.

For most databases, the service memory footprint can often be reduced by:

  • Reducing concurrency or implementing connection pooling
  • Tuning queries to limit result sets
  • Tuning indexes for query load
  • Dropping unused objects from storage

In cases where the working set no longer fits into memory, consider scaling your service.

Related pages