Memory management and persistence in Aiven for Valkey™
Learn how Aiven for Valkey™ addresses the challenges of high memory usage and high change rate. Discover how it implements robust memory management and persistence strategies.
Aiven for Valkey™ functions primarily as a database cache. Data fetched from a database is stored in the caching system. Subsequent queries with the same parameters first check this cache, bypassing the need for a repeat database query. This efficiency can lead to challenges such as increased memory usage and frequent data changes, which Aiven for Valkey is specifically designed to manage.
Data eviction policy in Aiven for Valkey
Data eviction policy is one of the most important caching settings and it is available in the Aiven Console.
Aiven for Valkey sets the maxmemory config, which determines the maximum amount
of data that can be stored. The data eviction policy specifies what happens when this
limit is reached. By default, all Aiven for Valkey services have the eviction policy
set to No eviction. If you continue storing data without removing anything, write
operations fail once the maximum memory is reached.
When data is consumed at a rate similar to how it is written, it is acceptable to use
the current eviction policy. However, for other use cases, it is better to use
the allkeys-lru eviction policy. This policy starts dropping the old keys based on
the least recently used strategy when maxmemory is reached. Another way to handle
the situation is to drop random keys.
If you continue to write data, you will eventually reach the maxmemory limit, regardless
of the eviction policy you use.
High memory and high change rate behavior
For all new Aiven for Valkey services, the maxmemory setting is configured to 70% of
available RAM (minus management overhead) plus 5% for the replication backlog for
multi-node services.
This configuration limits the memory usage to below 100%, accommodating operations
that require additional memory:
- When a new Valkey replica connects to the primary, the service process on the primary node forks and creates an RDB snapshot, which is streamed to the new Valkey replica node.
- A similar forking process occurs when the state of the Valkey service is persisted to disk, which for Aiven for Valkey happens every 10 minutes by default.
- To reduce snapshot frequency, set
frequent_snapshots=false. - To disable persistence, set
valkey_persistence=off.
When a fork occurs, all memory pages of the new process are identical to the parent and do not consume extra memory. However, any changes in the parent process cause the memory configurations to diverge, increasing actual memory allocation.
Example
If the forked process took 4 minutes to write the RDB snapshot to disk, with new data written at 5 megabytes per second, the system memory usage can increase by approximately 1.2 gigabytes during that time.
The duration of backup and replication operations depends on the total amount of memory in use. As the size of the plan increases, the memory usage also increases, which can cause a memory divergence. Therefore, the amount of memory reserved for completing these operations without using a swap is directly proportional to the total memory available.
If memory usage exceeds the available memory to the extent that a swap is required, the node may become unresponsive and require replacement.
In scenarios involving on-disk persistence, the child process attempts to dump all its memory onto the disk. If the parent process undergoes significant changes and uses up available RAM, it may force the child process to swap unfinished memory pages to disk. This transition makes the child process more I/O bound and further slows its operation. Increased divergence from the parent process results in more frequent swapping, further slowing the process. In such cases, the node is limited to writing and reading from swap memory.
The rate at which data can be written is influenced by the size of the data values and the specifics of the Aiven plan.
Writing at about 5 megabytes per second is manageable in most situations. However, attempting to write at about 50 megabytes per second is likely to lead to failures, particularly when memory usage approaches the allowed maximum or during the initialization of a new node.
Initial synchronization
During system upgrades or in a high-availability setup in case of node failure, a new Valkey replica node needs to be synchronized with the current primary. The new node starts with an empty state, connects to the primary, and requests a full copy of its current state. After receiving this copy, the new node begins following the replication stream from the primary to achieve complete synchronization.
Once initial synchronization is complete, the new node begins to follow the replication stream from the primary. The value of the replica client output buffer limit is set to 20% of the total memory Valkey is allowed to use. For a 28 gigabyte plan, this is around 4 gibibytes. Under memory pressure conditions, the value is set to 10%.
If the volume of changes during the initial sync exceeds the replica client output
buffer limit, the primary node logs Client [...] scheduled to be closed ASAP for overcoming of output buffer limits. The replica is disconnected, the buffer is
cleared, and the replication process starts from scratch.
Writing new changes at 5 megabytes per second, totaling 1.8 gigabytes over 6 minutes, allows the new node to start up successfully. A higher constant rate of change can cause the synchronization to fail unless the replica client output buffer limit is increased.
Mitigation
Aiven does not impose a rate limit on traffic for Valkey services because limiting only relevant write operations would require a specialized proxy, and restricting all traffic can negatively impact non-write operations.
Rate limiting might be considered in the future, but for now, it is advisable to manage your workloads carefully and keep write rates moderate to prevent node failures due to excessive memory usage or challenges in initializing new nodes.
If you frequently need to write large volumes of data, contact Aiven support to discuss service configuration options that can accommodate your needs.
Service memory limits
The practical memory limit will always be less than the service physical memory limit.
All services are subject to operating overhead:
- A small amount of memory is required by the operating system kernel to manage system resources, including networking functions and disk cache.
- Aiven's cloud data platform requires memory to monitor availability, provide metrics, logging and manage backups.
A server or node's usable memory can be calculated as:
usable memory = RAM - overhead
Where:
overheadis 350 MiB (≈ 0.34 GiB).
Services may utilize optional components, service integrations, connection pooling, or plug-ins, which are not included in overhead calculations.
If a service is overcommitted, the operating system, management layer, backups or availability monitoring, may fail status checks or operations due to resource contention. In severe instances, the node may fail completely with an out-of-memory condition.
Out of memory conditions
Many processes request more memory from the kernel than they will ever use or need. In these cases, the kernel overallocates memory. This allows it to satisfy multiple processes requesting more memory than is available, which is not used or is freed by the time any other process actually needs it.
However, if enough processes start using all their allocated memory
simultaneously there may not be enough physical memory available and an
Out Of Memory (OOM) condition occurs.
This situation is critical and must be resolved immediately.
The solution that the Linux kernel employs is to invoke the
Out of Memory Killer (or OOM Killer). This reviews all running processes
and kills one or more of them to free up system memory and keep the system running.
The OOM Killer selects process to kill based on an oom_score; a
calculation that balances how much memory the process is using with how
long the process has been running.
Processes that have been running for a long time are less likely to be killed. Subprocesses are summed with parent processes in terms of memory usage, so a process which forks many subprocesses, but itself does not use a lot of memory, may still be killed.
In most instances, the hosted data service, or a child process, will have the highest memory footprint and be a prime candidate for termination when the OOM Killer inspects the running processes.
Aiven's cloud data platform leverages kernel namespaces or containers to isolate processes from each other. Isolation has several benefits, including:
- A smaller footprint for security‑related concerns
- A smaller blast radius for failure
- Greater control of system resources
Left unchecked, the OOM Killer may opt to kill the primary service.
This is undesirable as unclean termination of the primary service can
lead to data loss, inconsistency, or corrupted backups.
Further, if Aiven's management platform detects that the primary service is unavailable for , the service will be marked as down and a failover will occur.
To mitigate this scenario, namespaces are used,
some with additional memory limits, in combination with an oom_score_adjust on the primary
process, to coax the OOM Killer into selection of less critical
processes.
This will still result in a service restart, but in a more controlled process, where the database is shut down, rather than killed; exposure to data loss is limited and recovery is faster when the service restarts, often avoiding failover.
Out of Memory conditions can still lead to unexpected behavior, including data unavailable or data loss conditions.
Avoid running low on memory
The OOM killer only runs when the system is critically low on memory. To prevent it from running, either reduce your memory usage or increase the available memory.
For most databases, the service memory footprint can often be reduced by:
- Reducing concurrency or implementing connection pooling
- Tuning queries to limit result sets
- Tuning indexes for query load
- Dropping unused objects from storage
In cases where the working set no longer fits into memory, consider scaling your service.
Related pages