Engula

Docs
Docs

Engula ValueSight (Memory Savings Assessment Tool)

Engula

Engula is a cache kernel product developed in-house by the Engula team, designed specifically for users running open-source Redis. Its core advantage is reducing the hardware cost of existing Redis clusters with extremely low risk. Key features include:

  • Minimal core code changes: ensures system stability and compatibility; the current public verification baseline is Redis 7.2.11, with the specific pass rate and differences detailed in proof-live.
  • Memory savings: significantly reduces metadata overhead and is powered by an in-house Compressed HybridLog storage engine that further lowers memory usage.
  • Performance optimization: optimizes end-to-end performance through improvements such as a redesigned HashTable implementation, an optimized network thread communication mechanism, a higher CPU prefetch hit rate, and an asynchronous thread framework.

Based on early users' real-world test results, Engula saves more than 50% of memory on average, with a performance overhead of less than 10%.

Engula ValueSight

Engula ValueSight is a tool that helps users quickly and accurately assess the value of Engula. Based on their own real business data, users can compare the memory usage of Engula and Redis with a single command, and quantify Engula's advantages in real business scenarios.

Core features include:

  • Measuring the RSS memory usage of the same dataset in both Engula and Redis.
  • Generating memory usage comparison charts for clear visual analysis.
  • Producing a memory usage analysis summary to help users understand the optimization results.

Quick Start Summary

Item Content
Tool overview Import a Redis RDB snapshot and generate a per-key memory comparison report of Engula vs. Redis.
Prerequisites Docker can run on the local machine; prepare a dump.rdb; prepare an analysis_output/ output directory.
Quick start docker run -it --rm -v "$(pwd)/dump.rdb:/tmp/dump.rdb" -v "$(pwd)/analysis_output:/engula/analysis_output" registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight
Reading the output Focus on the total memory savings ratio, the distribution by data type, the per-key details, and any abnormally large keys.
Full example See "Running Example" below.

Command-Line Arguments

Argument Purpose
-v <rdb>:/tmp/dump.rdb Specify a single RDB file.
-v <dir>:/tmp/rdb_dir Specify the RDB file directory in batch mode.
-v <dir>:/engula/analysis_output Specify the report output directory.
--batch Scan multiple RDB files under /tmp/rdb_dir and generate assessment results in batch.

Running the Tool

Runtime Environment

Engula ValueSight is provided as a Docker image and can run on Linux or macOS systems with Docker installed. The specific environment requirements are as follows:

  • CentOS version ≥ 7.9
  • Ubuntu version ≥ 18.04
  • The official Docker release is recommended (installation guide; prepare the image in advance if network access is slow)
  • A host with at least 2 CPU cores is recommended
TIPS

If the runtime environment cannot access the public internet and cannot docker pull the image, refer to the 4.5 Exporting and Importing the Image File section of this document to export and import the image.

Steps

  1. Prepare the RDB file: generate a Redis dump.rdb file on the host where it will run (you can use the BGSAVE command to generate it).
  2. Download the Engula ValueSight image:
1docker pull registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight
  1. Create the log directory:
1# The local directory name can be changed; just make sure the mapped directory in the final docker command is not misspelled
2mkdir -m 777 analysis_output
  1. Run the analysis:
1docker run -it --rm \
2  -v "<RDB file path>:/tmp/dump.rdb" \
3  -v "$(pwd)/analysis_output:/engula/analysis_output" \
4  registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight

Note: Replace <RDB file path> with the actual path to your dump.rdb file.

Running Example

Assuming Docker is already installed on the host and there is a dump.rdb file in the current directory, you can run the following commands to generate a compression assessment report for the dump.rdb file:

Note: Engula ValueSight is updated from time to time. If you have not used it for a while, it is recommended to run the pull command first to update it.

1# Create the log directory
2mkdir analysis_output
3
4# Update the Engula ValueSight image
5docker pull registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight
6
7# Run the analysis
8docker run -it --rm \
9  -v "$(pwd)/dump.rdb:/tmp/dump.rdb" \
10  -v "$(pwd)/analysis_output:/engula/analysis_output" \
11  registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight

Running Process

Running Process

Result

Result

After the analysis finishes, press Enter to view the details, and press Q / Ctrl+C to exit.

Batch Testing of Compression Ratio

Steps

  1. Create the necessary directories:
1# Create the directory for RDB files
2mkdir rdb_dir
3
4# Create the log directory
5mkdir analysis_output

  1. Place the RDB files:

    • Move the RDB files you want to analyze into the rdb_dir directory.
    • Subdirectory structures are supported.
    • The program automatically scans all files with the .rdb extension.

  2. Pull the analysis tool image:

1docker pull registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight
  1. Run the batch compression assessment command:
1docker run -it --rm \
2  -v "$(pwd)/rdb_dir:/tmp/rdb_dir" \
3  -v "$(pwd)/analysis_output:/engula/analysis_output" \
4  registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight --batch
  • The paths for rdb_dir and analysis_output must match the directories you actually created.
  • Carefully check the mapped path configuration to avoid problems caused by incorrect paths.

Shard Calculation Feature

The last underscore followed by a number in the RDB file name is treated as the number of shards and is factored into the final calculation, in order to obtain a more accurate estimate of the scale of memory savings.

Shard Example

In the example above, r_profile_100w_1000.rdb is treated as having 1000 shards.

Running Process

Running Process

Result

Result

Contents of the analysis_output Directory

For dump_1.rdb, there will be four files:

1_dump_1.rdb_cmd_output_for_engula.log // Log of dump_1.rdb during the engula assessment
2_dump_1.rdb_cmd_output_for_redis.log  // Log of dump_1.rdb during the redis assessment
3_dump_1.rdb_engula-stats-rdb.json     // Statistics of dump_1.rdb in engula
4_dump_1.rdb_redis-stats-rdb.json      // Statistics of dump_1.rdb in redis

A separate set of related files is generated for each RDB file. If you care about the details, or if errors occur, you can obtain the relevant information from these files. In addition, an info.log file is generated to record general log information.

Running in the Background

Since multiple RDB files need to be assessed in batch, the processing time may be long. It is recommended to run the task in the background to improve efficiency and avoid a long wait.

1nohup docker run --rm \
2  -v "$(pwd)/rdb_dir:/tmp/rdb_dir" \
3  -v "$(pwd)/analysis_output:/engula/analysis_output" \
4  registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight --batch \
5  > analysis_output/docker.log 2> analysis_output/docker.error.log &

You can check the task progress in the following ways:

1# View the output log
2tail -f analysis_output/docker.log
3
4# View the error log
5tail -f analysis_output/docker.error.log

FAQ

4.1 Docker Memory Setting Too Small

Symptom An error occurs at runtime: Error2: Child process quit abnormally, ExitCode=9

Cause and Solution This error is usually caused by insufficient Docker memory due to an overly large RDB file. Recommendations:

  • Increase the memory allocation in the Docker Desktop settings (≥ 4GB recommended).
  • Split a large RDB file and process it in batches.

4.2 Debugging Methods

After the analysis is complete, the analysis_output directory contains the program's runtime logs and statistics, which can be used for debugging. If you run into problems, you can contact the Engula team for assistance.

4.3 Installing the System Docker Version on Red Hat

On Red Hat Linux 9, the Docker version installed with the following command does not meet the requirements for this test.

1yum install docker -y
2
3docker --version
4# podman version 4.9.4-rhel

It is recommended to install Docker according to the official Docker documentation. The specific steps are as follows:

  1. Uninstall old versions:
1sudo yum remove docker \
2              docker-client \
3              docker-client-latest \
4              docker-common \
5              docker-latest \
6              docker-latest-logrotate \
7              docker-logrotate \
8              docker-engine \
9              podman \
10              runc
  1. Install yum-utils and add the official Docker repository:
1sudo yum install -y yum-utils
2sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
  1. Install the latest version of Docker:
1sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  1. Start the Docker service:
1sudo systemctl start docker

4.4 Batch Mode Error: "rdb file not found"

By default, the program in the image does not run as the root user; this error may be caused by that.

Solution: Add --user root to the batch command.

4.5 Exporting and Importing the Image File

Step 1. Export

On a machine that can access the public internet:

1docker pull --platform=amd64 registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight
2docker save registry.cn-guangzhou.aliyuncs.com/montplex/engula-valuesight > engula-valuesight.tar.gz

Tip: If the CPU architecture of the internet-connected machine differs from that of the target test machine, you need to set the --platform argument to the CPU architecture of the target test machine; for X86, this is amd64.

Step 2. Copy

Copy the exported tar.gz file to the target test machine.

Step 3. Import

On the target test machine:

1docker load -i engula-valuesight.tar.gz
2docker images | grep valuesight

Contact Us

If you encounter any problems while using Engula or Engula ValueSight, or if you would like to learn more about our products, feel free to contact us at any time:

Email: support@engula.com

The Engula team will respond to your inquiries promptly on business days from 9:00 to 18:00 (UTC+8).

ON THIS PAGE