misterli's Blog.

分析redis 大key

字数统计: 1.7k阅读时长: 8 min
2023/05/05

使用rdbtools工具

redis-rdb-tools 是一个 python 的解析 rdb 文件的工具,在分析内存的时候,我们主要用它生成内存快照。

主要有以下三个功能:

  • 生成内存快照
  • 转储成 json 格式
  • 使用标准的 diff 工具比较两个 dump 文件

安装

1
2
3
pip install rdbtools python-lzf
# 在安装python-lzf之前,要安装python-devel依赖包,否则安装会失败
yum install python-devel

rdbtools工具包括了3个可执行文件:

rdb – 解析整个rdb文件
redis-memory-for-key – 解析server里的单个key
redis-profiler –解析rdb文件成html格式

命令help

1、rdb –help:解析整个rdb文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
usage: rdb [options] /path/to/dump.rdb

Example : rdb --command json -k "user.*" /var/redis/6379/dump.rdb

positional arguments:
-- 要处理的dump文件
dump_file RDB Dump file to process

optional arguments:
-- 帮助
-h, --help show this help message and exit
-- 要处理的命令,-c后的有效参数为:json, diff,justkeys, justkeyvals, memory,protocol
-c CMD, --command CMD
Command to execute. Valid commands are json, diff,
justkeys, justkeyvals, memory and protocol
-- 输出文件
-f FILE, --file FILE Output file
-- 数据库号,可以提供多个数据库。如果未指定,则包括所有数据库。
-n DBS, --db DBS Database Number. Multiple databases can be provided.
If not specified, all databases will be included.
-- 要导出的key。这可以是一个正则表达式
-k KEYS, --key KEYS Keys to export. This can be a regular expression
-- key不导出。这可以是一个正则表达式
-o NOT_KEYS, --not-key NOT_KEYS
Keys Not to export. This can be a regular expression
-- 解析的数据类型。可能的值为string,hash,set,sortedset,list。可以输入多种类型提供。如果未指定,则为所有数据类型
-t TYPES, --type TYPES
Data types to include. Possible values are string,
hash, set, sortedset, list. Multiple typees can be
provided. If not specified, all data types will be
returned
-- 将key的内存输出限制为大于或等此值(以字节为单位)
-b BYTES, --bytes BYTES
Limit memory output to keys greater to or equal to
this value (in bytes)
-- 将内存按大小输出前N个key
-l LARGEST, --largest LARGEST
Limit memory output to only the top N keys (by size)
-- 将字符串转义为编码:raw(默认),print,utf8或base64。
-e {raw,print,utf8,base64}, --escape {raw,print,utf8,base64}
Escape strings to encoding: raw (default), print,
utf8, or base64.
-- 使用command protocol参数,从所有键中删除到期的key
-x, --no-expire With protocol command, remove expiry from all keys
-- 使用command protocol参数,将N秒添加到key的到期时间
-a N, --amend-expire N
With protocol command, add N seconds to key expiry
time

2,redis-memory-for-key –help:– 解析server里指定的单个key

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Usage: redis-memory-for-key [options] redis-key
Examples :
redis-memory-for-key user:13423
redis-memory-for-key -s localhost -p 6379 user:13423


Options:
-- 帮助
-h, --help show this help message and exit
-- 服务地址,默认127.0.0.1
-s HOST, --server=HOST
Redis Server hostname. Defaults to 127.0.0.1
-- 服务端口,默认6379
-p PORT, --port=PORT Redis Server port. Defaults to 6379
--服务密码
-a PASSWORD, --password=PASSWORD
Password to use when connecting to the server
-- 数据库号,默认0
-d DB, --db=DB Database number, defaults to 0

3、redis-profiler –help:

1
2
3
4
5
6
7
8
9
10
11
12
13
Usage: redis-profiler [options] /path/to/dump.rdb

Example 1 : redis-profiler -k "user.*" -k "friends.*" -f memoryreport.html /var/redis/6379/dump.rdb
Example 2 : redis-profiler /var/redis/6379/dump.rdb

Options:
-- 帮助
-h, --help show this help message and exit
-- 输出
-f FILE, --file=FILE Output file
-- 组合在一起的键。多个正则表达式
-k KEYS, --key=KEYS Keys that should be grouped together. Multiple regexes
can be provided

使用

每次运行以上工具时都需要指定一个命令,以指示对解析的RDB数据应执行的操作。操作有:

转储的JSON

rdb -c json test-redis.rdb -f test-redis.json

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@dev-tools ~]# rdb -c json test-redis.rdb  -f  test-redis.json
[root@dev-tools ~]# head -10 test-redis.json
[{
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370":"172800",
"yundt-cube-center-trade-test:_dtmqconsume/ORDERID_a20cae4d-bf89-44d8-9552-80b27ecd9241":"1",
"yundt-cube-center-scheduler-test:_dtmqconsume/ORDERID_f32d75ca-783b-4e19-8019-4975f82fc0ea":"1",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494":"172800",

过滤解析

正则表达式匹配key,并且仅打印键和值:

rdb -c justkeyvals --key "yundt-cube-center-inventory-test.*" test-redis.rdb

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@dev-tools ~]# rdb -c  justkeyvals --key "yundt-cube-center-inventory-test.*"  test-redis.rdb  >demo.txt
[root@dev-tools ~]# head -10 demo.txt

yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT1296917031476795318516d5a98-330a-44a2-abcb-f052398aa0da,2ee21658-46f3-417d-a5cc-5997e3dbcc7f 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT1296917031476795318ad9ec3a8-06db-4e1b-b566-61eb62779c90,b6837582-ce51-4509-9e8b-34a1c8aef520 172800,

生成内存报告

使用-c memory 运行会生成CSV报告,其中包含该键使用的近似内存。–bytes C 和 –largest N 可用于将输出限制为大于C字节的键或N个最大键。

rdb -c memory test-redis.rdb --bytes 128 -f redis-mem.csv

1
2
3
4
5
6
7
8
9
10
11
12
[root@dev-tools ~]# rdb -c memory test-redis.rdb --bytes 128 -f redis-mem.csv 
[root@dev-tools ~]# head -10 redis-mem.csv
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053,200,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370,168,string,8,8,
0,string,yundt-cube-center-trade-test:_dtmqconsume/ORDERID_a20cae4d-bf89-44d8-9552-80b27ecd9241,168,string,8,8,2022-10-09T20:39:00.057000
0,string,yundt-cube-center-scheduler-test:_dtmqconsume/ORDERID_f32d75ca-783b-4e19-8019-4975f82fc0ea,168,string,8,8,2022-10-09T21:34:00.100000
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370,168,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507,200,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370,168,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf,152,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494,200,string,8,8,

生成的CSV具有以下列:

  • database:数据库编号
  • type:数据类型
  • key:键
  • size_in_bytes:使用的内存:包括键,值和任何其他开销
  • encoding:RDB编码类型
  • num_elements:key中的value的个数
  • len_largest_element:key中的value的长度
  • expiry:过期时间

注意:内存使用情况是近似的。通常,实际使用的内存将略高于报告的内存。可以按键或数据库编号或数据类型过滤报告。内存报告应有助于检测由应用程序逻辑引起的内存泄漏。它还将帮助优化Redis的内存使用。

对key按内存大小排序

1
sort -t ',' -k4 -rn redis-mem.csv  >redis-mem-sort.csv

查找单键使用的内存

查找特定键使用的内存(运行整个内存报告非常耗时),使用redis-memory-for-key

1
2
3
4
5
6
7
8
redis-memory-for-key person:1
redis-memory-for-key -s localhost -p 6379 -a mypassword person:1
Key person:1
Bytes 111
Type hash
Encoding ziplist
Number of Elements 2
Length of Largest Element 8
CATALOG
  1. 1. 安装
  2. 2. 命令help
  3. 3. 使用
    1. 3.1. 转储的JSON
    2. 3.2. 过滤解析
    3. 3.3. 生成内存报告
    4. 3.4. 查找单键使用的内存