(磁帶已死,磁盤是新磁帶,閃存是新磁盤,隨機存儲器局部性是爲王道)
userId(用戶id) | weiboCount(微博數) |
1 | 2000 |
2 |
10 |
3 |
288 |
.... | ... |
1000000 | 1000 |
userId | hashKey | field |
1 | 0 | 1 |
2 | 0 |
2 |
3 | 0 |
3 |
... | .... | ... |
99 | 0 | 99 |
100 | 1 | 0 |
101 | 1 | 1 |
.... | ... | ... |
9999 | 99 | 99 |
100000 | 1000 | 0 |
注意:
3. 獲取方法:
#獲取userId=5003用戶的微博數
(1) get u:5003
(2) hget allUser u:5003
(3) hget u:50 f:3
4. 內存佔用量對比(100萬用戶 userId u:1~u:1000000)
#方法一 Memory
used_memory:118002640
used_memory_human:112.54M
used_memory_rss:127504384
used_memory_peak:118002640
used_memory_peak_human:112.54M
used_memory_lua:36864
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-3.6.0
---------------------------------------------------
#方法二 Memory
used_memory:134002968
used_memory_human:127.80M
used_memory_rss:144261120
used_memory_peak:134002968
used_memory_peak_human:127.80M
used_memory_lua:36864
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-3.6.0
--------------------------------------------------------
#方法三 Memory
used_memory:19249088
used_memory_human:18.36M
used_memory_rss:26558464
used_memory_peak:134002968
used_memory_peak_human:127.80M
used_memory_lua:36864
mem_fragmentation_ratio:1.38
mem_allocator:jemalloc-3.6.0
那麼爲什麼第三種能少那麼多內存呢?之前有人說用了共享對象的原因,現在我將key,field,value全部都變成了字符串,仍然還是節約很多內存。
之前我也懷疑過是hashkey,field的字節數少造成的,但是我們下面通過一個實驗看就清楚是爲什麼了。當我將hash-max-ziplist-entries設置爲2並且重啓後,所有的hashkey都變爲了hashtable編碼。
同時我們看到了內存從18.36M變爲了122.30M,變化還是很大的。
127.0.0.1:8000> object encoding u:8417
"ziplist"
127.0.0.1:8000> config set hash-max-ziplist-entries 2
OK
127.0.0.1:8000> debug reload
OK
(1.08s)
127.0.0.1:8000> config get hash-max-ziplist-entries
1) "hash-max-ziplist-entries"
2) "2"
127.0.0.1:8000> info memory
# Memory
used_memory:128241008
used_memory_human:122.30M
used_memory_rss:137662464
used_memory_peak:134002968
used_memory_peak_human:127.80M
used_memory_lua:36864
mem_fragmentation_ratio:1.07
mem_allocator:jemalloc-3.6.0
127.0.0.1:8000> object encoding u:8417
"hashtable"
內存使用量:
package com.carlosfu.redis;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.junit.Test;
import redis.clients.jedis.Jedis;
/**
* 一次string-hash優化
*
* @author carlosfu
* @Date 2015-11-8
* @Time 下午7:27:45
*/
public class TestRedisMemoryOptimize {
private final static int TOTAL_USER_COUNT = 1000000;
private final static String HOST = "127.0.0.1";
private final static int PORT = 6379;
/**
* 純字符串
*/
@Test
public void testString() {
int mBatchSize = 2000;
Jedis jedis = null;
try {
jedis = new Jedis(HOST, PORT);
List<String> kvsList = new ArrayList<String>(mBatchSize);
for (int i = 1; i <= TOTAL_USER_COUNT; i++) {
String key = "u:" + i;
kvsList.add(key);
String value = "v:" + i;
kvsList.add(value);
if (i % mBatchSize == 0) {
System.out.println(i);
jedis.mset(kvsList.toArray(new String[kvsList.size()]));
kvsList = new ArrayList<String>(mBatchSize);
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (jedis != null) {
jedis.close();
}
}
}
/**
* 純hash
*/
@Test
public void testHash() {
int mBatchSize = 2000;
String hashKey = "allUser";
Jedis jedis = null;
try {
jedis = new Jedis(HOST, PORT);
Map<String, String> kvMap = new HashMap<String, String>();
for (int i = 1; i <= TOTAL_USER_COUNT; i++) {
String key = "u:" + i;
String value = "v:" + i;
kvMap.put(key, value);
if (i % mBatchSize == 0) {
System.out.println(i);
jedis.hmset(hashKey, kvMap);
kvMap = new HashMap<String, String>();
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (jedis != null) {
jedis.close();
}
}
}
/**
* segment hash
*/
@Test
public void testSegmentHash() {
int segment = 100;
Jedis jedis = null;
try {
jedis = new Jedis(HOST, PORT);
Map<String, String> kvMap = new HashMap<String, String>();
for (int i = 1; i <= TOTAL_USER_COUNT; i++) {
String key = "f:" + String.valueOf(i % segment);
String value = "v:" + i;
kvMap.put(key, value);
if (i % segment == 0) {
System.out.println(i);
int hash = (i - 1) / segment;
jedis.hmset("u:" + String.valueOf(hash), kvMap);
kvMap = new HashMap<String, String>();
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (jedis != null) {
jedis.close();
}
}
}
}
方案 | 優點 | 缺點 |
string |
直觀、容易理解 |
|
hash |
直觀、容易理解、整合整體 |
|
segment-hash |
內存佔用量小,雖然理解不夠直觀,但是總體上是最優的。 |
理解不夠直觀。 |