【轉載】種序列化協議(protobuf,xstream,jackjson,jdk,hessian)相關數據對比
最近研究了下google protobuf協議,順便對比了一下json,xml,java序列化相關的數據對比,從幾個緯度進行對比。
別人的相關測試數據: http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking
測試緯度
- 序列化時間
- 反序列化時間
- bytes大小
測試代碼
準備protobuf文件
import "InnerMessage.proto";
package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "MessageProtos";
option optimize_for = SPEED ; //CODE_SIZE,LITE_RUNTIME
option java_generic_services = false;
message Message {
required string strObj = 1 [default="hello"];
optional int32 int32Obj = 2;
optional int64 int64Obj = 3;
optional uint32 uint32Obj = 4;
optional uint64 uint64Obj = 5;
optional sint32 sint32Obj = 6;
optional sint64 sint64Obj = 7;
optional fixed32 fixed32Obj = 8;
optional fixed64 fixed64Obj = 9;
optional sfixed32 sfixed32Obj = 10;
optional sfixed64 sfixed64Obj = 11;
optional bool boolObj = 12;
optional bytes bytesObj = 13;
optional float folatObj = 14 [deprecated=true];
repeated double doubleObj = 15 [packed=true]; //
optional InnerMessage innerMessage = 16;
}
Innermessage.proto代碼
import "EnumType.proto";
package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "InnerMessageProtos";
message InnerMessage {
optional string name = 1 [default = "name"];
optional int32 id = 2;
optional EnumType type = 3 [default = UNIVERSAL];
}
Enumtype.proto代碼package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "EnumTypeProtos";
enum EnumType {
UNIVERSAL = 0;
WEB = 1;
IMAGES = 2;
LOCAL = 3;
NEWS = 4;
PRODUCTS = 5;
VIDEO = 6;
}
基本上把protobuf支持的類型都囊括了,包括嵌套類型,枚舉類型,以及各種int,uint,bool,bytes。
依賴關係是Message.proto依賴了InnerMessage對象,而InnerMessage對象裏包含了一個自定義枚舉類型EnumType。
關於類型的使用可參見:
http://code.google.com/intl/zh/apis/protocolbuffers/docs/reference/java-generated.html
http://code.google.com/intl/zh/apis/protocolbuffers/docs/proto.html
生成protobuf javabean
cd /home/ljh/work/code/src/main/java
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/EnumType.proto
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/InnerMessage.proto
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/Message.proto
通過protobuf自帶的protoc進行編譯,指定了protobuf文件的路徑, 具體的文檔: http://code.google.com/intl/zh/apis/protocolbuffers/docs/proto.html#generating
運行腳本後就會生成對應的3個javabean文件: MessageProtos , InnerMessageProtos , EnumTypeProtos。
最後構造測試的protobuf bean代碼
Java代碼
private static MessageProtos.Message getProtobufBean() {
com.agapple.protobuf.data.MessageProtos.Message.Builder messageBuilder = MessageProtos.Message.newBuilder();
messageBuilder.setStrObj("message");
messageBuilder.setFolatObj(1f);
messageBuilder.addDoubleObj(1d);
messageBuilder.addDoubleObj(2d);
messageBuilder.setBoolObj(true);
messageBuilder.setBytesObj(ByteString.copyFrom(new byte[] { 1, 2, 3 }));
messageBuilder.setInt32Obj(32);
messageBuilder.setInt64Obj(64l);
messageBuilder.setSint32Obj(232);
messageBuilder.setSint64Obj(264);
messageBuilder.setFixed32Obj(532);
messageBuilder.setFixed64Obj(564);
messageBuilder.setSfixed32Obj(2532);
messageBuilder.setSfixed64Obj(2564);
messageBuilder.setUint32Obj(632);
messageBuilder.setUint64Obj(664);
com.agapple.protobuf.data.InnerMessageProtos.InnerMessage.Builder innerMessageBuilder = InnerMessageProtos.InnerMessage.newBuilder();
innerMessageBuilder.setId(1);
innerMessageBuilder.setName("inner");
innerMessageBuilder.setType(EnumType.PRODUCTS);
messageBuilder.setInnerMessage(innerMessageBuilder);
return messageBuilder.build();
}
準備純Pojo Bean
同樣的,爲了和json , xml以及java序列化有個很好的對比,新建了3個純的pojo bean: MessagePojo , InnerMessagePojo , EnumTypePojo。
屬性和proto的bean保持一致。
構建bean對象
private static MessagePojo getPojoBean() {
MessagePojo bean = new MessagePojo();
bean.setStrObj("message");
bean.setFolatObj(1f);
List<Double> doubleObj = new ArrayList<Double>();
doubleObj.add(1d);
doubleObj.add(2d);
bean.setDoubleObj(doubleObj);
bean.setBoolObj(true);
bean.setBytesObj(new byte[] { 1, 2, 3 });
bean.setInt32Obj(32);
bean.setInt64Obj(64l);
bean.setSint32Obj(232);
bean.setSint64Obj(264);
bean.setFixed32Obj(532);
bean.setFixed64Obj(564);
bean.setSfixed32Obj(2532);
bean.setSfixed64Obj(2564);
bean.setUint32Obj(632);
bean.setUint64Obj(664);
InnerMessagePojo innerMessagePojo = new InnerMessagePojo();
innerMessagePojo.setId(1);
innerMessagePojo.setName("inner");
innerMessagePojo.setType(EnumTypePojo.PRODUCTS);
bean.setInnerMessage(innerMessagePojo);
return bean;
}
具體的測試代碼
定義測試Template接口
interface TestCallback {
String getName();
byte[] writeObject(Object source);
Object readObject(byte[] bytes);
}
統一的測試模板
private static void testTemplate(TestCallback callback, Object source, int count) {
int warmup = 10;
// 先進行預熱,加載一些類,避免影響測試
for (int i = 0; i < warmup; i++) {
byte[] bytes = callback.writeObject(source);
callback.readObject(bytes);
}
restoreJvm(); // 進行GC回收
// 進行測試
long start = System.nanoTime();
long size = 0l;
for (int i = 0; i < count; i++) {
byte[] bytes = callback.writeObject(source);
size = size + bytes.length;
callback.readObject(bytes);
// System.out.println(callback.readObject(bytes));
bytes = null;
}
long nscost = (System.nanoTime() - start);
System.out.println(callback.getName() + " total cost=" + integerFormat.format(nscost) + "ns , each cost="
+ integerFormat.format(nscost / count) + "ns , and byte sizes = " + size / count);
restoreJvm();// 進行GC回收
}
在測試模板方法中,使用了warmup預熱的概念,就是預先執行目標方法一定的次數,用於避免因爲jit的優化影響系統測試。 同時包含了每次測試模板調用完成後system.gc保證下一輪的功能測試
相應的restoreJvm方法:
private static void restoreJvm() {
int maxRestoreJvmLoops = 10;
long memUsedPrev = memoryUsed();
for (int i = 0; i < maxRestoreJvmLoops; i++) {
System.runFinalization();
System.gc();
long memUsedNow = memoryUsed();
// break early if have no more finalization and get constant mem used
if ((ManagementFactory.getMemoryMXBean().getObjectPendingFinalizationCount() == 0)
&& (memUsedNow >= memUsedPrev)) {
break;
} else {
memUsedPrev = memUsedNow;
}
}
}
private static long memoryUsed() {
Runtime rt = Runtime.getRuntime();
return rt.totalMemory() - rt.freeMemory();
}
最後相應的測試例子:
final int testCount = 1000 * 500;
final MessageProtos.Message protoObj = getProtobufBean();
final MessagePojo pojoOBj = getPojoBean();
// Serializable測試
testTemplate(new TestCallback() {
public String getName() {
return "Serializable Test";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bout = new ByteArrayOutputStream();
ObjectOutputStream output = new ObjectOutputStream(bout);
output.writeObject(source);
return bout.toByteArray();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
ObjectInputStream input = new ObjectInputStream(bin);
return input.readObject();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// protobuf測試
testTemplate(new TestCallback() {
public String getName() {
return "protobuf test";
}
@Override
public byte[] writeObject(Object source) {
if (source instanceof MessageProtos.Message) {
MessageProtos.Message message = (MessageProtos.Message) source;
return message.toByteArray();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
return MessageProtos.Message.parseFrom(bytes);
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
}
return null;
}
}, protoObj, testCount);
// json測試
final ObjectMapper objectMapper = new ObjectMapper();
final JavaType javaType = TypeFactory.type(pojoOBj.getClass());
// JSON configuration not to serialize null field
objectMapper.getSerializationConfig().setSerializationInclusion(JsonSerialize.Inclusion.NON_NULL);
// JSON configuration not to throw exception on empty bean class
objectMapper.getSerializationConfig().disable(SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS);
// JSON configuration for compatibility
objectMapper.configure(Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
objectMapper.configure(Feature.ALLOW_UNQUOTED_CONTROL_CHARS, true);
testTemplate(new TestCallback() {
public String getName() {
return "Jackson Test";
}
@Override
public byte[] writeObject(Object source) {
try {
return objectMapper.writeValueAsBytes(source);
} catch (JsonGenerationException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
return objectMapper.readValue(bytes, 0, bytes.length, javaType);
} catch (JsonParseException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// Xstream測試
final XStream xstream = new XStream();
testTemplate(new TestCallback() {
public String getName() {
return "Xstream test";
}
@Override
public byte[] writeObject(Object source) {
return xstream.toXML(source).getBytes();
}
@Override
public Object readObject(byte[] bytes) {
return xstream.fromXML(new ByteArrayInputStream(bytes));
}
}, pojoOBj, testCount);
2011年3月10號補充 =========================================================
增加了hessian 3.1.5版本基於二進制序列化的測試
<dependency>
<groupId>com.caucho</groupId>
<artifactId>hessian</artifactId>
<version>3.1.5</version>
</dependency>
測試了3種情況:
- hessian 2協議
- hessian 2協議 + deflat壓縮
- hessian 1協議
測試代碼:
// hessian 2 with no deflat
testTemplate(new TestCallback() {
public String getName() {
return "hessian 2 with no deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Hessian2Output out = new Hessian2Output(bos);
// out.startMessage();
out.writeObject(source);
// out.completeMessage();
out.flush();
return bos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
Hessian2Input in = new Hessian2Input(bin);
// in.startMessage();
Object obj = in.readObject();
// in.completeMessage();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// hessian 2 with deflat
final Deflation envelope = new Deflation();
testTemplate(new TestCallback() {
public String getName() {
return "hessian 2 with deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Hessian2Output out = new Hessian2Output(bos);
out = envelope.wrap(out);
out.writeObject(source);
out.flush();
out.close(); // 記得關閉
return bos.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
Hessian2Input in = new Hessian2Input(bin);
in = envelope.unwrap(in);
Object obj = in.readObject();
in.close();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// hessian 1 with no deflat
testTemplate(new TestCallback() {
public String getName() {
return "hessian 1 with no deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
HessianOutput out = new HessianOutput(bos);
out.writeObject(source);
out.flush();
return bos.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
HessianInput in = new HessianInput(bin);
Object obj = in.readObject();
in.close();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
測試結果
序列化數據對比
bytes字節數對比
具體的數字:
protobuf | jackson | xstream | Serializable | hessian2 | hessian2壓縮 | hessian1 | |
序列化(單位ns) | 1154 | 5421 | 92406 | 10189 | 26794 | 100766 | 29027 |
反序列化(單位ns) | 1334 | 8743 | 117329 | 64027 | 37871 | 188432 | 37596 |
bytes | 97 | 311 | 664 | 824 | 374 | 283 | 495 |
- protobuf 不管是處理時間上,還是空間佔用上都優於現有的其他序列化方式。內存暫用是java 序列化的1/9,時間也是差了一個數量級,一次操作在1us左右。缺點:就是對象結構體有限制,只適合於內部系統使用。
- json格式在空間佔用還是有一些優勢,是java序列化的1/2.6。序列化和反序列化處理時間上差不多,也就在5us。當然這次使用的jackson,如果使用普通的jsonlib可能沒有這樣好的性能,jsonlib估計跟java序列化差不多。
- xml相比於java序列化來說,空間佔用上有點優勢,但不明顯。處理時間上比java序列化多了一個數量級,在100us左右。
- 以前一種的java序列化,表現得有些失望
- hessian測試有點意外,具體序列化數據上還步入json。性能上也不如jackjson,輸得比較徹底。
- hessian使用壓縮,雖然在字節上有20%以上的空間提升,但性能上差了4,5倍,典型的以時間換空間。總的來說還是google protobuf比較給力
總結
以後在內部系統,數據cache存儲上可以考慮使用protobuf。跟外部系統交互上可以考慮使用json。
有興趣的同學,可以研究一下google protobuf的marshall的方式:http://code.google.com/intl/zh/apis/protocolbuffers/docs/encoding.html