Protocol Buffers (“protobuf”) is a Google technology that lets you define messages declaratively, and then build library code for a myriad of different programming-languages. The way that messages are serialized is efficient and effortless, and protobuf allows for simple string assignment (without predefining a length), arrays and optional values, and sub-messages.
The only tough part comes during implementation. As protobuf is only concerned with serialization/unserialization, it’s up to you to deal with the logistics of sending the message, and this means that, for socket communication, you often have to:
- Copy and paste the code to prepend a length.
- Copy/paste/adapt existing code that embeds a type-identifier on outgoing requests, and reads the type-identifier on incoming requests in order to automatically handle/route messages (if this is something that you want, which I often do).
This quickly becomes redundant and mundane, and it’s why we’re about to introduce protobufp (“Protocol Buffers Processor”).
We can’t improve on the explanation on the project-page. Therefore, we’ll just provide the example.
We’re going to build some messages, push into a StringIO-based byte-stream (later to be whatever type of stream you wish), read them into the protobufp “processor” object, and retrieve one fully-unserialized message at a time until depleted:
from test_msg_pb2 import TestMsg from protobufp.processor import Processor def get_random_message(): rand = lambda: randint(11111111, 99999999) t = TestMsg() t.left = rand() t.center = "abc" t.right = rand() return t messages = [get_random_message() for i in xrange(5)]
Create an instance of the processor, and give it a list of valid message-types (the order of this list should never change, though you can append new types to the end):
msg_types = [TestMsg] p = Processor(msg_types)
Use the processor to serialize each message and push them into the byte-stream:
s = StringIO() for msg in messages: s.write(p.serializer.serialize(msg))
Feed the data from the byte stream into the processor (normally, this might be chunked-data from a socket):
Pop one decoded message at a time:
j = 0 while 1: in_msg = p.read_message() if in_msg is None: break assert messages[j].left == in_msg.left assert messages[j].center == in_msg.center assert messages[j].right == in_msg.right j += 1
Now there’s one less annoying task to distract you from your critical path.