Protocol Buffers (“protobuf”) is a Google technology that lets you define messages declaratively, and then build library code for a myriad of different programming-languages. The way that messages are serialized is efficient and effortless, and protobuf allows for simple string assignment (without predefining a length), arrays and optional values, and sub-messages.

The only tough part comes during implementation. As protobuf is only concerned with serialization/unserialization, it’s up to you to deal with the logistics of sending the message, and this means that, for socket communication, you often have to:

1. Copy and paste the code to prepend a length.
2. Copy/paste/adapt existing code that embeds a type-identifier on outgoing requests, and reads the type-identifier on incoming requests in order to automatically handle/route messages (if this is something that you want, which I often do).

This quickly becomes redundant and mundane, and it’s why we’re about to introduce protobufp (“Protocol Buffers Processor”).

We can’t improve on the explanation on the project-page. Therefore, we’ll just provide the example.

We’re going to build some messages, push into a StringIO-based byte-stream (later to be whatever type of stream you wish), read them into the protobufp “processor” object, and retrieve one fully-unserialized message at a time until depleted:

from test_msg_pb2 import TestMsg

from protobufp.processor import Processor

def get_random_message():
    rand = lambda: randint(11111111, 99999999)

    t = TestMsg()
    t.left = rand()
    t.center = "abc"
    t.right = rand()

    return t

messages = [get_random_message() for i in xrange(5)]

Create an instance of the processor, and give it a list of valid message-types (the order of this list should never change, though you can append new types to the end):

msg_types = [TestMsg]
p = Processor(msg_types)

Use the processor to serialize each message and push them into the byte-stream:

s = StringIO()

for msg in messages:
    s.write(p.serializer.serialize(msg))

Feed the data from the byte stream into the processor (normally, this might be chunked-data from a socket):

p.push(s.getvalue())

Pop one decoded message at a time:

j = 0
while 1:
    in_msg = p.read_message()
    if in_msg is None:
        break

    assert messages[j].left == in_msg.left
    assert messages[j].center == in_msg.center
    assert messages[j].right == in_msg.right

    j += 1

Now there’s one less annoying task to distract you from your critical path.