Today the majority of HTTP APIs use JSON as the data format. However, it's not the only data format available to modern applications. Depending on the use case, a different format might sometimes be a better fit. For example, if an application needs to send binary data (e.g., sound, pictures, etc.), a binary data format could be more suitable than JSON, which is text-based. Or, when you work in an environment where network bandwidth is constrained to a certain limit (e.g., mobile data with monthly data caps), you have to make sure that the messages are as small as possible.
In this post, we'll take a look at Protocol Buffers, a binary data format developed by Google.
A JSON message like {tx:18912,rx:15000,temp:-9.3,humidity:89} has a size of 41 bytes. The same data encoded with Protocol Buffers results in a message that has a size of only 14 bytes. To be fair, the difference would not be that big if the message contained strings. JSON can be made much smaller when you use shorter keys ({t:18912,r:15000,e:-9.3,h:89}), and it can be compressed very efficiently when the message gets bigger and contains arrays of similar objects. But in general, Protocol Buffers messages are smaller than JSON messages.
For the following example, we'll create a sender and receiver program in Java. Protocol Buffers is not limited to Java. The set of officially supported languages has grown since this post was first published, so check the current overview on protobuf.dev for the up-to-date list. Third-party libraries add support for even more languages.
Before applications can send and receive Protocol Buffer messages, the messages need to be defined in a text file and then run through a compiler that creates source code for serializing and deserializing objects in the target language to and from the binary wire format. Usually, the definition file has the suffix .proto. For this example, we'll name our file SensorMessage.proto.
syntax = "proto3";
option java_package = "ch.rasc.protobuf";
message SensorMessage {
uint32 tx = 1;
uint32 rx = 2;
float temp = 3;
uint32 humidity = 4;
}
This definition is written in version 3 of the Protocol Buffers syntax. If you omit the syntax keyword, the compiler uses version 2, which is a bit different. The message keyword encapsulates a message. One .proto file may contain more than one message definition. In proto3, fields are optional by default, although modern protobuf guidance recommends using optional explicitly when field presence matters. Every field has a data type assigned to it. Protocol Buffers supports different data types such as integers, floats, booleans, strings, binary data, and enums. You can find the list of all supported types on the official documentation page.
After the field name comes the tag (for example, = 1). The tag must be a unique number and should never be changed when the message is in use. The tag is what Protocol Buffers sends over the wire to identify a field. Tags from 1 to 15 take one byte in the binary message, tags from 16 to 2047 take two bytes, and so on.
If you later remove a field, reserve both the old field number and the old field name so they do not get reused accidentally. That guidance has become more explicit in the current protobuf documentation.
After we've defined the message, we have to compile it. For this, we need the Protocol Buffer compiler, which can be downloaded from the installation page.
However, because we're writing our example application in Java and using Maven as the build system, we can take advantage of a Maven plugin that compiles our .proto file.
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.11.4</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocVersion>4.32.0</protocVersion>
<inputDirectories>
<include>src/main/protobuf</include>
</inputDirectories>
</configuration>
</execution>
</executions>
</plugin>
After you've added the plugin to pom.xml, copy the .proto file to the <project_home>/src/main/protobuf folder and start the compiler with mvn generate-sources. The plugin automatically downloads and runs the protobuf compiler. protoc creates one Java file for each .proto file and puts it into the <project_home>/target/generated-sources/ folder. The generated Java source code depends on the protobuf Java library, so we also have to add that dependency to pom.xml.
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>4.34.0</version>
</dependency>
Sender ¶
Now we have everything set up and can start coding our application. First, we'll create the sender application. For this example, we want to reduce the bandwidth even more and send the messages over UDP. UDP has an overhead of only 28 bytes, but it is unreliable and packets can get lost. That is not a problem for this demo application because we'll send and receive the messages through the loopback device (127.0.0.1).
SensorMessage sm = SensorMessage.newBuilder().setTx(18912).setRx(15000).setTemp(-9.3f)
.setHumidity(89).build();
try (DatagramSocket socket = new DatagramSocket()) {
byte[] buf = sm.toByteArray();
System.out.printf("Number of Bytes: %d\n", buf.length);
InetAddress address = InetAddress.getLoopbackAddress();
DatagramPacket packet = new DatagramPacket(buf, buf.length, address, 9992);
socket.send(packet);
}
The sender first creates an instance of the message. SensorMessage is the class that protoc generated. The toByteArray() method serializes the object to a byte array that contains the Protocol Buffers message in binary form. The program then sends the data with Java's built-in UDP support to port 9992 on localhost.
Receiver ¶
On the receiving side, we'll use Spring Integration, which provides convenient support for receiving and sending UDP packets.
@EnableIntegration
public class Receiver {
public static void main(String[] args) throws InterruptedException {
try (AnnotationConfigApplicationContext ctx = new AnnotationConfigApplicationContext(
Receiver.class)) {
TimeUnit.MINUTES.sleep(2);
}
}
@Bean
public IntegrationFlow flow() {
return IntegrationFlow.from(Udp.inboundAdapter(9992))
.transform(this::transformMessage).handle(this::handleMessage).get();
}
private SensorMessage transformMessage(byte[] payload) {
try {
return SensorMessage.parseFrom(payload);
}
catch (InvalidProtocolBufferException e) {
LogFactory.getLog(Receiver.class).error("transform", e);
return null;
}
}
private void handleMessage(Message<?> msg) {
SensorMessage sensorMessage = (SensorMessage) msg.getPayload();
System.out.println(sensorMessage);
}
}
The program creates a simple Spring Integration flow that starts with the UDP inbound adapter listening on port 9992 for incoming packets. Each incoming packet contains binary data and flows into a transformer that deserializes the data with the static parseFrom() method into a SensorMessage instance. After the transformation, the message flows to a handler that prints the object to System.out.
You can find the complete project on GitHub.