First look at Cap'n Proto in Rust

A computer scientist working in open source towards a more hopeful future.

Posted on 2015-02-11, around 7 minutes of reading.

So, I found this (this, and this) Cap'n Proto library for Rust and wanted to explore it a bit. Unfortunately the documentation is sparse for the library, and I haven't played with Cap'n Proto before.

Cap'n Proto is what's billed as a 'Data Interchange Format' and 'Capability-Based RPC System'. Or a 'Cerealization Protocol'.

The concept for working with Cap'n Proto is:

  1. Build schemas out of it's schema language which define the data you're working with.
  2. Compile them into code structures for your preferred language using using capnpc.
  3. Code your application (optimally, using an RPC-style system, I won't be exploring that yet.)

There is a great talk here.

Prerequisites

Rust

Follow the Guide. You'll want the nightly provided by rustup.sh ideally.

Cap'n Proto

First you should probably grab the capnp tool. If you're on a Mac with brew you can do this with:

brew install capnp

Then grab capnpc-rust like so:

git clone https://github.com/dwrensha/capnpc-rust && \
cd capnpc-rust && \
cargo build

Then move target/capnpc-rust to your local ~/bin folder, or somewhere else on your $PATH.

Diving In

Note, Rust is in flux at the time of writing and the information here may not necessarily be up to date.

As a first step, we'll set up a simple project to pack up an example person and write the coresponding Cap'n Proto data to stdout which we can explore with capnp.

Initializing the Project

Use cargo to create a new package.

cargo new capntest --bin
cd capntest

Add the capnproto-rust to your Cargo.toml

echo "[dependencies]\ncapnp = \"*\"\ncapnpc = \"*\"" >> Cargo.toml

Building a Schema

First, we need to generate a unique ID for Cap'n Proto to identify our schema.

capnp id

You'll see these at the top of every .capnp schema file.

Now, Cap'n Proto has a pretty interesting schema language that seems adequate for most purposes. I'd be interested in see what it can (and cannot) represent.

Let's save the example on the aforementioned page as src/schema/person.capnp.

@0xdbb9ad1f14bf0b36;  # unique file ID, generated by `capnp id`

struct Person {
  name @0 :Text;
  birthdate @3 :Date;

  email @1 :Text;
  phones @2 :List(PhoneNumber);

  struct PhoneNumber {
    number @0 :Text;
    type @1 :Type;

    enum Type {
      mobile @0;
      home @1;
      work @2;
    }
  }
}

struct Date {
  year @0 :Int16;
  month @1 :UInt8;
  day @2 :UInt8;
}

This format probably feels familiar to you. Types come after names just like in Rust. However you may have noticed the @0, @1, ... notations. This is so Cap'n Proto knows how the data is laid out and can evolve over time. If, for example, we start storing a Person's spouse as well, we would add spouse @4 :Person;.

While designing your schemas, it's probably best to try to keep the @n notations monotonically increasing, that is, keep them in order.

Now we can build the schema into some rust code with:

capnp compile -o rust src/schema/*.capnp

You can check src/schema/person_capnp.rs to see the generated code. It's not super pretty, but there is a lot going on.

Importing the Schema

Currently, due to this and this you need to use your generated schema in the top level module.

This is the minimal main.rs file which should give you access to what we need from person.

extern crate capnp;
extern crate capnpc;

mod person_capnp {
    include!("./schema/person_capnp.rs");
}

use person_capnp::person;

fn main() {

}

Double check that this compiles for you use cargo build before going further.

Aside: Docs

Currently, there is little documentation about how to use the library, only a pair of examples. Examples here and here.

If you'd like to actually have something to reference, git clone https://github.com/dwrensha/capnproto-rust and run cargo doc in the directory. Then visit something like file:///Users/hoverbear/capnproto-rust/target/doc/capnp/index.html in your browser. Don't expect this to help much, but it might give you some hints.

You can also check the Cap'n Proto C++ docs here and here, as far as I can tell the interface is rather similar.

If you feel like being awesome, help write some docs!

Using the Schema

Keep in mind, before we begin, that Cap'n Proto is a bit different than normal Rust stuff.

Cap'n Proto structs are build via the use of functions in the Builder implementation generated.

If you look in the ./schema/people_capnp.rs file you'll see:

impl <'a> Builder<'a> {
	// ...
    // Setters
    // ...
}

That's how you can create your structs. Here's how we create a person and dump it to stdout:

#![feature(path)]
extern crate capnp;
extern crate capnpc;
mod person_capnp {
    include!("./schema/person_capnp.rs");
}
use person_capnp::person as Person;
use capnp::serialize_packed;
use capnp::{MessageBuilder, MallocMessageBuilder};
use std::old_io::stdout;

fn main() {
    // Allocate.
    let mut message = MallocMessageBuilder::new_default();
    {
        let mut person = message.init_root::<Person::Builder>();
        // Setters
        person.set_name("Foo Bar Baz");
        person.set_email("foo@bar.baz");
        {
            // Use a Scope to limit lifetime of the borrow.
            let mut birthdate = person.borrow().init_birthdate();
            birthdate.set_day(1 as u8);
            birthdate.set_month(1 as u8);
            birthdate.set_year(1 as i16);
        }
        {
            // Use a Scope to limit lifetime of the borrow.
            let mut phones = person.borrow().init_phones(2); // 2 phones
            // Unforuntately these borrows must be there.
            phones.borrow().get(0).set_number("(123) 123-1234");
            phones.borrow().get(0).set_type(Person::phone_number::Type::Home);
            phones.borrow().get(1).set_number("(123) 123-1235");
            phones.borrow().get(1).set_type(Person::phone_number::Type::Mobile);
        }
    }
    let mut out = WriteOutputStream::new(stdout());
    serialize_packed::write_packed_message_unbuffered(&mut out(), &mut message);
}

Okay, so, this seems fairly straightforward. The enum variants seem to work like in Rust, but there is some interesting allocation stuff going on. There are also a lot of scopes happening to handle borrow lifetimes.

So what's happening? The way Cap'n Proto stores it's structured laid out a very specific way in memory such that it doesn't need to serialize and deserialize data. When you're building a Cap'n Proto structure you're actually working with this memory.

Running cargo build we can run ./target/capntest and see the following:

➜  capntest git:(master) ✗ ./target/capntest
Qb'�Foo zar Baz�z�(123) 12?3-1234�(123) 12?3-1235%

Inspecting the Packed Data

What we got out of our program is what's called "packed" data. You can learn more about it here. We can use the capnp tool to unpack a struct.

➜  capntest git:(master) ✗ ./target/capntest | capnp decode --packed ./src/schema/person.capnp Person
( name = "Foo Bar Baz",
  email = "foo@bar.baz",
  phones = [
    ( number = "(123) 123-1234",
      type = home ),
    ( number = "(123) 123-1235",
      type = mobile ) ],
  birthdate = (year = 1, month = 1, day = 1) )

Closing

That should be enough to perk your interest in Cap'n Proto and get you started with the library. I'm hoping to write another article soon about working with Cap'n Proto input data (hint, change set to get!) and RPC calls.

If you can't wait to dig in, our worked example closely matches Address Book example given in the repository... Check the print_address_book function out here.

Discussion of this post is on Reddit.

            231666f96329f6c37d7736c5c0be5d5634043289