capnp-conv: proc macro ergonomics for capnproto-rust

[realcr]

This PR adds a new crate called capnp-conv, allowing for automatic proc macro based code generation for serialization and deserialization.
Should solve issue #136

[dwrensha]

Thanks for the pull request!

I'm not going to have time to look at this immediately, partially because I'm focusing on sorting out our alignment story, but I hope to have a look before too long.

[zimond]

@dwrensha is this happening anytime soon?

[dwrensha]

It looks like this only works with serialize_packed. Is there a way to generalize this so it would also work for the non-packed case?

[dwrensha]

It seems to work for me if I do assert_eq!(float_struct.my_float32, float_struct2.my_float32).

[realcr]

Comparing floats is what makes me wake up in fear in the middle of the night. I am still new to understanding how it works in Rust, if you feel comfortable with this comparison I'm cool with it too.

[dwrensha]

I gather that this is for testing a struct that generic on the Rust side. How does this library handle structs that are generic (i.e. have type parameters) on the capnp side? Like this one:

capnproto-rust/capnpc/test/test.capnp

Lines 738 to 744 in 4393eb5

	struct Map(Key, Value) {
	entries @0 :List(Entry);
	struct Entry {
	key @0 :Key;
	value @1 :Value;
	}
	}

[realcr]

capnp-conv doesn't handle the case of generic structs on the capnp side. I admit I am not really sure what would be the expected behaviour. This will require some thinking.

[dwrensha]

Can this library handle structs that have some union fields and some non-union fields? E.g.

struct Foo {
   a @0 :UInt64;
   b @1 :Text;
   union {
      c @2 :Data;
      d @3 :UInt32;
  }
}

[realcr]

I am almost sure capnp-conv doesn't support this. What would be the Rust equivalent of this construct?

[realcr]

@dwrensha : Thanks for the review!

1. I am not familiar with the packed vs non-packed case. Where can I get more information about it?
2. In case it was not obvious, you should feel free to make any changes to this piece of code (:

[kscz]

1. I am not familiar with the packed vs non-packed case. Where can I get more information about it?

Hi @realcr - I know I wasn't who you asked, but since it appears no one is answering your question, I figured I would give it a shot!

One of the big things which makes Cap'n Proto unique compared to many other serialization formats which came before it, is that the in-memory representation is identical to the on-the-wire representation. This means that "serializing" can, for all intents and purposes, simply be a memory-copy into the output buffer! The downside is that this format means that details like padding bytes and unset optional fields end up being transmitted over the wire, which wastes bandwidth.

To mitigate this bandwidth issue, instead of having a "free" serialization step, you can use the "Packed" representation which basically tries to compress away as many zeros as possible.

For reading about either of these representations - https://capnproto.org/index.html talks about the "free" encoding step, and https://capnproto.org/encoding.html#packing talks about the "packed" representation.

[opeik]

Hey there, could I be of any help on this issue? I'd appreciate some macro magic to reduce boilerplate.

[ajosecueto]

any news?

[lu-zero]

what is blocking this?

[praveenperera]

Any updates?

[p--b]

For those wanting to use this, NB that as an alternative, @aikalant has published https://github.com/aikalant/capnp_conv, which is available on crates.io as capnp_conv.

[rupert648]

wanted to bump this again, this would be great to have in the core crate. Currently living with and maintaining a 800+ line serialisation file which is mostly similar boilerplate. Tests are also necessary to ensure that I don't miss out some serialisation call on a field. This would solve both those problems