Rust, MongoDB & Wither 0.6
Hello everyone! I would like to share an update on an open source project which I have been developing for a little while now. It is a Rust project called Wither which attempts to “provide a simple, sane & predictable interface into MongoDB, based on data models”. This post is about the 0.6 release of this crate, and I would like to dive into some of the aspects of developing this release which I really enjoyed or which I found interesting.
Some new Rust things I was able to touch on include usage of compiletest-rs (pretty awesome stuff), writing a custom derive proc-macro, parsing Rust AST with the syn crate, and a few other small topics. If you are not a 🦀 rustacean 🦀, or you are new to Rust, I will do my best to keep everything simple and approachable.
First, I have recently started a new company called DocQL which provides services for the GraphQL ecosystem, and we use MongoDB quite a lot for standard data storage purposes. As our entire stack is written in Rust, we do our database IO from our Rust microservices. The Wither project has helped to reduce the amount of code we need to write for database interfacing immensely! Wither is still a humble and fairly new project, and my first commit was just a little over a year ago at the time of this writing.
Wither has come a long way since then, and I am happy to say that it seems to have found a balance between doing too much for you (risking obfuscation), and not doing enough for you (risking not having any value). When reading and writing data, you want that process to be simple, and straightforward. Having to serialize/deserialize constantly, handle deeply nested results, each of which could return errors … it detracts from the intent of your business logic. Instead, being able to use a data model with serialization and error handling all wrapped up in a clean interface leads to business logic with clear intent and fewer abstractions. Here is a basic example of how you create a Wither data model using the 0.6 release.
Imports and usage are left out of the above example, but as you can see, it is pretty straightforward to get things going. Even the indexes are defined as Rust field attributes. From there, all you would have to do is call User::find, User::find_one , User::update or any of the other provided methods which Wither exposes. Clean, concise, one location to handle DB errors, and all you have to do is #[derive(Model)].
proc-macro — custom derive — syn
The major objective that I wanted to accomplish with the 0.6 release was to remove the need to manually implement the Model trait on your data structs. Though that was already pretty straightforward, being able to take a declarative approach is WAY better. It is less code that you have to maintain. So, I opened an issue for it, have been hacking on it intermittently for a few months, and just wrapped it up a few hours ago 🙌
Rust’s metaprogramming system is … so awesome. You take in AST, do things with it, and then return some new AST which the compiler will then use. You literally write the output AST as only very slightly modified standard Rust code.
As you can see, the quote! macro can be used to write plain old Rust code. Interpolations are managed via the #var pattern. This is how you generate the output AST and interpolate values accumulated during your processing.
One major thing I would like to highlight is what the AST parsing/processing typically looks like. It took a bit to get used to this, because I kept thinking that there must have been a higher-level API, a more abstract API, which would have made this a bit more terse. I was sure that there was something I was overlooking. In serde’s code base, which is significantly more mature, it seems to be about the same thing. At the end of the day, I didn’t find anything else, and I was pretty much left to iterate over lots of syn::Meta elements, accumulate matched patterns, extract needed values, rinse and repeat.
This is fine, once you get the hang of it. syn seems to be a pretty well structured crate. Here is some code taken directly from the Wither codebase showing how I parsed out various struct-level #[model(...)] attributes.
When writing a proc-macro, the compiler expects you to return a TokenStream , not a Result . So if you encounter a situation where the user has declared an attribute which you do not recognize, or a value which is illegal, then you simply panic! with a helpful error message. You could just ignore it, but that tends to lead to confusion for the end-user. IMHO, best to be explicit about these things.
compiletest-rs
compiletest-rs is an excellent project. In essence it allows you to write rust programs which should fail compilation, and then allows you to assert against the failure output. This is powerful for ensuring that your proc-macro handles unexpected situations gracefully and clearly. Here is a screenshot showing how I organized the compile-fail tests in the Wither project.
I ended up coming up with an ad hoc naming convention for the files. As this project grows, we will definitely have to develop and stick to a proper nomenclature. For now, it got the job done. Each of the files is dead simple, and is treated as a standalone program. Here is an example of one of the compile-fail tests.
As you can see, there is nothing too crazy going on here. The main thing to notice would be the comments at the bottom of the file. The compiletest-rs system uses these comment patterns to develop the assertions which you would like to make against the output of the compilation failure. Check out the project for more details. If you are developing proc-macros (custom derive, attribute-like or function-like) then I highly recommend that you use compiletest-rs. It will help take your project to the next level.
feature(external_doc)
The last item I would like to mention is a feature which I have been eager to experiment with for quite some time now, the external_doc feature. This is still an unstable feature (as of 2018.11.14), and has been in the works for a decent bit of time. I am happy to report that it works really well, and I was able to get the documentation for this project organized very nicely because of it. It allows you to declare that some external markdown document should be pulled in and used as part of the crate’s documentation, as such: #[doc(include="some-file.md")] .
As the Wither project targets the stable compiler, there was a tiny hoop to jump through to get a nightly feature working in a stable crate. The main idea was to use conditional compilation via cargo’s features system. I introduced a feature called docinclude , and then used the cfg_attr system to look for this feature at compilation time to activate the unstable external_doc compiler feature.
The local doc tests & doc build commands for the crate simply need to be invoked with the +nightly option to cargo: cargo +nightly ... . And as it turns out, the docs.rs system automatically builds your docs using the nightly compiler. Only thing that needed to be done there was to include some metadata in the Cargo.toml to ensure docs.rs would use the docinclude feature when compiling this crate, which you can see in the above screenshot. The [package.metadata.docs.rs] section does the trick.
conclusion
All in all, I’m pretty stoked about the future of this project. I’m hoping to see async IO support (instead of just threading support) in the underlying MongoDB project. Things are growing and expanding so rapidly in the Rust community these days, that I would not be surprised to see this happen very soon. I am definitely hoping to have the time to contribute to this effort as well.
I mentioned DocQL earlier. We are already using the 0.6 release of the Wither project and we will be continuing to develop this and other related crates to continue to maximize performance and stability. Definitely check us out if you are using GraphQL at all. GraphQL & MongoDB seem to work quite well together. It is a combination that I would definitely recommend.
Thanks for the read! Hopefully you found the knowledge share to be beneficial. Feel free to hit me up on twitter @AnthonyJDodd.
