Use tower gRPC for TiKV

In TiKV, we use gRPC for network communication, and we also have developed a Rust gRPC library — grpc-rs, which wraps the Google C gRPC with Rust Futures, and provides an ergonomic API for outside use.

But there are some problems for grpc-rs:

  • grpc-rs is not pure Rust
  • Its performance is not very good. It is slower than our customized protocol.
  • Sadly, we even met some panic in gRPC in production environments before.

So sometimes, we want to embrace the Rust community, and use a gRPC library fully written in Rust.

Of course, we don’t want to reinvent the wheels. Luckily, there are two existing libraries — grpc-rust and tower-grpc. Before we developed grpc-rs, we tried grpc-rust for some time, but this library was not stable before so we gave up and never use it again. Maybe it has already become better :-)

So now we pay attention to tower-grpc. Although tower-grpc is new and not stable too, it still has some benefits:

  • The authors of tower-grpc are famous in the Rust world, like carllerche who develops mio, and seanmonstar who develops hyper.
  • It is based on tokio, and we think tokio is the future of the network programming in Rust.

For us, the earlier embracing tower-grpc, the closer we cooperate with Rust community, and of course, we can improve tower-grpc and tokio at the same time.

But now, there is a problem for TiKV to try tower-grpc. TiKV uses rust-protobuf to encode/decode protobuf, but tower-grpc uses prost. Changing all associated codes is not an easy work, and needs lots of time. But we can still try tower-grpc simply.

At first, we need to clone kvproto — the definition of TiKV gRPC protocol, and create a tower directory:

git clone https://github.com/pingcap/kvproto.git
cd kvproto
mkdir tower

Refer to tower-grpc example Cargo.toml, we can create a similar Cargo.toml.

Notice: I met build error when built tower-grpc with Rust version rustc 1.29.0-nightly (4f3c7a472 2018-07-17) , if you meet the same problem, you need to clone the tower-grpc, and add following lines to tower-grpc/src/lib.rs , then use the local tower-grpc in your example.

#![feature(extern_prelude)]
#![feature(crate_in_paths)]

Then we create build.rs:

extern crate tower_grpc_build;
fn main() {
tower_grpc_build::Config::new()
.enable_server(true)
.enable_client(true)
.build(&["../proto/tikvpb.proto"], &["../proto", "../include"])
.unwrap_or_else(|e| panic!("protobuf compilation failed: {}", e));
// Same for pdpb.proto and other proto files
}

At last, we create an empty file src/lib.rs and run cargo build, which generates the Rust files of the gRPC protocol in the out directory like target/debug/build/tower-9000a2ba77585c7e/out, we need to copy them to other directory manually for later use (of course, we can do this automatically in build.rs later).

The generated file, e.g. metapb.rs looks like below:

#[derive(Clone, PartialEq, Message)]
pub struct Cluster {
#[prost(uint64, tag="1")]
pub id: u64,
/// max peer count for a region.
/// pd will do the auto-balance if region peer count mismatches.
///
/// more attributes......
#[prost(uint32, tag="2")]
pub max_peer_count: u32,
}

You can’t import this file and use it directly, you need to include it, like:

pub mod metapb {
include!("proto/metapb.rs");
}

Now, we can create another project, refer to client.rs, we create a TiKV client, which will send raw_put command to TiKV:

pub fn main() {
let _ = ::env_logger::init();
    let uri: http::Uri = format!("http://127.0.0.1:20161").parse().unwrap();
    let h2_settings = Default::default();
let mut make_client = client::Connect::new(Dst, h2_settings, DefaultExecutor::current());
    let say_hello = make_client.make_service(())
.map(move |conn| {
use tikvpb::client::Tikv;
use tower_http::add_origin;
            let conn = add_origin::Builder::new()
.uri(uri)
.build(conn)
.unwrap();
            Tikv::new(conn)
})
.and_then(|mut client| {
use kvrpcpb::{Context, RawPutRequest};
use metapb::{RegionEpoch, Peer};
            let ctx = Context{
region_id: 2,
region_epoch: Some(RegionEpoch{
conf_ver: 1,
version: 1,
}),
peer: Some(Peer{
id: 3,
store_id: 1,
is_learner: false,
}),
term: 6,
priority: 0,
isolation_level: 0,
not_fill_cache: false,
sync_log: false,
handle_time: false,
scan_detail: false,
};
            client.raw_put(Request::new(RawPutRequest {
context: Some(ctx),
key: b"a".to_vec(),
value: b"123".to_vec(),
cf : "default".to_string(),
})).map_err(|e| panic!("gRPC request failed; err={:?}", e))
})
.and_then(|response| {
println!("RESPONSE = {:?}", response);
Ok(())
})
.map_err(|e| {
println!("ERR = {:?}", e);
});
    tokio::run(say_hello);
}

We forcibly send command to Region 2, you can get the region information through tools pd-ctl and tikv-ctl.

After we run the client, the output is:

RESPONSE = Response { http: Response { status: 200, version: HTTP/2.0, headers: {"content-type": "application/grpc", "grpc-accept-encoding": "identity,deflate,gzip", "accept-encoding": "identity,gzip"}, body: RawPutResponse { region_error: None, error: "" } } }

We use tikv-ctl to query the data:

tikv-ctl --host 127.0.0.1:20161 print -k za
value: 123

Every key written to TiKV will be added z as the prefix, so here we use za, and we can see the value is “123”. Great, we have inserted a key-value successfully into TiKV.

As you can see, using tower-grpc is easy, and we have been considering migrating to tower-grpc https://github.com/tikv/tikv/issues/3951. If you are interested in this, feel free to contact us.