Kafka producer note

code note

The Overview

Usage: send `topic` with `value` and `key` pair data(also `partition`), and send with callback or send and handle future returned.

Async I/O operation under implements.

send process summary:
- fetch metadata for target topic and partition(with cache, but remote fetch when pid undefine or pid out of cache meta)
- serialize key and value
- calculate partition if not supplied
- add to accumulator, and wake up sender to send when meet a full batch.

Choose Partition

- rr in key level
- rr in global when there are no key

Accumulator

just as its name, it will accumulate datas.
first reuse or create new `ProducerBatch`, then append data into it.

Sender Prepare Request

Before sender send data, `accumulator.ready` will make connection to target partition’s leader ready~

`accumulator.drain` will drain batched data and gather them into `map<NodeID, BatchData>`

regather request again as `map<Topic, BatchData>` and setup callback on request.

Sender Reverse Response

response from node need convert to partition and `done` to call request’s callback

Network Client

Open nonblocking socket and register `OP_CONNECT` and add it to Selector..

Not remove `OP_WRITE` and return null to solve partial write/read problem.

NetworkClient send will first enqueue into `inflight` then doRequest…
remove immediately from inflight if no need to response; or remove after receive response.

Like what you read? Give lysu a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.