Adventures in Streaming to Http Requests in NodeJS
After two days of searching I found that the internet lacked simple instructions for streaming a readable stream into an http requests and then piping the response into a writeable stream.
Just the working code:
- You must call req.end() before you can expect any response from the server — if you don’t call req.end() your code will just hang while it waits.
- _flush(callback) is called on the transform when its finished consuming the pipe and when you call the callback() thats when whatever transforms downstream get closed, so you need to connect the callback into the request’s response logic and call it when the response’s “end” event triggers.
- Its a bit confusing but transform.write() adds data to the head of the pipe, transform.push() adds data to the tail — I think the response object is technically a ReadStream, but I couldn’t figure out any way to pipe the response to the WriteStream of the transform — So instead of doing that I just handle the ‘data’ and the ‘end’ events of the ClientResponse.
- An instance of a Transform is a single use consumable thing. So if you try to reuse the instance you will get an infinite loop. That’s why the call to new Transform() needs to be wrapped in a generator function.
It’s okay internet. Nobodies perfect. And hey — after five years as a software developer I finally did a thing.