Adventures in Streaming to Http Requests in NodeJS

  1. You must call req.end() before you can expect any response from the server — if you don’t call req.end() your code will just hang while it waits.
  2. _flush(callback) is called on the transform when its finished consuming the pipe and when you call the callback() thats when whatever transforms downstream get closed, so you need to connect the callback into the request’s response logic and call it when the response’s “end” event triggers.
  3. Its a bit confusing but transform.write() adds data to the head of the pipe, transform.push() adds data to the tail — I think the response object is technically a ReadStream, but I couldn’t figure out any way to pipe the response to the WriteStream of the transform — So instead of doing that I just handle the ‘data’ and the ‘end’ events of the ClientResponse.
  4. An instance of a Transform is a single use consumable thing. So if you try to reuse the instance you will get an infinite loop. That’s why the call to new Transform() needs to be wrapped in a generator function.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store