Open-Sourcing Sftp Throughput Improvements in Alpakka
The number of open source libraries I have used is impossible to count and there’s hardly a month goes by without finding a new way to use one or a new one to use. There are so many widely used and hugely useful libraries in the Java ecosystem that we often take them for granted but there is a community of people who dedicate their free time (or in many cases, all their time) to enhancing and fixing these libraries so that we all benefit. Contributing back to open source can become complicated when you work as a Software Engineer due to issues around intellectual property or possible unintended exposure of credentials or other sensitive data. While researching solutions to a problem at work I found a bottleneck in Alpakka’s SFTP connector and took the opportunity to open-source a solution to the problem. This was my first time contributing back to the open source community that I have gained so much from and I really enjoyed the experience. In this post I’ll go into the technical details of the issue at hand, how I improved Alpakka to get around the issue and how easy the process of contributing my changes back to the Alpakka project was.
Identifying a problem
I was looking at ways we could use streaming when fetching files via SFTP. At Workday we run many thousands of these jobs each day and the general pattern…
