Looks really impressive. I’m an ETL developer who has build a multitude of data integration solutions with both WYSIWYG tools (Pentaho Kettle, Clover ETL, SAS Enterprise Guide) and programming languages (Python, Ruby, Base SAS). Despite loving the ability to see data flow diagrams, I’ve been burned by the WYSIWYG tools, because I inevitably come across a problem that the devs didn’t think to incorporate into a widget and the workaround are usually nightmares. With full programing languages, we can build virtually anything and gain all of the benefits of modern software development (modularization, unit testing, code reviews, etc.), but we lose those nice data flow diagrams. I’ve been struggling with how to reconcile the two for years and this project looks exceptionally promising (looks like in Luna, the code drives the GUI, rather the other way around as in other tools).
I was browsing the website and couldn’t come up with answers to a few questions and was hoping you could help shed some light on them:
- Does Luna have any sort of unit and feature testing mechanism for the individual functions?
- Can Luna handle individual record-level file-parsing errors (e.g., expecting a field to be a date, but it comes in as a string) gracefully, without having the whole integration fail?
- How can Luna handle data volumes that can’t fit into memory?
- Can Luna talk to other data processing frameworks, like Apache Spark or Pandas?
- Does Luna support data streaming or can it hook into something like RabbitMQ or Apache Kafka?
- What kind of database connectors are supported?