Creative Xchange and Apache Airflow
Creative Xchange, is a startup solving a few of the most challenging problems in the music industry today. Creative Xchange is developing a platform creating opportunities for artists to earn sustainable income.
Being a small seed stage startup, with limited resources, it is important for us to spend our time wisely and select the right components that bring value to our platform and make it maintainable and sustainable.
In this blog post, we will cover why we chose to go with airflow, and how we leverage it for our platform.
Why Airflow
We were looking to select an application or framework that would help us automate some of our backend processes, such as our reporting and CRM automation. We ended up choosing airflow for four main reasons:
Configuration as Code:
One of our wishes was to have configuration as code, we wanted to be able to track the changes being made to our workflows, be able to code review them and rollback the changes easily. Airflow caters to that need through its’ DAG configuration files, essentially a piece of code as configuration.
An added benefit to having the entire workflow being setup as code means , that there is less need for cross-functional work between a CRM platform expert and an engineer doing data integration. Our engineers can do end to end work, and not be hindered by meetings our internal dependencies.
Batteries included:
Python’s source distribution philosophy is to be “Batteries included”, providing all that you need to run your applications. Airflow does that workflow management, it provides a programmatic interface to orchestrate your workflow, has deep task dependencies management, offers a monitoring and management interface, has built in retry policy, a quick way to access your task run logs and an altering systems.
Flexible:
Airflow is based on python and is able to surface PythonOperators to execute custom python code. As such it is able to basically handle any non-real time problem being thrown at it, be it a data pipeline task, a machine learning task or generating some API calls. Airflow has a series of Operators made to simplify the work, be it to source data from a storage (s3 operators), execute docker containers (DockerOperator) or to send email (EmailOperator).
Future proof:
We see Airflow as a future proof application that will be able to handle the load as needed without pre-mature optimizations. Airflow can scale horizontally by adding extra worker nodes, can scale work elastically using its’ Kubernetes (K8s) executor or even defer processing work to other systems such as AWS Athena or Spark.
How we use Airflow?
At Creative Xchange, we use airflow as our main framework and interface to our backend processes, we setup our data-flows and reporting structure within the tools as well as setup marketing automation journeys and send out emails.
Database and Reporting
We use airflow the traditional way, in order to build data-flows and data pipelines for reporting purposes.
One of the automation that we have built upon, is the automated sending of our weekly report, reporting our main business KPIs.
Marketing Automation and CRM
We send a series of mails to our platform members (Artists and Venues), so far we have developed a series of three journeys:
An on-boarding journey, where we requests from artists that they complete their profile. Profile completion is one of our growth lever and this journey ensures that we have high quality profile on our platform.
A flyer automation journey, where we first ask selected artists to provide pictures of themselves or their bands, these are then forwarded to a designer to create flyers for the artists to promote their events with Creative Xchange. Once the designer has completed the task, flyers are sent to both artists and venues for them to promote their events.
An event review journey, where we send a request to both the participating artists and venue to review the event they had together.
An invoicing email journey, we have integrated our invoicing setup with Zoho. This means, we can send invoicing reminder emails directly through the same platform as our normal journeys.
Where next?
We plan to take our automation and airflow to the next level. Our focus on the platform is to increase the overall level of engagement of our users and increasing the overall features of our platform
We plan to do this through:
- An increase flow of notification, be it for reminders for both artists and venues, notifications on users of user activity (followers, comments etc..)
- Embedding, some of our business processes deeper in to airflow, such as when an artists declines a booking request by a venue. We want to be able to serve venues, with emails informing them and containing personalized recommendation of suitable alternative Artists available during that booking slot.
Wrap Up
Our experience leveraging Airflow has been great, we have been able to consolidate our backend business process using one single tool, that provides alerting and monitoring built in.
We have been able to setup marketing automation journeys without the need of paying $10k+ annual licenses fee to the likes of Salesforce or other SAAS providers to get started, and without the need for extensive integration.
We have been able to successfully leverage airflows’s automation and workflow into our core business processes and make them an integral part of the CX experience.
To know about events happening near you please visit us.
Follow us on Instagram to see events and know about our musicians.
To start booking musicians for your venue or your events signup on the platform.
