Making Airflow Pods Use a Private Google Cloud SQL Connection
Bluecore’s Data Science team uses Airflow for our model workflows. In our Airflow pods, we had been, until recently, using a Cloud SQL proxy as a sidecar container. The Cloud SQL connection handles database connections. We can get information about the state of a task or XComs, for example. Google has recently allowed users to connect to Cloud SQL using a VPC. Because of this, we decided to remove the proxy and implement a private connection to be more secure and save resources. As a result, Cloud SQL connections are no longer made via the localhost, and we stopped creating a sidecar container just for Cloud SQL connections.
XComs
XComs allow “cross-communication” of information between task instances. You can either “push” or “pull” an XCom.
def get_result_one(**kwargs):
# code to calculate some result
kwargs['ti'].xcom_push(key='some_key', value={result})operator_1 = PythonOperator(
task_id='task1'
python_callable=get_result_one,
provide_context=True,
dag=dag,
)def get_result_two(task_ids…