Airflow variables are global and static, while XCom is a dynamic way to share data between tasks.
You need to tell Airflow to use the new backend class.
Your specific (AWS, GCP, Azure, or On-Premise) airflow xcom exclusive
The modern TaskFlow API simplifies data passing. When you return a value from a decorated @task , Airflow creates an implicit connection. : You don't manually call xcom_pull .
export AIRFLOW__CORE__XCOM_BACKEND="include.custom_xcom_backend.S3XComBackend" Use code with caution. Airflow variables are global and static, while XCom
: Many operators (and all functions decorated with @task in the TaskFlow API) automatically push their return value to a key called return_value .
+--------------------+ Implicit/Explicit Return +----------------------+ | | -----------------------------------> | | | Upstream Task A | | Airflow Metadata DB | | | <----------------------------------- | (xcom table) | +--------------------+ .xcom_pull(task_ids) +----------------------+ | | .xcom_pull() v +----------------------+ | | | Downstream Task B | | | +----------------------+ Implicit vs. Explicit XComs When you return a value from a decorated
: Use the run_id filter to isolate data to the current DAG run. 3. Custom XCom Backends
You can manually push data at any point within your task execution context using context['ti'].xcom_push(key='my_key', value=data) . The Downstream Pull