Xcom: Airflow

@dag(schedule_interval='@daily', start_date=datetime(2023, 1, 1), catchup=False) def my_taskflow_dag():

: Standard database records do not auto-delete. Set up a periodic maintenance DAG that runs a clean-up query against the xcom database table to purge entries older than your history retention policy (e.g., 30 days). If you are currently designing a data pipeline, tell me: airflow xcom

If you need to pass large objects (e.g., Pandas DataFrames), you can implement a backend that automatically saves to S3. | Feature | Recommendation | | :--- |

| Feature | Recommendation | | :--- | :--- | | | Keep < 48KB. Use pointers (filepaths) for large data. | | Python Code | Prefer TaskFlow API ( @task ) over ti.xcom_pull . | | SQL/Templating | Use ti.xcom_pull(...) Jinja syntax. | | Security | Never push passwords/tokens. Use Connections. | | Storage | Default is Metadata DB. Use Custom Backend for scaling. | | | SQL/Templating | Use ti

# Pull the return value (default key) ret_val = ti.xcom_pull(task_ids='push_task')

While both store data inside the metadata database, their operational scope and architectural use cases differ completely. Feature Metric Airflow XComs Airflow Variables Local to a specific Task Instance run. Global across the entire Airflow deployment. Lifecycle Automatically created and removed with DAG runs. Persistent until explicitly updated or deleted. Primary Use Inter-task dependencies and pipeline state.