How to trigger Airflow DAG using REST API

admin July 23, 2022

This post will discuss triggering dag using rest API in Airflow version 2. Airflow released Airflow API (Stable) (1.0.0) to access its object or trigger dag utilizing a range of REST API endpoints. In this post, we will directly discuss how to trigger dag using rest API we are not explaining how to set up and run airflow. If you want to set up airflow first then refer here.

Step 1: Enable the Rest API in Airflow

You need to edit the airflow config file as by default airflow does not accept any request through REST API. Browse airflow.cfg file and set auth_backends as

# auth_backends = airflow.api.auth.backend.session

auth_backends = airflow.api.auth.backend.basic_auth
 

You can also restart the webserver if this is not working and You can also check the REST API, You simply need to pass the Authorization in the form of username:password where the username of the airflow user and password is the password of that user.

I have set these parameters in the Postman API app like this every different platform or framework provides a different method to set Authorization.

Step 2: Enable CORS

Write these lines below the [API] section in airflow.cfg and also replace https://exampleclientapp1.com and https://exampleclientapp2.com with your website name or localhost. You can also refer here to know more about CORS.

access_control_allow_headers = origin, content-type, accept
access_control_allow_methods = POST, GET, OPTIONS, DELETE
access_control_allow_origins = https://exampleclientapp1.com https://exampleclientapp2.com

Step 3: Test the API by Listing all your dags available

We can list all our dags by hitting GET request on /api/v1/dags endpoint. If you are running airflow on your localhost you can get all your dags by hitting GET request to http://localhost:8080/api/v1/dags also don't forget to set the Authorization.

Step 4: Setup your dag file

Here is my sample dag file code put this code in a file that is in the dags folder if there is no file then create one name sample_dag.py.

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
def main(**kwargs):
print("Hello World!")
 
args = {
'owner': 'deep',
'depends_on_past': False,
'start_date': days_ago(2),
'email': ['sample@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'schedule_interval': None,
'provide_context': True
}
dag = DAG(
dag_id='my-dag',
default_args=args,
tags=["newdag"]
)
t1 = PythonOperator(task_id = "my_function", 
python_callable = main,
dag = dag
)
t1

As you already know this code now we quickly see how to trigger this dag using REST API.

Note: Dag run id can be anything you can pass any text or string in dag_run_id.

Step 5: Access conf in your code or function

We can also access the variables which we have set in the conf passed in our request. You just need to write provide_context=True in your PythonOperator function and access it using kwargs['dag_run'].conf.get("here is your variable name").

Updated code:

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
def main(**kwargs):
#line changed
variable = kwargs['dag_run'].conf.get('variable') 
print(variable)
 
args = {
'owner': 'deep',
'depends_on_past': False,
'start_date': days_ago(2),
'email': ['sample@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'schedule_interval': None,
'provide_context': True
}
dag = DAG(
dag_id='my-dag',
default_args=args,
tags=["newdag"]
)
t1 = PythonOperator(task_id = "my_function", 
python_callable = main,
provide_context=True,
dag = dag
)
t1

I have shown you how you can trigger your dag from the rest API and I hope this will help you to write your code and setup up rest API.

python data_engineering apache_airflow setup