10/16/2023 0 Comments Airflow python callable with argumentsreturn 'materialmarm', 'materialmbew', 'materialmdma' If you want to learn more about the BranchPythonOperator, check my post, I. For example, you want to execute materialmarm, materialmbew and materialmdma, you just need to return those task ids in your python callable function. Trigger DAG2 using TriggerDagRunOperator.ĭAG 2 - Create tasks depending on the Airflow Variable updated in DAG 1. import json import csv import os import pandas as pd from airflow import DAG from import PythonOperator from . The BranchPythonOperaror can return a list of task ids. Say, if Synapse has 3, then I need to create 3 tasks.ĭAG 1 - Access Azure synapse and get Variable. Based on retrieved variable, I need to create tasks dynamically. I need to access Azure Synapse and get a variable (Say 3). Python_callable is depreciated in TriggerDagRunOperator - Airflow 2.0. I am getting the below error : : Invalid arguments were passed to TriggerDagRunOperator (task_id: test_trigger_dagrun). Ti.xcom_push(key='res', value=result_dictionary) My_var = t("num_runs", len(result_dictionary)) from airflow import DAG from import DummyOperator from import PythonOperator from time import sleep from datetime import datetime def myfunc (opargs): print (opargs) return opargs 0 with DAG ('pythondag. Perfom some operations based on asqldb_kv and populate the result_dictionary list This is how you can pass arguments for a Python operator in Airflow. Since you use the task decorator on task1(), what PythonVirtualenvOperator gets instead is an Airflow operator (and not the function task1()). My parent_dag code is : from datetime import datetimeįrom _operator import PythonOperatorįrom _dagrun import TriggerDagRunOperator PythonVirtualenvOperator expects a function to be executed as an argument to its pythoncallable parameter. In am trying to call DAG from another DAG( target_dag from parent_dag).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |