Running Airflow Tasks In Parallel - Nothing Gets Scheduled
I just went through the process of configuring my Airflow setup to be capable of parallel processing by following this article and using this article. Everything seems to be workin
Solution 1:
I'm not sure which of these steps exactly solved my problem and I'm not sure exactly what the root cause of the problem was but I did this:
I literally just reset everything. First I shut down the webserver
and scheduler
using kill theirPIDs
or ctrl + c
if it's open still in the terminal. Then I deleted all the entries under /home/ec2-user/airflow/dags/__pycache__
. Then I restarted the postgre database using sudo /etc/init.d/postgresql restart
then I ran airflow resetdb
. Then I reran airflow webserver
and airflow scheduler
. I went in the UI and turned on the DAG and voila it went into the running state and then worked successfully. No idea what was going on though.....
Post a Comment for "Running Airflow Tasks In Parallel - Nothing Gets Scheduled"