![]() The contents of these files are shown below. The configuration for the sequential executor local environment consists of a Dockerfile and a docker-compose.yml file. However, when you are just getting started, the sequential executor is likely sufficient. Even for development use it can become a bottleneck because it runs tasks sequentially, one at a time. The simplest executor is the sequential executor, which is not recommended for production usage. The three development environments I created, which can be found in my data-analytics-prototypes repository, utilize sequential, local, and celery executors. The major difference between these environments comes down to the executor, which is the component of Airflow that runs scheduled tasks. I’ve created multiple Airflow development environments of varying degrees of complexity. Since Airflow often has a complex setup with multiple containers, I use Docker Compose to orchestrate them. No Airflow dependencies are needed on your host machine with this approach. With Docker, you have an Airflow environment that works across different operating systems and is started with a single command. While it is possible to run Airflow on the host machine of your development environment, a more elegant approach is to use Docker. In the image below, the last two runs of the hello_world DAG are shown, both of which were successful.Īirflow provides many ways to view DAGs and environment configurations, but the pages shown above are the ones I’ve found most useful these past six months. This page shows the results of all the prior DAG runs. Hovering over a task supplies more information about it, and clicking on the task provides options such as viewing the logs or re-running the task.Ĭlicking on the "Log" button displays the logs for the task run, which is very useful for debugging.Īnother useful page is the DAGs tree view. ![]() In this case, both tasks ran successfully, as denoted by both tasks being outlined in green. The graph view shows the DAG and the result of the previous run. Clicking on a DAG shows the following view:Īirflow DAGs have multiple views the view shown above is called the graph view. It also gives options to toggle DAGs on and off (the switch to the left of the DAG name) and run DAGs (the play button in the "Actions" column). This list of DAGs displays basic information about each DAG, such as their execution schedules, and the results of recent runs. After signing into the Airflow UI, the initial page displays all the DAGs in the Airflow environment. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |