Docker: Airflow 실행

실습

실행 과정

1. airflow-setup Github repo clone

git clone https://github.com/learndataeng/learn-airflow.git

 

2. airflow-setup/yml 다운로드

cd learn-airflow
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.5.1/docker-compose.yaml"

 

3. docker-compose.yaml

• version
• x-airflow-common
•  services
  • postgres
  • redis
  • airflow-webserver
  • airflow-scheduler
  • airflow-worker
  • airflow-trigger
  • airflow-init

• volumes
  • postgres-db-volume

docker-compose -f docker-compose.yaml pull
docker-compose -f docker-compose.yaml up

# docker-compose.yaml 수정된 부분

x-airflow-common:
 &airflow-common
 image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.1}

 enviroment:
 &airflow-connection-env
 AIRFLOW_VAR_DATA_DIR: /opt/airflow/data
 AIRFLOW_CONN_TEST_ID: test_connection
 _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS: -yfinance pandas numpy oauth2client gspread}

 volumes:
  - ${AIRFLOW_PROJ_DIR: -.}/data:/opt/airflow/data
 
 airflow-init:
  mkdir -p /sources/logs /sources/dags /sources/plugins /sources/data
  chown -R "${AIRFLOW_UID}:0" /sources/{logs, dags, plugins, data}

docker-compose up -d	# Detached 모드 실행

 

4. 웹 UI 로그인 : http://localhost:8080 및 airflow:airflow 사용

docker exec -it learn-airflow-airflow-scheduler-1 airflow variables get DATA_DIR /opt/airflow/data