airflow config with OSX

Target are listed below:

• install airflow
• setup database with mysql & install provider with mysql

---

install airflow on OSX

Follow the link to see the details of how to install airflow with constraints file. https://airflow.apache.org/docs/apache-airflow/stable/start.html

AIRFLOW_VERSION=2.6.2

# Extract the version of Python you have installed. If you're currently using Python 3.11 you may want to set this manually as noted above, Python 3.11 is not yet supported.
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example this would install 2.6.2 with python 3.7: https://raw.githubusercontent.com/apache/airflow/constraints-2.6.2/constraints-3.7.txt

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Please consider the command which will install 2.6.2 on python 3.10

pip install "apache-airflow==2.6.2" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.2/constraints-3.10.txt"

---

Run Airflow standalone:

airflow standalone

Run Airflow with seperate service:

airflow db init

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

airflow webserver --port 8080

airflow scheduler

---

Change the airflow working database to mysql:

Install mysql connector:

pip install mysql-connector-python

Create airflow database on mysql:

CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; 

Edit the connection string in airflow.cfg:

sql_alchemy_conn = mysql+mysqlconnector://<user>:<password>@<host>[:<port>]/<dbname>

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-mysql-database

---

install apache-airflow-providers-mysql on OSX

simply use PyPI to install the apache-airflow-providers-mysql on MAC OSX may not succeed due to the mysql client package missing.

pip install apache-airflow-providers-mysql

https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html#apache-airflow-providers-mysql

https://pypi.org/project/apache-airflow-providers-mysql/

Check on PyPI site, ‘apache-airflow-providers-mysql’ should have 3 depended packages:

PIP package	Version required
apache-airflow	>=2.4.0
apache-airflow-providers-common-sql	>=1.3.1
mysqlclient	>=1.3.6

While ‘mysqlclient’ on OSX may need a customized installation. On OSX, we should install the server or client for mysql with brew, before install the mysqlclient with PyPI tool.

nstall MySQL and mysqlclient:

# Assume you are activating Python 3 venv
$ brew install mysql pkg-config
$ pip install mysqlclient

If you don’t want to install MySQL server, you can use mysql-client instead:

# Assume you are activating Python 3 venv
$ brew install mysql-client pkg-config
$ export PKG_CONFIG_PATH="/opt/homebrew/opt/mysql-client/lib/pkgconfig"
$ pip install mysqlclient

https://pypi.org/project/mysqlclient/

Then install the ‘apache-airflow-providers-mysql’ with PyPI will pass.