Cannot Run Airflow DAGs/Tasks as Different User

[tahaislam]

Running/backfilling DAGs as any other user rather than Airflow always fails with PermissionError (tested on Airflow 2.5.1 on Morbius)

[2023-07-13 11:14:32,216] {local_executor.py:130} ERROR - Failed to execute task [Errno 1] Operation not permitted: '/etc/airflow/logs/dag_id=unit_validation_Lyft/run_id=backfill__2020-07-01T10:00:00-04:00/task_id=P1.BasicValidation.raw_data_detection'.
Traceback (most recent call last):
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/executors/local_executor.py", line 126, in _execute_work_in_fork
    args.func(args)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/cli.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/cli/commands/task_command.py", line 384, in task_run
    ti.init_run_context(raw=args.raw)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/models/taskinstance.py", line 2414, in init_run_context
    self._set_context(self)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
    set_context(self.log, context)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
    flag = cast(FileTaskHandler, handler).set_context(value)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/log/file_task_handler.py", line 71, in set_context
    local_loc = self._init_file(ti)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/log/file_task_handler.py", line 382, in _init_file
    self._prepare_log_folder(Path(full_path).parent)
  File "/home/airflow/airflow_venv/lib64/python3.9/site-packages/airflow/utils/log/file_task_handler.py", line 358, in _prepare_log_folder
    directory.chmod(mode)
  File "/usr/lib64/python3.9/pathlib.py", line 1339, in chmod
    self._accessor.chmod(self, mode)
PermissionError: [Errno 1] Operation not permitted: '/etc/airflow/logs/dag_id=unit_validation_Lyft/run_id=backfill__2020-07-01T10:00:00-04:00/task_id=P1.BasicValidation.raw_data_detection'

[tahaislam]

Airflow log folder needs to have 777 permissions

After chmod -R 777 /etc/airflow/logs/, still not working. Looks like the file mode comparison in Airflow isn’t perfect (the extra 40 is the directory mode as extracted from the inode 😢)

>> import os
>> directory='/etc/airflow/logs/dag_id=unit_validation_Lyft/run_id=backfill__2020-07-01T10:00:00-04:00/task_id=Collision.BasicValidation.raw_data_detection'
>>> oct(os.lstat(directory).st_mode)
'0o40777'
>>> oct(mode)
'0o777'

[tahaislam]

setting AIRFLOW__CORE__DEFAULT_IMPERSONATION isn't working. Also setting run_as_user on the DAG/task level not working most probably they require sudo access for impersonation
https://airflow.apache.org/docs/apache-airflow/stable/security/workload.html

[tahaislam]

Airflow log folder needs to have 777 permissions

After chmod -R 777 /etc/airflow/logs/, still not working. Looks like the file mode comparison in Airflow isn’t perfect (the extra 40 is the directory mode as extracted from the inode 😢)

>> import os
>> directory='/etc/airflow/logs/dag_id=unit_validation_Lyft/run_id=backfill__2020-07-01T10:00:00-04:00/task_id=Collision.BasicValidation.raw_data_detection'
>>> oct(os.lstat(directory).st_mode)
'0o40777'
>>> oct(mode)
'0o777'

This issue might be resolved in Airflow version >= 2.6.0
apache/airflow#29112

[tahaislam]

A temporary solution (for testing DAGs) might be to save logs somewhere else by setting AIRFLOW__LOGGING__BASE_LOG_FOLDER. For example like:

env AIRFLOW__LOGGING__BASE_LOG_FOLDER='~/logs' airflow dags backfill dag_id