[AIRFLOW-5274] dag loading duration metric name too long

[feng-tao]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

• Here are some details about my PR, including screenshots of any UI changes:
  Make dag loading duration metric name not too long when the dag file has multiple dags.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[feng-tao]

cc @milton0825

[marengaz]

do we need to replace slashes, or some other special chars?

[ashb]

We should add a note to UPDATING.md about the changed metric name

[ashb]

I think for consistency we should always use the filename - otherwise any monitoring dashboards of load time will be "mixed"

[kaxil]

I agree

[feng-tao]

make sense, will do.

[feng-tao]

@kaxil @ashb @milton0825 PR updated.

[milton0825]

Question: Is file_stat.file just the file name or it is the file path that contains /?

[feng-tao]

Good point, I double check and the file name is /subdag/dagname.py something. Update the pr.

[marengaz]

it's possible to have the same file name in different folders. please can we join the split, instead of just taking the filename? eg.

filename = '.'.join(file_stat.file.split('/')).replace('.py', '')

[feng-tao]

I think it would be a bad practice to have the same file name for different DAGs. If you think this would be an issue on your side, feel free to raise a pr.

[feng-tao]

Besides if we join all the subdir, it will make those stats unreadable if the dir for the dag file is very deep. Personally I am not favor of this. Not sure how other committer thinks.

[marengaz]

i was imagining a file structure like the for example. it doesn't seem too unreasonable a structure?

dag_dir
   +->dag_module_1
   |       +->dag_definition.py
   |       |->callbacks.py
   |       |->functions.py
   |       |->config.json
   +->dag_module_2
           +->dag_definition.py
           |->callbacks.py
           |->functions.py
           |->config.json

[marengaz]

but yea, we dont use that dir structure, so im happy with the merge :)