Improve process terminating in scheduler_job

[mik-laj]

We cannot perform operation join because the process may already be terminated. To wait for the process to complete is the wait method.

This occurs when I send ctrl-c to the scheduler (running with MySQL backend)

File "/opt/airflow/airflow/utils/dag_processing.py", line 633, in _exit_gracefully
    self.terminate()
  File "/opt/airflow/airflow/utils/dag_processing.py", line 1206, in terminate
    processor.terminate()
  File "/opt/airflow/airflow/jobs/scheduler_job.py", line 222, in terminate
    self._process.join(5)
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 122, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process

---

Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[muscovitebob]

Hey folks is there a linked issue here?

[mik-laj]

we do not have. I just got the error message that is in the PR description.

[muscovitebob]

Alright. We get this error quite often in running composer-1.10.4-airflow-1.10.6 with Python 3 out of the box, just trying to assemble some context on why this occurs, thanks!

[mik-laj]

@muscovitebob It is nothing important. You can ignore it. The Cloud Composer team is aware of this issue and even this has been fixed upon their request. This will be fixed in future Cloud Composer releases. I heard on Airflow Summit that 1.10.10 is to be released soon.

[swalkowski]

I don't see this change in any of:

• https://github.com/apache/airflow/blob/1.10.10/airflow/jobs/scheduler_job.py
• https://github.com/apache/airflow/blob/1.10.11/airflow/jobs/scheduler_job.py

Is this intentional? Will it be included in 1.10.12 or a later version?

[mik-laj]

@swalkowski It seems to me that this is just an oversight. I added these changes to the milestone for the next release.