on_load method is not working for plugins located in folder

[maiorBoltach]

Apache Airflow version: 1.10.12

Environment:

• Cloud provider or hardware configuration: Google Cloud, custom-8-16384
• OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)
• Kernel (e.g. uname -a): 3.10.0-1062.18.1.el7.x86_64 Improving the search functionality in the graph view #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

What happened:

Class method on_load of AirflowPlugin class doesn't run after plugins located in plugin folder, only for plugins installed through pip.

What you expected to happen:

Method 'on_load' should run on plugins initialization, that are located in plugin folder

How to reproduce it:

1. Create plugin with next code and place it in plugins folder:

class TestPlugin(AirflowPlugin):
    menu_links = [MenuLink(category='Google', name='Google', url='google.com')]

    @classmethod
    def on_load(cls, *args, **kwargs):
        raise AirflowPluginException("Test On Load Exception")

2. Reload webserver.

Result: Webserver logs are empty, new menu appeared in UI.
Expected result: Webserver logs should contain information with raised exception, new menu shouldn't appear in UI.
Reason: on_load method is called only for airflow.plugins.* entry_points

airflow/airflow/plugins_manager.py

Line 152 in ce66bc9

plugin_instance.on_load()

Additional information: It is not clear what is the reason for skipping this method call for plugins contained in the folder. At the same time, plugin validation successfully run for plugins from a folder and installed via a package. Validation uses validate method in same AirflowPlugin class and can be extended by user.

airflow/airflow/plugins_manager.py

Line 96 in b9dc3c5

def validate(cls):

[maiorBoltach]

If this is really a bug, can I contribute?

[mik-laj]

@maiorBoltach Yes. We are open to contributions from everyone. I assigned you to this ticket.

Contributor guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst

[mik-laj]

It' a real bug. See: airflow.plugins_manager.load_plugins_from_plugin_directory This does not call the on_load method.

[maiorBoltach]

@mik-laj should I also make PR for 1-10 version or we leave it only in 2.0?

[mik-laj]

Only for master. we are not making non-critical changes to 1.10