Add transform support for LightGBM by open source FreeForm2 library

[chjinche]

Add transform support for LightGBM by open sourcing FreeForm2 library.

In this PR, we introduce a built-in way for LightGBM to support feature transformation, which is more convenient for development and iteration. Compared to separate data transformation in advance, this way could put transformation and model in one place, so keep offline and online consistency.

FreeForm2 is a flexible type of transforms, created by Microsoft Core Ranking and used widely over Microsoft production model training. As the name indicates, FreeForm2 empowers users to compose a free combination of features as they like. It is expressed by formulas to be applied in the model inputs. The surface syntax is s-expression, with parentheses in a LISP-like fashion to delimit. FreeForm2 has implicit type systems and evaluate a single, nested expression that returns a floating-point number. Please see more info about FreeForm in doc PR #4735.

[jameslamb]

Thank you for your interest in LightGBM!

But we will not review this PR and the others you recently created in their current state. Between this PR, #4734 and #4735, you are asking maintainers of this project to review and support 70,000+ new lines of code, without providing an explanation for the proposed benefit.

Please open an issue at https://github.com/microsoft/LightGBM/issues thoroughly describing what FreeForm2 is and why you think LightGBM would benefit from integrating it.

I'm closing this PR, #4734, and #4735 for now. Let's have all discussion in the issue you create.

[shiyu1994]

Dear @jameslamb and @StrikerRUS,

These PRs integrate a feature transformation module (which is known as FreeForm2, and is a module used by some products within our company) into LightGBM. FreeForm2 automatically transforms original features into new ones according to the expressions specified by users. The purpose of this integration is to enable LightGBM to be used in some of our products. I think we can have detailed discussion for any concerns.

Actually, we've worked with @chjinche during the past two months about this plan. And we have been pushing forward the process to make FreeForm2 open source into LightGBM during last month. I'm sorry that we did not inform the community about this plan earlier for discussion.

In our current plan, though FreeForm2 will be added as a part of source code into LightGBM, its source code will work independently with the main part of LightGBM source code. And only #4734 requires some modification of our current code base.

I think open source of FreeForm2 into LightGBM is a great contribution overall. And let's discuss how to make a better integration.

Thanks for your support for LightGBM all the time. Your hard work definitely keeps LightGBM a great software. Sorry if these large PRs disturb you.

[shiyu1994]

A separate issue will be open to discuss this.

[jameslamb]

I'm sorry that we did not inform the community about this plan earlier for discussion.

Sorry if these large PRs disturb you.

Thanks very much for opening #4742.

I do understand that there will be meaningful discussions about this project that happen inside Microsoft which I and the the community of non-Microsoft-employee contributors and collaborators cannot be a part of. Totally fine!

But I hope that in the future such conversations can make it out into the open much earlier in the process, and definitely before large pull requests are opened here.

Some large things that surprised me in the last few months:

• WIP: [CUDA] New CUDA version #4528 being opened in August, with a commit history suggesting you'd been working on it since April.
  • Until that PR was opened I did not know there were significant efforts happening to write a new CUDA implementation, and if someone from NVIDIA (for example) had asked if we'd be open to an outside contribution of such a change I would have told them "yes absolutely!"
• you and @guolinke saying, as if it had already been decided, that LightGBM would drop support for OpenCL-based GPU builds ([CUDA] New CUDA version Part 1 #4630 (comment), bin size 257 cannot run on GPU #4082 (comment)).
• this current proposal about FreeForm2

Sorry for the long post, and thank you for listening. I'm very grateful to you and the other maintainers here for trusting and supporting me as I try to contribute to this project where I can.

[shiyu1994]

Hi @jameslamb. Thanks for your suggestion. We'll make such discussions open earlier next time.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.