Add isCompanionFunction to function metadata

[pramodsatya]

No description provided.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	0dccd1c
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/66d099be4298e10008d40f35

[aditi-pandit]

@pramodsatya : How do we take care that we get only Prestissimo functions from the registry ? If there is a conflict between a Prestissimo and Spark function with the same name, then how do we disambiguate ?

[pramodsatya]

Thanks for the feedback @aditi-pandit, @czentgr. We are not checking that only Prestissimo functions are retrieved from the registry. Instead we rely on the Prestissimo worker to have registered only the presto functions, such as in this function, so there are no Spark functions in the registry. Please let me know if this is fine or whether we should have an additional way to distinguish between Presto and Spark functions in the registry.

[pramodsatya]

Hi @aditi-pandit, @czentgr, could you please take another look at this PR?

[pramodsatya]

Thanks for the feedback @aditi-pandit, addressed the comments. Could you please take another look?

[aditi-pandit]

Thanks @pramodsatya

[aditi-pandit]

Thanks @pramodsatya. This is looking very good to close.

[pramodsatya]

Thanks @aditi-pandit, addressed the comments, could you please take another look?

[czentgr]

Moved it back to CoverageUtil, but now velox_function_registry would have a dependency on velox_coverage_util, and this results in a circular dependency between them. It doesn't appear to be causing any issues but I would prefer to avoid this.

Yes, you don't want a circular dependency. Building the Velox library statically will work but building it as a dynamic library is then broken due to the circular dependency.

[aditi-pandit]

@pramodsatya : The PR is looking good minus the issues with the isCompanionFunctionName function.

[pramodsatya]

Thanks @aditi-pandit, addressed the comments. Could you please take another look?

[aditi-pandit]

Thanks @pramodsatya. Changes look good.

[pramodsatya]

Thanks for the feedback @aditi-pandit. @czentgr suggested that because this API modification is centered around detecting and removing companion functions, it would be better to include information about whether a function is a companion function in it's metadata. Since the concept of companion functions is specific to velox and not prestissimo, it would be better to get this information from the velox function metadata and detect companion functions from the function metadata in prestissimo (and handle them as per the requirement in prestissimo). This also helps avoid a dependency on velox_exec in velox_function_registry, and detect companion functions in a cleaner manner instead of gathering this information by finding specific substrings in the registered function name.

I have updated the PR accordingly to include companion function information in the function metadata (in VectorFunctionMetadata for vector functions, in AggregateFunctionMetadata for aggregates, and in WindowFunction::Metadata for window functions). Could you please help review the updated changes and share if this approach is fine, @aditi-pandit @mbasmanova ?

[czentgr]

Thanks, looks good now.

[czentgr]

Maybe rename to registerMergeExtractFunctionInternal.

[aditi-pandit]

@pramodsatya. @czentgr : This is a reasonable solution as well, if we are okay exposing companion function concept to the services using Velox.

With the other API to return all functions with a boolean parameter to say 'skipInternalFunctions' a single function could be used to skip all companion or any other internal functions added by the Velox framework.

@mbasmanova : wdyt ?