Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(hogql): bytecode, local evaluation and action matching in plugin server #16189

Closed
wants to merge 60 commits into from

Conversation

mariusandra
Copy link
Collaborator

@mariusandra mariusandra commented Jun 22, 2023

Problem

Actions with HogQL can't trigger webhooks because HogQL can't be used for local evaluation.

Changes

Well now it is.

This PR :

  • introduces a over-the-wire HogQL "bytecode" format, which is easy to execute and parse on the client side, including in the plugin server. This can be used for feature flags in the future.
  • implements code to convert a HogQL AST into this bytecode format
  • implements two VMs to execute said bytecode. One in Python, one in TypeScript
  • adds a new bytecode field on the actions table, which contains a condensed HogQL bytecode for all steps in the action
  • adds local evaluation into the plugin server, when matching actions that already have a populated bytecode field
  • adds =~ and !~ regex matching shorthand support into HogQL

End result: Slack Webhooks on HogQL

2023-06-22 15 55 09

See the readme for the bytecode format

(Technically this is not 8-bit "byte"-code, but just an array of instructions... I just didn't need to serialize it as a binary string yet, but it should serialize very nicely if we ever need to 🦐)

Questions

  • Should this be enabled for everyone, or just for us for testing? I did add a try block around the bytecode execution.

  • I added '_h' as the first element of the instructions to specify what this is. Shall I add another version instruction, or will we go _h -> _ho -> _hog when incrementing versions? :D

  • The only thing that is not working right now is a case when you have an action that simultaneously matches a cohort and a HogQL string. Both work if used independently, so there is no regression. The solution is to implement a cohort_filters_to_hogql function, at least for all cases that can work with local evaluation. I'd love to do this in another PR. This has gotten long enough.

  • There are no indications on the frontend if any action/filter combo works with webhooks or not. I'd like to implement the cohort features first, and then make this clear.

How did you test this code?

Wrote tests. Recorded a screencast.

TODO

  • Use int enums
  • Move both implementations into a shared folder
  • Support ~ regex search
  • Unified regexp handling, or document differences (documented that it's not supposed to work)
  • Describe effects with null in the spec
  • Test element mapping with

@mariusandra mariusandra changed the title feat(hogql): bytecode mode feat(hogql): inline hogql action matcher in plugin server Jun 22, 2023
@mariusandra
Copy link
Collaborator Author

Closing and continuing in #16304

@mariusandra mariusandra deleted the hogql-vm branch June 29, 2023 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants