chore: Omit bytewax deps and materialization engine

[sudohainguyen]

What this PR does / why we need it:

After the successful merge of PR #4087, this PR aims to remove Bytewax entirely, including its dependencies and materialization engine. This is crucial for unblocking efforts to support Python 3.11 as outlined in #4046.

Additional notes

Next action is to remove related docs and compose the new k8s engine doc

[HaoXuAI]

LGTM!

[breno-costa]

@sudohainguyen why are you removing bytewax? Is there any issue or RFC explaining reasons behind this decision?

[sudohainguyen]

@sudohainguyen why are you removing bytewax? Is there any issue or RFC explaining reasons behind this decision?

There are 2 main reasons:

• the current version of bytewax we're using is blocking us from supporting python 3.11
• we're not leveraging all features of bytewax: if you look at bytewax engine implementation, it is simply a kubernetes job with multiple of parallel pods reading files separately and writing data to the online store, which is powered by pyarrow zero-copy mechanism, so we decided to re-write the engine called k8s engine which remains the logic but without bytewax interface.

as a result, we see it is safe to remove bytewax 😄 hope this helps

[tokoko]

@breno-costa see #4087 for a replacement that Harry pointed to. The discussions around the topic started out here #3993 and then continued on slack and community calls, so unfortunately no single place to see it all.

[franciscojavierarceo]

@tokoko any chance you can compile a quick doc with the decision just so we can add it as a note somewhere? Even a paragraph in the community notes would probably be sufficient.

[sudohainguyen]

@franciscojavierarceo let me handle it

[franciscojavierarceo]

@sudohainguyen thank you!

[breno-costa]

@sudohainguyen @tokoko Thanks for clarifying that point. We wanted to try a more scalable materialization engine and bytewax would be our first candidate. Let us try this new k8s implementation.