Schema DDL: add support for ZSTD encoding

[miike]

Should ZSTD be implemented and considered as a compression type particularly for VARCHAR columns?

[chuwy]

Thanks @miike! That's question for @bogaert, I think.

[bogaert]

@miike have you run any tests to see how this compares to LZO (our current default)?

[miike]

@bogaert I'm getting pretty good compressions ratios all round for varchar columns but ZSTD works on all types. Is there any sort of standardised datasets we can use for Snowplow that would make sense so we can make like for like comparisons?

[alexanderdean]

Is there any sort of standardised datasets we can use for Snowplow that would make sense so we can make like for like comparisons?

We have a retail dataset that we could potentially use - @bogaert knows more...

[bogaert]

I have created an internal card to see what we can do. I'll keep you posted @miike.

[miike]

Thanks @bogaert!

[bernardosrulzon]

@miike @alexanderdean @bogaert I'm seeing very good results with ZSTD. The org.w3/performance_timing table was reduced to 30% its original size after encoding eveything as ZSTD. It should be safe to use as default as it works on all data types.

[miike]

Bumping this - we've just compressed a moderately sized (~3TB) atomic.events dataset down to 30% of it's size using ZSTD that's resulted in some speed improvements. I think it's worth further testing and including in Iglu as a default over LZO (analyze compression will now recommend ZSTD over LZO apart from some instances in which columns contain a large number of nulls).