feat!: Use HashStreamValidator for Data Validation
#1951
Conversation
- Removes `fast-crc32c` support
product-auto-label
bot
added
size: l
danielbankhead
changed the title
HashStreamValidator for Data Validation HashStreamValidator for Data Validation
| @@ -874,6 +876,9 @@ class File extends ServiceObject<File> { | |||
| pathPrefix: '/acl', | |||
| }); | |||
|
|
|||
| this.crc32cGenerator = | |||
| options.crc32cGenerator || this.bucket.crc32cGenerator; | |||
There was a problem hiding this comment.
Is there a reason to look to the bucket if one is not passed as an option? Should there be a default within the File class?
There was a problem hiding this comment.
It appears all Files must have a bucket (looking at the construction) and all buckets have a generator (whether custom or the default) - this way customers can set things at a storage, bucket, or per-object level
| // http://stackoverflow.com/questions/25096737/ | ||
| // base64-encoding-of-crc32c-long-value | ||
| failed = !validateStream.test('crc32c', metadata.crc32c.substr(4)); | ||
| failed = !validateStream.test('crc32c', metadata.crc32c); |
There was a problem hiding this comment.
Any reason you know of that this was previously removing the first 4 bytes? I think you mentioned it was only checking those bytes?
There was a problem hiding this comment.
Yea… unfortunately we weren’t checking the complete hash before - just the last byte
There was a problem hiding this comment.
Any idea why? Also any idea on the performance impact of now checking the entire range of bytes?
There was a problem hiding this comment.
| } | ||
| } | ||
|
|
||
| _flush(callback: () => void) { |
There was a problem hiding this comment.
Might be good to also just mark these functions with private.
There was a problem hiding this comment.
Wanted to, but TS doesn’t like it as the parent class has it public
* feat!: Use `HashStreamValidator` for stream validation - Removes `fast-crc32c` support * fix: remove logs * test: `crc32cGenerator` option * test: improve `compression` test reliability * chore: add temp debug logs * chore: more debug logs * fix: use static gzip content to remove test variance * chore: remove debug logs * chore: lint
Fixes #1910 🦕