-
Notifications
You must be signed in to change notification settings - Fork 343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support simd_json as an optional alternative to serde_json #934
base: main
Are you sure you want to change the base?
Support simd_json as an optional alternative to serde_json #934
Commits on Oct 18, 2024
-
Support simd_json as an optional alternative to serde_json
The approach is to abstract _all_ original serde_json references to a new lib crate in the workspace - json-impl. That crate as a feature swicth between serde_json and simd_json. In The initial phase, is to to get all the other crates to depend on json-impl with the serde feature, instead of serde_json directly, so in the end nothing should change. The next phase is to activate the simd feature and see what fails, then start mitigating the failures.
Martin Bartlett committedOct 18, 2024 Configuration menu - View commit details
-
Copy full SHA for f522ea8 - Browse repository at this point
Copy the full SHA f522ea8View commit details
Commits on Oct 19, 2024
-
Start hacking stuff into mutable slices
The _problem_ with simd_json is that it requires _mutable_ slices to operate while serde_json does not. This FIRST PASS (certainly not the end solution) is to provide &mut adapter of the from_slice function for serde_json, a pseudo-safe adapter of from_str for simd_json, and then change all the places where stuff is flagged as needing to be mutable to be so - MOSTLY, however, this is in TESTS - so eventually we'll clean that up.
Martin Bartlett committedOct 19, 2024 Configuration menu - View commit details
-
Copy full SHA for 0807675 - Browse repository at this point
Copy the full SHA 0807675View commit details -
Hack tests to use mutable strings/slices
Now we see the REAL problems: * No "std::cmp::Eq" available on simd::OwnedValue (at least as it is now) * SOME use of &str passed to a deser visitor and then fed back into simd_json::from_str ... which needs a mut reference (and is technically unsafe)
Martin Bartlett committedOct 19, 2024 Configuration menu - View commit details
-
Copy full SHA for cba3fdf - Browse repository at this point
Copy the full SHA cba3fdfView commit details
Commits on Oct 22, 2024
-
So, now that simd-json Values can be Eq ...
... all we have a couple of missing functions and a mutability difference/error (in the base crates - haven't gotten onto the main client interface crates yet.
Martin Bartlett committedOct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 6125836 - Browse repository at this point
Copy the full SHA 6125836View commit details -
Fix "missing method" issues (expose and use the right simd_json trait)
Martin Bartlett committedOct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for f20d03c - Browse repository at this point
Copy the full SHA f20d03cView commit details -
Adaptation to simd json parsing
Martin Bartlett committedOct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 3c696e4 - Browse repository at this point
Copy the full SHA 3c696e4View commit details -
Amazingly, we appear to have resolved all compile issues outside of l…
…ambda_http Of course, lambda_http IS the target for all this and the thing that will expose the mut and unsafe aspects of using simd to the world - that's for another day!
Martin Bartlett committedOct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for f95180b - Browse repository at this point
Copy the full SHA f95180bView commit details -
Martin Bartlett committed
Oct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 12cb88f - Browse repository at this point
Copy the full SHA 12cb88fView commit details -
Revert to live simd_json crate - supports Eq values now
Martin Bartlett committedOct 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 6f8ba9e - Browse repository at this point
Copy the full SHA 6f8ba9eView commit details
Commits on Oct 23, 2024
-
Complete replacement of serde_json reference by aws_lambda_json_impl …
…references And add a very useful "from_bytes" combinator to aid in parsing from stuff like HTTP bodies (though from_reader is probably cheaper).
Martin Bartlett committedOct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 79c9d6e - Browse repository at this point
Copy the full SHA 79c9d6eView commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 0940f12 - Browse repository at this point
Copy the full SHA 0940f12View commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 5ec7dba - Browse repository at this point
Copy the full SHA 5ec7dbaView commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 8344c12 - Browse repository at this point
Copy the full SHA 8344c12View commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for f2b146e - Browse repository at this point
Copy the full SHA f2b146eView commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for d24f7bd - Browse repository at this point
Copy the full SHA d24f7bdView commit details -
Martin Bartlett committed
Oct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 137a210 - Browse repository at this point
Copy the full SHA 137a210View commit details -
A better approach to intended use
Plus the from_vec using an owned vec gets us around some awkward lifetime issues when trying to write "generic" code for both simd_json and serde_json
Martin Bartlett committedOct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for f38e250 - Browse repository at this point
Copy the full SHA f38e250View commit details -
One error left in this round - missing trait on
simd_json deserializer
Martin Bartlett committedOct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 3c1d23e - Browse repository at this point
Copy the full SHA 3c1d23eView commit details -
Fix need for 'from_reader' on deserializer
THAT is NOT easy to create for simd_json.
Martin Bartlett committedOct 23, 2024 Configuration menu - View commit details
-
Copy full SHA for 334246e - Browse repository at this point
Copy the full SHA 334246eView commit details
Commits on Oct 24, 2024
-
We're down to the core of the issue
We've removed Deserialize from of lambda_http::request::LambdaRequest ... that makes EVERYTHING compile in the simd_json context ... with the exception of th calls to lambda_runtime::run which require the expected request type to be serde::Deserialize. So now it's back to that issue - which, if I recall well, was the blocker back when I first started this!
Martin Bartlett committedOct 24, 2024 Configuration menu - View commit details
-
Copy full SHA for e0317b2 - Browse repository at this point
Copy the full SHA e0317b2View commit details -
All compile errors "fixed" ... but SERIOUSLY unsafe at the moment
At the moment the only way around the deserializer-must-be-simd issue is an unsafe "cast" ... and even the type equivalence check that I tried before that cast will itself not compile, so that is currently commented out.
Martin Bartlett committedOct 24, 2024 Configuration menu - View commit details
-
Copy full SHA for fbe4802 - Browse repository at this point
Copy the full SHA fbe4802View commit details
Commits on Oct 25, 2024
-
Explain the unsafe code and the constraints that make it 'safe'
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 1893e6a - Browse repository at this point
Copy the full SHA 1893e6aView commit details -
Fixes for when using serde_json
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 2c26efd - Browse repository at this point
Copy the full SHA 2c26efdView commit details -
Clippy lints for simd_json code
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 4c0a2ee - Browse repository at this point
Copy the full SHA 4c0a2eeView commit details -
Martin Bartlett committed
Oct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for bd6a3d8 - Browse repository at this point
Copy the full SHA bd6a3d8View commit details -
MARKER: All original tests work in serde_json mode
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 0548317 - Browse repository at this point
Copy the full SHA 0548317View commit details -
Doc and change simd_json provenance
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 72e3636 - Browse repository at this point
Copy the full SHA 72e3636View commit details -
MARKER: All but one test passed in simd_json mode!
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for bcf29ef - Browse repository at this point
Copy the full SHA bcf29efView commit details -
And this mod was proof that simd_json does, indeed, mess with the source slice! Docs on switching to simd_json will need to make that very clear.
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for ebf675f - Browse repository at this point
Copy the full SHA ebf675fView commit details -
MARKER: serde_json mode works with no adverse affect on performance
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 353d14e - Browse repository at this point
Copy the full SHA 353d14eView commit details -
simd_json http request deserialization isn't working from external caller
Martin Bartlett committedOct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for bbc41da - Browse repository at this point
Copy the full SHA bbc41daView commit details -
Martin Bartlett committed
Oct 25, 2024 Configuration menu - View commit details
-
Copy full SHA for 2935c9f - Browse repository at this point
Copy the full SHA 2935c9fView commit details
Commits on Oct 26, 2024
-
Yeah, that didn't work AT ALL - literal crash!
Martin Bartlett committedOct 26, 2024 Configuration menu - View commit details
-
Copy full SHA for 4e0c417 - Browse repository at this point
Copy the full SHA 4e0c417View commit details
Commits on Oct 28, 2024
-
The unsafe cast that works (using proposed features of simd_json)
Turns out I was casting to the wrong thing
Martin Bartlett committedOct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 4ab9c3c - Browse repository at this point
Copy the full SHA 4ab9c3cView commit details -
MARKER: Tests work for HTTP request deserialization via simd_json
Martin Bartlett committedOct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 23e5950 - Browse repository at this point
Copy the full SHA 23e5950View commit details -
Configuration menu - View commit details
-
Copy full SHA for 08dd1b0 - Browse repository at this point
Copy the full SHA 08dd1b0View commit details -
MARKER: Local integration tests work
Martin Bartlett committedOct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 4950699 - Browse repository at this point
Copy the full SHA 4950699View commit details -
Martin Bartlett committed
Oct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 614b8a3 - Browse repository at this point
Copy the full SHA 614b8a3View commit details -
Martin Bartlett committed
Oct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 7b2f0bb - Browse repository at this point
Copy the full SHA 7b2f0bbView commit details -
Comment out restart - it seems it's not required
Martin Bartlett committedOct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 1f3f984 - Browse repository at this point
Copy the full SHA 1f3f984View commit details -
Tests without restart work - not net needed in this implementation
Martin Bartlett committedOct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 05c0a0b - Browse repository at this point
Copy the full SHA 05c0a0bView commit details -
Martin Bartlett committed
Oct 28, 2024 Configuration menu - View commit details
-
Copy full SHA for a939b8f - Browse repository at this point
Copy the full SHA a939b8fView commit details
Commits on Oct 29, 2024
-
Simplify the cast and adjust the doc
Martin Bartlett committedOct 29, 2024 Configuration menu - View commit details
-
Copy full SHA for 825e343 - Browse repository at this point
Copy the full SHA 825e343View commit details -
Martin Bartlett committed
Oct 29, 2024 Configuration menu - View commit details
-
Copy full SHA for a853fb9 - Browse repository at this point
Copy the full SHA a853fb9View commit details