Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support simd_json as an optional alternative to serde_json #934

Draft
wants to merge 44 commits into
base: main
Choose a base branch
from

Commits on Oct 18, 2024

  1. Support simd_json as an optional alternative to serde_json

    The approach is to abstract _all_  original serde_json references to a new lib crate
    in the workspace - json-impl.
    
    That crate as a feature swicth between serde_json and simd_json. In
    
    The initial phase, is to to get all the other crates to depend on json-impl with the serde feature,
    instead of serde_json directly, so in the end nothing should change.
    
    The next phase is to activate the simd feature and see what fails, then start mitigating
    the failures.
    Martin Bartlett committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    f522ea8 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2024

  1. Start hacking stuff into mutable slices

    The _problem_ with simd_json is that it requires _mutable_  slices to operate while serde_json does not.
    
    This FIRST PASS (certainly not the end solution) is to provide &mut adapter of the from_slice function for serde_json,
    a pseudo-safe adapter of from_str for simd_json, and then change all the places where stuff is flagged as needing to
    be mutable to be so - MOSTLY, however, this is in TESTS -  so eventually we'll clean that up.
    Martin Bartlett committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    0807675 View commit details
    Browse the repository at this point in the history
  2. Hack tests to use mutable strings/slices

    Now we see the REAL problems:
    
    * No "std::cmp::Eq" available on simd::OwnedValue (at least as it is now)
    * SOME use of &str passed to a deser visitor and then fed back into simd_json::from_str ... which needs a mut reference (and is technically unsafe)
    Martin Bartlett committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    cba3fdf View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. So, now that simd-json Values can be Eq ...

    ... all we have a couple of missing functions and a mutability
    difference/error (in the base crates - haven't gotten onto the main
    client interface crates yet.
    Martin Bartlett committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    6125836 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f20d03c View commit details
    Browse the repository at this point in the history
  3. Adaptation to simd json parsing

    Martin Bartlett committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    3c696e4 View commit details
    Browse the repository at this point in the history
  4. Amazingly, we appear to have resolved all compile issues outside of l…

    …ambda_http
    
    Of course, lambda_http IS the target for all this and the thing that will expose
    the mut and unsafe aspects of using simd to the world - that's for another day!
    Martin Bartlett committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    f95180b View commit details
    Browse the repository at this point in the history
  5. Explanations

    Martin Bartlett committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    12cb88f View commit details
    Browse the repository at this point in the history
  6. Revert to live simd_json crate - supports Eq values now

    Martin Bartlett committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    6f8ba9e View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Complete replacement of serde_json reference by aws_lambda_json_impl …

    …references
    
    And add a very useful "from_bytes" combinator to aid in parsing from
    stuff like HTTP bodies (though from_reader is probably cheaper).
    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    79c9d6e View commit details
    Browse the repository at this point in the history
  2. cleanup

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    0940f12 View commit details
    Browse the repository at this point in the history
  3. fix dependencies

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    5ec7dba View commit details
    Browse the repository at this point in the history
  4. Fix simd-related imports

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    8344c12 View commit details
    Browse the repository at this point in the history
  5. Don't rename from_slice

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    f2b146e View commit details
    Browse the repository at this point in the history
  6. Generalize tests

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    d24f7bd View commit details
    Browse the repository at this point in the history
  7. Don't rename from_slice

    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    137a210 View commit details
    Browse the repository at this point in the history
  8. A better approach to intended use

    Plus the from_vec using an owned vec gets us around
    some awkward lifetime issues when trying to write "generic" code for both simd_json and serde_json
    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    f38e250 View commit details
    Browse the repository at this point in the history
  9. One error left in this round - missing trait on

    simd_json deserializer
    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    3c1d23e View commit details
    Browse the repository at this point in the history
  10. Fix need for 'from_reader' on deserializer

    THAT is NOT easy to create for simd_json.
    Martin Bartlett committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    334246e View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. We're down to the core of the issue

    We've removed Deserialize from of lambda_http::request::LambdaRequest ...
    that makes EVERYTHING compile in the simd_json context ... with the exception
    of th calls to lambda_runtime::run which require the expected request type to be
    serde::Deserialize.
    
    So now it's back to that issue - which, if I recall well, was the blocker back when I
    first started this!
    Martin Bartlett committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    e0317b2 View commit details
    Browse the repository at this point in the history
  2. All compile errors "fixed" ... but SERIOUSLY unsafe at the moment

    At the moment the only way around the deserializer-must-be-simd issue is
    an unsafe "cast" ... and even the type equivalence check that I tried before that
    cast will itself not compile, so that is currently commented out.
    Martin Bartlett committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    fbe4802 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Configuration menu
    Copy the full SHA
    1893e6a View commit details
    Browse the repository at this point in the history
  2. Fixes for when using serde_json

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    2c26efd View commit details
    Browse the repository at this point in the history
  3. Clippy lints for simd_json code

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    4c0a2ee View commit details
    Browse the repository at this point in the history
  4. Test fixes and clippy lints

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    bd6a3d8 View commit details
    Browse the repository at this point in the history
  5. MARKER: All original tests work in serde_json mode

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    0548317 View commit details
    Browse the repository at this point in the history
  6. Doc and change simd_json provenance

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    72e3636 View commit details
    Browse the repository at this point in the history
  7. MARKER: All but one test passed in simd_json mode!

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    bcf29ef View commit details
    Browse the repository at this point in the history
  8. All tests work...

    And this mod was proof that simd_json does, indeed, mess with the source slice! Docs on switching to simd_json will
    need to make that very clear.
    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    ebf675f View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    353d14e View commit details
    Browse the repository at this point in the history
  10. Add debugging

    simd_json http request deserialization isn't working from external caller
    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    bbc41da View commit details
    Browse the repository at this point in the history
  11. More debug

    Martin Bartlett committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    2935c9f View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. Yeah, that didn't work AT ALL - literal crash!

    Martin Bartlett committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    4e0c417 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. The unsafe cast that works (using proposed features of simd_json)

    Turns out I was casting to the wrong thing
    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    4ab9c3c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    23e5950 View commit details
    Browse the repository at this point in the history
  3. Remove explicit simd_json source

    Using local patch instead
    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    08dd1b0 View commit details
    Browse the repository at this point in the history
  4. MARKER: Local integration tests work

    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    4950699 View commit details
    Browse the repository at this point in the history
  5. fix doc tests

    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    614b8a3 View commit details
    Browse the repository at this point in the history
  6. A bit more debugging

    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    7b2f0bb View commit details
    Browse the repository at this point in the history
  7. Comment out restart - it seems it's not required

    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    1f3f984 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    05c0a0b View commit details
    Browse the repository at this point in the history
  9. Use simd-json main

    Martin Bartlett committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    a939b8f View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Simplify the cast and adjust the doc

    Martin Bartlett committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    825e343 View commit details
    Browse the repository at this point in the history
  2. Support feature pass_through

    Martin Bartlett committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    a853fb9 View commit details
    Browse the repository at this point in the history