Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Awkward --> C++ with Cling. #1300

Merged
merged 21 commits into from
Feb 23, 2022

Conversation

jpivarski
Copy link
Member

@jpivarski jpivarski commented Feb 19, 2022

This is for the Awkward → RDataFrame step in ianna/awkward-to-rdf, but it won't include any RDataFrame-specific stuff (just Cling). @ianna will be able to use this, so it's a blocker for the to-RDF step, but it's not a blocker for the from-RDF step.

Things to do:

  • Extract the Lookup class and tolookup function(s) from the Numba implementation to use them here. There's nothing Numba-specific about them, but those functions are spread throughout all the Numba Content Types as classmethods. Probably the right way to do it is to make a suite of LookupTypes that is independent of both Numba and C++, then the Numba Content Types would be subclasses. It's relevant because the LookupTypes are, in part, defined by the cls.STARTS, cls.STOPS, cls.CONTENT constants that define the order of the lookup table, and we want to keep that code together as a unit.
  • Set up a test for development. This is all v2.
  • Define a superclass ak::ArrayView with start, stop, which_array, and array_ptrs (32 bytes). All of the Awkward data will be subclasses of this, but they will add no member data and only the subclasses would ever get instantiated. (Find out if this means there will be an unwanted vtable or not.)
  • Also an ak::RecordView, which doesn't derive from the above (sequences and scalars are distinct).
  • Set up the abstract ak::List<T>, from which concrete types will derive. If T is a primitive type, the concrete type will either be another ak::List<T> or a ROOT::RVec<T>, depending on a user configuation. This question prescribes the set of methods expected on an immutable sequence. Include a placeholder for a to_RVec() const method.
  • Generate concrete record classes, deriving from ak::RecordView.
  • Generate examples of std::optional to get a sense of how that works. No need for an Awkward-specific class.
  • Generate examples of std::variant to get a sense of how that works. No need for an Awkward-specific class.
  • Do NumpyArray (only real, 1-D arrays, no EmptyArray, multidimensional, or non-contiguous because the conversion to Lookup handles that).
  • Do RegularArray.
  • Do ListArray (no ListOffsetArray because the conversion to Lookup makes them all ListArray).
  • Do IndexedArray.
  • Do IndexedOptionArray.
  • Do ByteMaskedArray.
  • Do BitMaskedArray.
  • Do UnmaskedArray.
  • Do RecordArray.
  • Do UnionArray (couldn't be done in Numba, but C++ has std::variant, so maybe).

That should do it. In the end, it should have a clean API for @ianna to use.

@codecov
Copy link

codecov bot commented Feb 19, 2022

Codecov Report

Merging #1300 (509a3f0) into main (8e68200) will decrease coverage by 0.39%.
The diff coverage is 66.89%.

Impacted Files Coverage Δ
src/awkward/_v2/_connect/cling.py 0.00% <0.00%> (ø)
src/awkward/_v2/_connect/numba/arrayview.py 96.76% <ø> (ø)
src/awkward/_v2/_connect/numba/layout.py 84.34% <ø> (ø)
src/awkward/_v2/_prettyprint.py 63.79% <0.00%> (ø)
src/awkward/_v2/_slicing.py 84.43% <ø> (ø)
src/awkward/_v2/contents/__init__.py 100.00% <ø> (ø)
src/awkward/_v2/forms/__init__.py 100.00% <ø> (ø)
src/awkward/_v2/identifier.py 55.69% <ø> (ø)
.../awkward/_v2/operations/convert/ak_from_buffers.py 85.71% <ø> (ø)
src/awkward/_v2/operations/convert/ak_from_cupy.py 25.00% <0.00%> (-50.00%) ⬇️
... and 168 more

@jpivarski jpivarski marked this pull request as ready for review February 23, 2022 04:01
@jpivarski jpivarski enabled auto-merge (squash) February 23, 2022 04:01
@jpivarski jpivarski merged commit 893c2a6 into main Feb 23, 2022
@jpivarski jpivarski deleted the jpivarski/awkward-to-cpp-converter-with-cling branch February 23, 2022 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant