Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for views with Lens transforms #2311

Merged
merged 5 commits into from
Feb 15, 2024

Conversation

AndrewSisley
Copy link
Contributor

Relevant issue(s)

Resolves #2070

Description

Adds support for views with Lens transforms.

@AndrewSisley AndrewSisley added feature New feature or request area/query Related to the query component labels Feb 13, 2024
@AndrewSisley AndrewSisley added this to the DefraDB v0.10 milestone Feb 13, 2024
@AndrewSisley AndrewSisley requested a review from a team February 13, 2024 19:55
@AndrewSisley AndrewSisley self-assigned this Feb 13, 2024
@AndrewSisley AndrewSisley force-pushed the 2070-lens-views branch 2 times, most recently from bf5d3ef to 0c6e63c Compare February 13, 2024 20:03
Copy link

codecov bot commented Feb 13, 2024

Codecov Report

Attention: 52 lines in your changes are missing coverage. Please review.

Comparison is base (765156e) 74.12% compared to head (aa21960) 74.07%.

❗ Current head aa21960 differs from pull request most recent head 3bb134d. Consider uploading reports for the commit 3bb134d to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2311      +/-   ##
===========================================
- Coverage    74.12%   74.07%   -0.05%     
===========================================
  Files          259      260       +1     
  Lines        25796    25946     +150     
===========================================
+ Hits         19119    19217      +98     
- Misses        5341     5383      +42     
- Partials      1336     1346      +10     
Flag Coverage Δ
all-tests 74.07% <70.45%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
client/descriptions.go 65.96% <ø> (ø)
http/client.go 44.70% <100.00%> (ø)
http/handler_store.go 78.84% <100.00%> (ø)
planner/datasource.go 88.00% <100.00%> (ø)
planner/planner.go 79.50% <100.00%> (+0.11%) ⬆️
planner/view.go 83.08% <100.00%> (+4.65%) ⬆️
db/txn_db.go 54.36% <50.00%> (ø)
db/view.go 65.57% <72.73%> (+0.87%) ⬆️
cli/view_add.go 70.59% <53.12%> (-17.41%) ⬇️
planner/lens.go 67.35% <67.35%> (ø)

... and 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 765156e...3bb134d. Read the comment docs.

Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, just the one main concern regarding the explain debug test similar to this comment when view node was added: #2114 (comment)

Copy link
Member

@shahzadlone shahzadlone Feb 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Since a new plan node is being added, I would make the same comment as: #2114 (comment)

tldr: Please add the new name to allPlanNodeNames variable in the tests/integration/explain.go file for debug node and add a test for the debug explain under the folder tests/integration/explain/debug similar to tests/integration/explain/debug/with_view_test.go. You don't have to make it explainable however in this PR, that discussion can be had elsewhere but would suggest opening an issue for it.

Edit: I crossed-out the suggestion of opening an explain issue, as @fredcarle has already made explain issue(s).

Copy link
Contributor Author

@AndrewSisley AndrewSisley Feb 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Add explain test stuff

@shahzadlone
Copy link
Member

LGTM. I've create a few follow up issues that stemmed from reviewing this.

Ahh cheers for making the explain issues. Are the issues for discussion if we need lens and view nodes, or we already decided that we need explain for them?

@AndrewSisley
Copy link
Contributor Author

Are the issues for discussion if we need lens and view nodes, or we already decided that we need explain for them?

I don't think that they needing explain is a given, although I think the fact that we have this question and hide some nodes from our users is an unwanted complication for both us and them and plan on chatting about this in the standup (it came up in some of my other work).

@AndrewSisley
Copy link
Contributor Author

LGTM. I've create a few follow up issues that stemmed from reviewing this.

Cheers Fred :)

@AndrewSisley AndrewSisley merged commit aec7a61 into sourcenetwork:develop Feb 15, 2024
27 of 28 checks passed
@AndrewSisley AndrewSisley deleted the 2070-lens-views branch February 15, 2024 15:10
}

func (n *lensNode) Next() (bool, error) {
hasNext, err := n.output.Next()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: var with name hasValue would fit better here

// the resultant field set. We cannot use [append] because the index of each field
// must still correspond to it's field ID.
originalProps := properties
properties = make([]any, fieldIndex+1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: it will be more efficient to run through indexes to find the maximum value and then allocate properties. It can be also done before the loop.
Otherwise there might be scenarios when properties can be allocated several times.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which one is more efficient depends on the two sizes really. Not every indexes key will be present in the map and your suggestion will over-allocate when that happens.

Your suggestion is probably better more of the time, but I don't think it is worth a new PR.

shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
## Relevant issue(s)

Resolves sourcenetwork#2070

## Description

Adds support for views with Lens transforms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/query Related to the query component feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for un-cached Lens-driven Defra Views
4 participants