Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v0.13.0 Introducing the Rolodex. #187

Merged
merged 75 commits into from
Nov 4, 2023
Merged

v0.13.0 Introducing the Rolodex. #187

merged 75 commits into from
Nov 4, 2023

Conversation

daveshanley
Copy link
Member

@daveshanley daveshanley commented Oct 24, 2023 •
edited
Loading

The rolodex is the solution libopenapi has needed for some time. It reorganizes how the index works, how references are composed and how things are looked up when building the model.

The rolodex creates virtual filesystems for exploded specifications that are scattered across multiple local or remote locations. The rolodex can contain multiple local or remote filesystems for documents, and can lookup anything
via its fully qualified location.

Custom filesystems can be loaded in, instead of the local and remote ones provided by libopenapi as the rolodex uses fs.FS as an interface, so any custom implementation can be used.

This is a large breaking change for how the index operates, there are breaking changes to all low-level models, mainly with the introduction of a context.Context parameter. This is used by the rolodex to know where to resolve references from, when deeply nested recursive reference resolution.

The index no-longer operates the way it used to and how indexes are used by the rolodex is also new.

There should be minimal configuration changes at the top-level API for documents, and no changes to the high-level model.

@daveshanley
Adding the rolodex
ec3bf9e 
Signed-off-by: quobix <[email protected]>
@daveshanley
Working through the rolodex design.
8952d76 
assembling the low level blocks, all based on `fs.FS` interfaces.

Signed-off-by: quobix <[email protected]>
@daveshanley
working through rolodex build
81902ef 
Signed-off-by: quobix <[email protected]>
@daveshanley
Performing some major surgery on the index
de85651 
To make this work correctly, this needs completely shaking up and a transfer of ownership. The index is now local,
the rolodex is now global.

Signed-off-by: quobix <[email protected]>
@daveshanley
Major surgery on the index and resolver. A complete flip in design.
511843e 
Signed-off-by: quobix <[email protected]>
@daveshanley
working through rolodex design and using it externally via vacuum
8b795c6 
this is some complex and messy work.

Signed-off-by: quobix <[email protected]>
@daveshanley
a first working engine of the new design.
d5f72a2 
There is a horrible amount of work to be done to clean this up, and wire in remote support. but so far, this is working as expected and is now a much cleaner design, (once everything has been cleaned up that is)

Signed-off-by: quobix <[email protected]>
@daveshanley
chopping through index changes, basic design works.
cea7bb0 
seems to be holding, more tests to change.

Signed-off-by: quobix <[email protected]>
@daveshanley
More surgery on rolodex and the index
0fcd55e 
Bringing the tests back online, bit by bit.

Signed-off-by: quobix <[email protected]>
@daveshanley
Another massive surgical strike with the rolodex and index reshuffle.
5197176 
Signed-off-by: quobix <[email protected]>
@daveshanley
resolver tests all operational
48c83dd 
time to start some cleanup.

Signed-off-by: quobix <[email protected]>
@daveshanley
cleaning up a little.
7781906 
Signed-off-by: quobix <[email protected]>
@daveshanley
digital ocean now running correctly
9ee1afe 
Sucking in all the files!

Signed-off-by: quobix <[email protected]>
@daveshanley
working through logging now and further tests
054103b 
starting the circle dance now.

Signed-off-by: quobix <[email protected]>
@daveshanley
bashing through usecases and updating tests as we go.
b295e8f 
so many things that can go wrong. have to catch them all.

Signed-off-by: quobix <[email protected]>
@daveshanley
All spec_index tests pass
1bf772a 
It’s so, so much faster than before, intelligent and ready for scale. I’m excited!

Signed-off-by: quobix <[email protected]>
@daveshanley
More fine tuning, handling resolving and edge cases now.
afe8945 
Signed-off-by: quobix <[email protected]>
@daveshanley
whacking the shit out of exploded use-cases
bf270d3 
Before everything worked, but was completely accurate, now everything works and everything is absolute and can be resolved. Phew, what a mission!

Signed-off-by: quobix <[email protected]>
@daveshanley
index tests all pass! now time to clean.
be7e477 
Signed-off-by: quobix <[email protected]>
@daveshanley
First sweep at cleaning up dead code
28047d0 
first round of a number I am sure, lots to clean.

Signed-off-by: quobix <[email protected]>
@daveshanley
Another round of cleaning.
3bf830c 
Signed-off-by: quobix <[email protected]>
@daveshanley
An enormous amount of surgery on the low level model.
8717b3c 
Every `Build()` method now requires a `context.Context`. This is so the rolodex knows where to resolve from when locating relative links. Without knowing where we are, there is no way to resolve anything. This new mechanism allows the model to recurse across as many files as required to locate references, without loosing track of where we are in the process.

Signed-off-by: quobix <[email protected]>
@daveshanley
A massive test update to bring everything inlne with the new `Buildab…
6e9db7f 
…le` signature.

All tests in index and datamodel now pass. The rolodex fixes all the things.

Signed-off-by: quobix <[email protected]>
@daveshanley
All tests pass! logs of tests fixed and tuning API for high level
d30ac24 
Document configuration has been simplified, no more need for AllowRemote stuff in the document configuration, it’s assumed by setting the baseURL or the basePath.

Signed-off-by: quobix <[email protected]>
@daveshanley
Changed document signatures to use error instead of []error
5d717bd 
Also removed old swagger `CreateDocument` method that has been deprecated.

Signed-off-by: quobix <[email protected]>
@daveshanley
Working through test coverage
c1cf240 
This will be a bit of a slog, new code built in the hot path will need some love and attention.

Signed-off-by: quobix <[email protected]>
@daveshanley
bumped go version and build
f644fbb 
Signed-off-by: quobix <[email protected]>
@daveshanley
Updated logic to handle a single core
e26897d 
The remote loader was blocking the only thread.

Signed-off-by: quobix <[email protected]>
@daveshanley
Copy link
Member Author

Looks like the lovely single core runners are choking on locks.

@daveshanley
Pipeline is failing because it’s hanging somewhere
b82b46e 
cannot re-create the problem locally, even when setting GOMAXPROCS to 1

Signed-off-by: quobix <[email protected]>
@daveshanley
More cleaning and added docs.
80b2b2d 
We’re ready for review

Signed-off-by: quobix <[email protected]>
@daveshanley daveshanley marked this pull request as ready for review November 1, 2023 23:54
@daveshanley
Copy link
Member Author

For end users of the high-level model. You should notice no real changes. However this PR is loaded with breaking changes, I had to open up the heart of the library and add new bones to the skeleton.

TristanSpeakEasy
TristanSpeakEasy approved these changes Nov 2, 2023
View reviewed changes
Copy link
Contributor

@TristanSpeakEasy TristanSpeakEasy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good to me, nothing too scary in here from a migration perspective. Going to start a branch in our product and give the upgrade a try and see if we find any early issues

@daveshanley
Addressed comments from review and fixed bug with schema props
8bbb022 
props did not have context, therefore they had no idea where they were or where to resolve from.

Signed-off-by: quobix <[email protected]>
@daveshanley
added context to schema buildout
a8a0e1d 
model was failing on subschemas with refs, needed context

Signed-off-by: quobix <[email protected]>
@daveshanley
Tuning parameter exraction for circular ref handling
713aeec 
Lots and lots of variations. means lots of branches to check.

Signed-off-by: quobix <[email protected]>
@daveshanley
cleaning up unreachable code.
78763fd 
Signed-off-by: quobix <[email protected]>
@daveshanley
fixed issue with what-changed and path detection #186
ddb761c 
Signed-off-by: quobix <[email protected]>
@daveshanley daveshanley mentioned this pull request Nov 2, 2023
Closed
@daveshanley
Merge branch 'main' of https://github.com/pb33f/libopenapi into v0.13.0
91edf28 
Signed-off-by: quobix <[email protected]>

# Conflicts:
#	what-changed/model/info_test.go
@daveshanley
Copy link
Member Author

It's time to merge.

@daveshanley daveshanley merged commit 771baaf into main Nov 4, 2023
3 checks passed
@daveshanley daveshanley deleted the v0.13.0 branch November 4, 2023 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TristanSpeakEasy TristanSpeakEasy TristanSpeakEasy approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@daveshanley @TristanSpeakEasy