-
Notifications
You must be signed in to change notification settings - Fork 890
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql plugin #5679
sql plugin #5679
Conversation
0fddb06
to
f640b59
Compare
whoa I needed this! testing. |
f640b59
to
31f62e1
Compare
I was trying various inputs and it looks like this one breaks it:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very interesting PR, but I am not quite convinced we want to a) make it non-experimental and b) provide any guarantees of backwards compatibility to it going forward. It is all new code, contains a number of antipatterns that need justifications for their safety (string concatenation for queries), the ergonomics seems rather dubious to me, and thus promising that it'll not change in future is strange. It also comes in very shortly before we're cutting an RC, not giving much time to figure out the edge cases.
For these reasons I'm tempted to either downgrade the guarantees by making it experimental or delay merging until after the release. What would you prefer?
tools/flattenschema.py
Outdated
@@ -0,0 +1,43 @@ | |||
#! /usr/bin/env python3 | |||
# Script to turn JSON schema into simple CSV describing fields. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More CSVs that aren't really CSVs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are separated by commas :)
Yeah, I'm tempted to rework this to include the raw json schemas instead (or maybe strip them down to just names and types, but leave them in JSON). Doing half the processing here and half at plugin init time is just awkward (esp. as the later patches make the plugin do more complex analysis).
b859c0c
to
5c746b2
Compare
5c746b2
to
1632cbe
Compare
a01ff90
to
2ca0c5d
Compare
2ca0c5d
to
70df95d
Compare
Rebased on master now all prerequisites have been merged. We now handle future deprecations correctly! |
3dc5b46
to
2847a6d
Compare
Rebased on master for man page / generated rust conflicts, and fixed naked |
cfb6116
to
f80a8e3
Compare
Rebased on new CI system! Let's see if that's an improvement! |
f80a8e3
to
c6eada8
Compare
It's a core concept in the spec which isn't directly exposed. Signed-off-by: Rusty Russell <[email protected]> Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
`hash` is a tighter requirement than simply `hex`. Signed-off-by: Rusty Russell <[email protected]>
We actually had a partid allowed (in the oneOf clauses), but didn''t document it. Signed-off-by: Rusty Russell <[email protected]>
It's actually two separate u16 fields, so actually treat it as such! Cleans up zombie handling code a bit too. Signed-off-by: Rusty Russell <[email protected]>
This is useful when you're writing routines to scan it. Signed-off-by: Rusty Russell <[email protected]>
connectd is the only one who uses these routines now. The rest can be linked into a plugin. Signed-off-by: Rusty Russell <[email protected]>
We only handle top-level objects with an array of objects: make sure it is one before we call the routines. Signed-off-by: Rusty Russell <[email protected]>
I don't like it, but we do expose some times like this :( Signed-off-by: Rusty Russell <[email protected]>
If we have a specific type (not just "hex") the length is implied. Signed-off-by: Rusty Russell <[email protected]>
We have "secret" and "hash" types which are often more appropriate. Signed-off-by: Rusty Russell <[email protected]>
This is designed to allow you to perform complex server-side queries. Signed-off-by: Rusty Russell <[email protected]>
Rather than two arrays "columns" (for names) and "fieldtypes" (for types), use a struct. This makes additions easier for successive patches. Also pull process_json_obj() out of the loop in list_done(). Signed-off-by: Rusty Russell <[email protected]>
This requires us to rename "index" fields, rename fields if we have a sub-object, and create sub-tables if we have an array, and handle the fact that some listX commands don't contain array X (listsendpays contains "payments"). Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Painfully created by hand from the source. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
…sip_store. It's quite a lot of code, but these are the most expensive commands we do so it's worth it. Signed-off-by: Rusty Russell <[email protected]>
`"deprecated": true` is obsolete; we don't document them anyway. Where it would have otherwise changed the GRPC wrappers, I actually put the version number in. We allow "listchannels" to have "satoshis" since we have some tests that run in deprecated-api mode. Signed-off-by: Rusty Russell <[email protected]>
For now, we ignore every deprecated field, but put in the logic so that future deprecations will work as expected. Signed-off-by: Rusty Russell <[email protected]>
We now add tables to the strmap as we allocate them, since we don't want to call "finish_td" when we're merely invoked for the documentation, and don't need a database. Signed-off-by: Rusty Russell <[email protected]>
In particular, we generate the schema part from the plugin itself. Signed-off-by: Rusty Russell <[email protected]> Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
And document them! Signed-off-by: Rusty Russell <[email protected]>
This *would* be a 1-line change (add it to Makefile) except that we previously assumed a "list" prefix on commands. These use the default refreshing, but they could be done better using the time-range parameters. Suggested-by: @niftynei Signed-off-by: Rusty Russell <[email protected]>
Good for detection of what fields are present. Signed-off-by: Rusty Russell <[email protected]>
Prompted by @ShahanaFarooqui's playing with examples and finding common errors. Signed-off-by: Rusty Russell <[email protected]>
Also, put the "added" lines in the request schemas for new commands: this doesn't do anything (yet?) but it keeps `make schema-added-check` happy. Signed-off-by: Rusty Russell <[email protected]>
acf6a0c
to
c2face4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The gossip store rebase and zombie update looks solid, as do the corrections.
ACK c2face4
u32 index = bcast->index; | ||
|
||
/* We assume flags is the first field! */ | ||
BUILD_ASSERT(offsetof(struct gossip_hdr, flags) == 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this method of validation.
[ Rebased on #5825 which is in draft ]Applied!This plugin provides a fairly nice way of using SQL queries on our various list commands, without too much penalty (and, with great time savings if the alternative is to pull all the data remotely to query it).
It goes to great lengths to cache listnodes and listchannels.
Feedback welcome!