ABCI: Switch to list of lists of tags for transaction results, begin block, and end block #1859

zramsay · 2018-07-02T19:49:08Z

cwgoes · 2018-07-11T22:28:13Z

cwgoes · 2018-07-24T02:29:14Z

Also, community suggestion: easy way via API to query all events for a particular block (which can then be parsed through and stored elsewhere, e.g. SQL DB).

cwgoes · 2018-07-24T02:30:46Z

(thanks @ajcronk for the suggestion!)

ValarDragon · 2018-08-10T01:14:18Z

Is this needed prelaunch? AFAICT this is to increase indexing speed. (Which I think can be punted to postlaunch, unless we don't want to break ABCI soon after launch. If this is true, I think we should make stubs for more ABCI changes). Per Anton's comment in the original issue, I agree that we shouldn't re-invent the wheel here, and should aim to be compatible with existing databases. e.g. #1161

cwgoes · 2018-08-10T01:35:41Z

Is this needed prelaunch? AFAICT this is to increase indexing speed. (Which I think can be punted to postlaunch, unless we don't want to break ABCI soon after launch. If this is true, I think we should make stubs for more ABCI changes). Per Anton's comment in the original issue, I agree that we shouldn't re-invent the wheel here, and should aim to be compatible with existing databases. e.g. #1161

Why would this increase indexing speed?

The reason on the SDK side is to allow prefixed error codes for particular modules and avoid the necessity of inventing a complex codespacing system to deal with potential module conflicts.

ValarDragon · 2018-08-10T01:56:45Z

Oh woops your right, I completely misunderstood the comments on the prev issue.

If this is the goal though, why don't we make the results a list of structs consisting of a kv pair, k being byte slice and v being list of byte slices. Then we could also sort the results by key to get log n searching.

cwgoes · 2018-08-10T07:53:07Z

If this is the goal though, why don't we make the results a list of structs consisting of a kv pair, k being byte slice and v being list of byte slices. Then we could also sort the results by key to get log n searching.

How would that preserve the connection between tags associated with a particular event?

It's not that we need to support multiple values-per-tag-key, it's that we want to be able to return multiple indexable/searchable "events" per transaction / BeginBlock / EndBlock, e.g. for multiple validators being slashed for downtime in the same BeginBlock.

ebuchman · 2018-09-02T06:08:16Z

@cwgoes @faboweb thoughts on prioritizing this pre/post launch?

cwgoes · 2018-09-03T11:03:52Z

@cwgoes @faboweb thoughts on prioritizing this pre/post launch?

Probably the most important use case for this at launch is slashing events, multiple of which can happen per abci.RequestBeginBlock. Without this, I don't see an easy way to report any number of slashing events in an easy-to-query manner. I guess we could prefix tags or try another disambiguation method.

ebuchman · 2018-10-03T01:00:12Z

Noting #2314 (comment)

Also not entirely sure what @jaekwon is saying in #2314 (comment)

melekes · 2018-10-03T08:03:22Z

Didn't we explicitly get rid of types here at one point?

yes, but I'd vouch to rethink if the current design does not satisfy our needs

This is cool, but I think it means we'd need to unmarshal the JSON and index the keys in there too if we want to support eg. a query for "validator X slashed"

that won't not be a problem for Postgresql indexer, KV indexer will require modification

melekes · 2018-10-03T08:16:26Z

Do you know if the reasoning for this was documented anywhere?

#1369 (comment) I think the reasoning was it's just simpler and we don't loose that much in term of performance ([]byte string requires parsing) #1369 (comment)

I wasn't able to find anything else, sorry.

jackzampolin · 2018-12-19T22:35:28Z

So @alexanderbez spent some time trying to implement this in the SDK and his judgement was that we needed it to be a tendermint change. cosmos/cosmos-sdk#3171

Is it possible to start work on this feature?

melekes · 2018-12-20T07:00:04Z

Linking #1385 (comment) for other ways to solve the core problem here.

ebuchman · 2019-01-13T23:38:45Z

See the original PR here: #1803

Shouldn't be too difficult to resuscitate it.

Also see #1803 (comment) - an event should look like:

message Event {
  string type
  repeated common.KVPair attributes
}

Then Responses can include repeated Event events

ebuchman · 2019-01-16T15:00:43Z

This is still an important feature, but has been deemed less critical from a Cosmos launch perspective. Since we're no longer planning to hash tags into the header at this point (#1007), making this change will not break existing blockchains, so we can make it more leisurely. I will move it to a slightly later milestone

ethanfrey · 2019-05-07T12:09:32Z

#1385 was closed in favor of this, but I don't understand how this works for the client queries and suscriptions. This proposes a solution (list of list of tags), but the rationale behind it, or specific examples of how it will make life easier for the client queries, are missing.

Can you please add a summary of the changes to the issue description?

melekes · 2019-05-07T18:12:55Z

Sorry for confusion. We'll write a spec. There are two distinct issues, but new events format will require modifying indexing and querying code anyway. So I was hoping we could fix both issues.

alexanderbez · 2019-05-13T14:31:06Z

@ethanfrey mind taking a look/review of #3643? It accomplishes what is outlined in this issue. Namely, using list-of-lists so we can support aggregate and more complex queries in ABCI responses. In addition, it also utilizes your suggestion from #1385 in using map[string][]string, so you can have subscriptions like you've mentioned in that issue.

## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <[email protected]> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <[email protected]> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <[email protected]> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <[email protected]> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test

melekes · 2019-06-12T12:52:44Z

Merged #3643 to develop. Thanks @alexanderbez 👍

…ermint#1863) * Updates go crypto package to v0.17.0 (tendermint#1859) (cherry picked from commit fd87fda) # Conflicts: # go.mod # go.sum * Solving conflict --------- Co-authored-by: lasaro <[email protected]>

zramsay assigned cwgoes Jul 2, 2018

zramsay mentioned this issue Jul 2, 2018

Switch to list of lists of tags for transaction results, begin block, and end block tendermint/abci#273

Closed

zramsay added the C:abci Component: Application Blockchain Interface label Jul 2, 2018

ebuchman added this to the launch milestone Jul 5, 2018

xla added the T:enhancement Type: Enhancement label Jul 8, 2018

cwgoes mentioned this issue Jul 11, 2018

[WIP] Port tendermint/abci#279 - codespace string, list of lists of tags #1803

Closed

4 tasks

ebuchman added API T:breaking Type: Breaking Change labels Jul 27, 2018

xla unassigned cwgoes Aug 1, 2018

ebuchman mentioned this issue Sep 2, 2018

[ADR] ABCI errors and events #2314

Merged

ebuchman modified the milestones: v1.0, launch Sep 13, 2018

ebuchman modified the milestones: launch, v1.0 Oct 4, 2018

melekes mentioned this issue Nov 26, 2018

Missing documentation: non-unique tag keys #2905

Closed

cwgoes mentioned this issue Nov 29, 2018

Remove action tag from individual handlers cosmos/cosmos-sdk#2941

Closed

4 tasks

ebuchman modified the milestones: v1.0, v0.30.0 Jan 13, 2019

ebuchman mentioned this issue Jan 13, 2019

Add events to results hash #1007

Closed

ebuchman modified the milestones: v0.30.0, v0.29.0 Jan 13, 2019

ebuchman modified the milestones: v0.29.0, v0.31.0 Jan 16, 2019

ebuchman modified the milestones: v0.31.0, v0.32.0 Feb 4, 2019

melekes mentioned this issue Mar 18, 2019

Allow setting tags to index at runtime #3437

Closed

ebuchman modified the milestones: v0.33.0, v0.32.0 May 1, 2019

alexanderbez mentioned this issue May 4, 2019

Expose all distribution actions/events/transactions/operations cosmos/cosmos-sdk#3956

Closed

4 tasks

melekes mentioned this issue May 6, 2019

Subscribing to tx event works for multiple kvpairs using same key #1385

Closed

This was referenced May 8, 2019

Tagging Refactor: Events using List of Lists #3642

Closed

abci: Refactor tagigng events using list of lists #3643

Merged

melekes closed this as completed Jun 12, 2019

firelizzard18 pushed a commit to AccumulateNetwork/tendermint that referenced this issue Feb 1, 2024

Updates go crypto package to v0.17.0 (tendermint#1859)

fd87fda

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ABCI: Switch to list of lists of tags for transaction results, begin block, and end block #1859

ABCI: Switch to list of lists of tags for transaction results, begin block, and end block #1859

zramsay commented Jul 2, 2018

cwgoes commented Jul 11, 2018

cwgoes commented Jul 24, 2018

cwgoes commented Jul 24, 2018

ValarDragon commented Aug 10, 2018

cwgoes commented Aug 10, 2018 •

edited

Loading

ValarDragon commented Aug 10, 2018

cwgoes commented Aug 10, 2018 •

edited

Loading

ebuchman commented Sep 2, 2018

cwgoes commented Sep 3, 2018 •

edited

Loading

ebuchman commented Oct 3, 2018

melekes commented Oct 3, 2018

melekes commented Oct 3, 2018

jackzampolin commented Dec 19, 2018

melekes commented Dec 20, 2018

ebuchman commented Jan 13, 2019

ebuchman commented Jan 16, 2019

ethanfrey commented May 7, 2019

melekes commented May 7, 2019

alexanderbez commented May 13, 2019

melekes commented Jun 12, 2019

ABCI: Switch to list of lists of tags for transaction results, begin block, and end block #1859

ABCI: Switch to list of lists of tags for transaction results, begin block, and end block #1859

Comments

zramsay commented Jul 2, 2018

cwgoes commented Jul 11, 2018

cwgoes commented Jul 24, 2018

cwgoes commented Jul 24, 2018

ValarDragon commented Aug 10, 2018

cwgoes commented Aug 10, 2018 • edited Loading

ValarDragon commented Aug 10, 2018

cwgoes commented Aug 10, 2018 • edited Loading

ebuchman commented Sep 2, 2018

cwgoes commented Sep 3, 2018 • edited Loading

ebuchman commented Oct 3, 2018

melekes commented Oct 3, 2018

melekes commented Oct 3, 2018

jackzampolin commented Dec 19, 2018

melekes commented Dec 20, 2018

ebuchman commented Jan 13, 2019

ebuchman commented Jan 16, 2019

ethanfrey commented May 7, 2019

melekes commented May 7, 2019

alexanderbez commented May 13, 2019

melekes commented Jun 12, 2019

cwgoes commented Aug 10, 2018 •

edited

Loading

cwgoes commented Aug 10, 2018 •

edited

Loading

cwgoes commented Sep 3, 2018 •

edited

Loading