Emit Transfer Contract Events for Classic Transfers

[leighmcculloch]

What

I propose we unify transfer events on the network in smart contract events, by exploring emitting transfer contract events, like those defined by SEP-41, CAP-46-6, and those potentially defined by #1534, for classic transfers (payments, path payments, etc).

Why

To create a singe unified data stream of all movements of assets / tokens on Stellar.

Today there is no single way to consume transfer events on the network. Transfers are divided by the classic vs smart contract divide.

The RPC is the future way developers are intended to stream and ingest events occurring on Stellar. The RPC exposes smart contract events. However any application wanting to ingest transfers must implement complex logic merging the results of streaming smart contract events and transactions containing payments, path payments, offers, so on. The shortest path to a payments implementation today is to integrate with Horizon's /payments endpoint, however Horizon is the past and RPC is the future. There should be a way for developers to ingest all transfers from RPC.

We could explore ways to add payment information to RPC, such as a similar Horizon-like /payments endpoint, however I think that is making the same mistakes of the past and that would solve the problem at the wrong layer.

RPC's API is designed to provide a raw view into the network and live / current network data. It exposes raw transaction XDR, and raw events. RPC doesn't contain "business logic" for understanding network data. This is critical because if you build on RPC, or raw meta, you have access to the same data, so applications can move between them without being disadvantaged, and without artificial / convenience forcing functions causing folks to interact with one when another would be more appropriate.

This has the future impact of not causing folks to push the RPC in a direction (with feature requests) that would be better served by Galexie or some other service.

If the data is added to network events, then applications built on RPC, or Galexie, or other CDP tooling, or directly on transaction meta from core, have access to the same data.

If we solve the problem at the network layer, then events on Stellar can display a unified view into the movement of value on the network.

(In the future we can even emit events for other non-transfer classic events, such as account creation, merge, etc. The Soroban events structure already support events from G addresses.)

How

All transfers / payments on Stellar classic are the result of interactions with Stellar Assets. All Stellar Assets have a reserved contract address and a reserved installation of the built-in Stellar Asset Contract. For example, on mainnet (pubnet) the native asset has contract address CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA. Stellar classic operations could emit Soroban events using the asset's reserved contract address, even if the reserved contract address hasn't been deployed to yet for that contract.

Links

Discussion on Discord: https://discord.com/channels/897514728459468821/1285736582426394760

[Shaptic]

Stellar classic operations could emit Soroban events using that contract address

Define "emit"? I don't think this is even logistically possible from Core's perspective. Since smart contracts operate in their own land of execution and thus emit events into metadata via host functions, how would classic operations replicate this? Are you suggesting they inject ContractEvents into the meta despite (a) not coming from a contract, (b) possibly referring to a nonexistent contract address, and (c) having no understanding of ScVals? I should let them speak to its feasibility, though.

Don't get me wrong, this sounds really nice, but I just can't see how it's possible.

[leighmcculloch]

Are you suggesting they inject ContractEvents into the meta despite (a) not coming from a contract

"Not coming from a contract" is an implementation detail. The Stellar Asset Contract is a code API into existing classic data. Events are data about how data has changed, and transfer events provide information about how classic data has changed. The proposal here is to emit events about changes in the same data.

possibly referring to a nonexistent contract address

This is unfortunate, but I think a trade-off that's worth the benefits. It's not a nonexistent contract address in anyway that is unsafe. Contract addresses are reserved. The event contains enough information to identify the asset reliably, so even if the SAC contract isn't deployed, the asset is still identifiable.

Because of state archival, it's always possible for a contract to emit an event, and shortly after for that event to reference a contract that's not "live" on chain, so problems that arise from a SAC not existing would be similar to the problems that arise from state archival anyway.

having no understanding of ScVals

ScVal's are just an XDR type. Core knows about all XDR types. For example: https://github.com/search?q=repo:stellar/stellar-core%20ScVal&type=code

this sounds really nice

💯 ❤️

[Shaptic]

Great points all around. Couple more points to follow up:

• I'd like to suggest renaming ContractEvent for XDR clarity since it wouldn't always come from contracts.
• When you say "transfer events," you mean the exact same format as the SAC today, right?
• Does this proposal need a detailed enumeration of the things that happen on the native operations' side that would result in a transfer event?

[leighmcculloch]

Yup a formal proposal / CAP would need a detailed enumeration of classic happenings to events.

[dmkozh]

While this seems easy enough to support for simple payments, I wonder what should we do for things like path payments, DEX trades, claimable balances etc. In Soroban world these would contain a number of transfer events, as well as maybe some other events. I don't think that's feasible for classic, as we wouldn't want to have an operation to emit a lot of events (these won't have a fee, unlike in Soroban case).

I wonder if a subset of operations that move balances is sufficient to support. Basically, it would be nice to identify the requirements/use cases that we can support with certain 'classic' events.

[leighmcculloch]

I wonder if a subset of operations that move balances is sufficient to support. Basically, it would be nice to identify the requirements/use cases that we can support with certain 'classic' events.

+1. If there's general consensus that this is a direction to explore, then I think the next step would be to do some assessment of all possible movements of value on classic, and mapping / excluding, so that we could talk about the implications more specifically.

[Shaptic]

I don't think that's feasible for classic, as we wouldn't want to have an operation to emit a lot of events (these won't have a fee, unlike in Soroban case).

Yeah, this separates it from Horizon's /payments which are 1:1 and is more like its /effects which are 1:N, where we derive all of the side-effects for an operation (e.g., account_debited, offer_updated, account_credited for a simple path payment). N is typically > 1 but I don't think long paths are very common which is the only operation that can have a large number of events. So isn't it generally okay that there won't be fees since there's no VM cost being incurred and the events can't get arbitrarily large?

Alternatively +1 to exploring mapping / excluding.

[dmkozh]

The event fee actually goes towards the meta storage (we don't want it to be possible to just spam meta with arbitrary events for free), but I guess the missing fee is probably acceptable given that the event size will be proportional to the diffs we emit anyway. But I'm still not sure if implementing very granular event reporting is worth the effort, which is why I'd like to understand what's actually useful.

[mollykarcher]

RPC's API is designed to provide a raw view into the network and live / current network data. It exposes raw transaction XDR, and raw events. RPC doesn't contain "business logic" for understanding network data.

Just wanted to big ➕ this! There have been a handful of conversations lately (internal + external) that have assumed or implied that we will be rebuilding or recreating the unified data models that we built out in Horizon, but now in RPC. Repeating this behavior would only recreate a lot of the issues we saw with Horizon, and allow RPC to become the defacto backend/monolith for any and all applications that want on-chain data. So yes, I'd wholeheartedly agree that the right layer to solve this at feels like the source-of-truth of the events data (ie. Core).

I wonder if a subset of operations that move balances is sufficient to support

I think this is the right bit to start with as there's generally a lot of interest around a unified understanding of "what is a payment" or "what moves value" (lots of interest/talk around an open-source payment indexer, for example).

[JakeUrban]

I'm a big fan of this proposal. I want to point out that there are two layers in this unification effort -- the source of the data and the format of the data.

Providing a stream for all stellar asset transactions at the protocol level via events, as opposed to at the application level (like Horizon's /payments endpoint), makes sources of this data interchangeable. Clients can use RPC, a txmeta lake (produced by galaxie), captive core, or some other source of txmeta.

However, I think it is equally important that we unify the format of the data provided. We could add events for all stellar asset transfers, providing a unified the stream for this data, but if the format of these events vary, clients will inevitably support a subset of these formats. We have this same problem right now with Horizon's /payments endpoint -- most clients support the simple payment format, but don't support the path payment or SAC transfer formats.

Unfortunately I think we've already introduced multiple data formats in cases where a single format could have sufficed. For example, mint, burn, and clawback are all effectively transfers, but clients may unintentionally omit support for these events in their implementation because the first topic is the name of the contract function, rather than "transfer". If clients wanted to know more about the nature of the transfer (i.e. whether it was a mint, burn, clawback, simple transfer, path payment, etc.) they could inspect other information included in the associated TransactionMetaV3.

So lets take path payments for example. I would advocate that the event(s) emitted as a result of a path payment operation use the ["transfer", from, to, asset]: amount format. Clients interested in this event can inspect the associated txmeta to determine if it was a path payment if they'd like.

I'd like to see events in this format emitted for the following classic operations:

• payment
• both path payment types
• create account
• merge account

Now there are some classic operations where using the same format may not be the best path because there is not an obvious from and to address.

• create claimable balance
• claim claimable balance
• amm deposits
• amm withdraws

In these cases, we may need to use a different format. The good news is that most applications won't need to care about these events because they don't use claimable balances or native AMMs.

Finally, I think we should also consider emitting events for transaction fees. These are a kind XLM transfer, and anyone using events to update balances won't be able to maintain an accurate XLM balance without this event.

[dmkozh]

These are a kind XLM transfer, and anyone using events to update balances won't be able to maintain an accurate XLM balance without this event.

This is a very important point that I've alluded to above. If we want to make it possible to track the exact balances only via events, then the things will become much more complicated, as we'd need to cover every balance movement at once. It's not obvious to me that this should be a requirement. I think an argument can be made that getting the balance is a very common operation and in general a user shouldn't ingest each and every event concerning a given account (and also interpret these correctly), just to figure out something as simple as balance is. Thus I'm not sure this should be a requirement.

[leighmcculloch]

Just to clarify some history and why on these things:

mint, burn, and clawback are all effectively transfers, but clients may unintentionally omit support for these events in their implementation because the first topic is the name of the contract function, rather than "transfer"

Transfers is the movement of an amount between two addresses and SEP-41 does use "transfer" as the topic of any event that moves an amount between accounts. For example, the transfer_from function generates a "transfer" event.

The mint, burn, clawback events are separate because they create and destroy amounts, they do not move amounts between the issuer and the holder, even though they do affect the balance of a holder. On Classic stellar these operations were represented by transfers, but in Soroban it was decided not to represent those happenings as transfers. An application can still ingest these events also though.

[Shaptic]

Just to clarify

On Classic stellar these operations were represented by transfers

More accurately, Horizon interprets the side effects of these operations as payments and presents them as such, where a payment is literally just a concept we made up. Just to echo the general sentiment that standardization of a transfer at the source of truth would go a long way 👍