TokenTransferProcessor and events - Data modeling compliant with CAP-67 unified events #5580
Comments
|
Thanks for the detailed writeup @karthikiyer56. A couple questions / comments:
[
{
"eventType": "Transfer",
"from": "LPhash",
"to": "AccountA",
"asset": "assetA",
"amount": "12345"
},
{
"eventType": "Transfer",
"from": "LPhash",
"to": "AccountA",
"asset": "assetB",
"amount": "12345"
}
]Is this array supposed to contain the events for the operation, the transaction, or the ledger? Clients could determine which events were a result of executing a given operation and transaction on their own given the txhash and operation index, so grouping by operation or transaction isn't necessary but I wanted to clarify.
For example, if I deposit funds in the Blend protocol, a blend contract will call the token's contract which will emit the transfer event. However, it seems as if this processor will not tell me anything about the invoking contract. Clients like Freighter would like to represent this example as a deposit into blend, rather than a transfer to a contract. Another example is the consumption of an allowance. According to SEP41, calls to the What I assume is the answer to both of these questions is that the client will need to inspect the transaction's footprint and auth tree to gather this information itself. This is doable, but I think its worth considering whether this package should implement some of this behavior itself in order to lessen the implementation burden of clients. |
|
Re these 2 comments:
The grouping you mentioned was simply for illustration purposes, specifically to show that - a path payment operation can emit these 4-5 events, based on who it was filled against I have added a section about downstream usage by clients to highlight what the library public functions might look like and how a client can use it. There is no grouping that will be done by the library. Clients can choose to do their own grouping on the fields in the But yes, in a nutshell, extra processing should be done outside the scope of the TokenTransferProcessor package. |
|
@karthikiyer56 overall, looks good! I have a suggestion for a slightly different representation for your protobuf schema. I think each event type could be a different message and the syntax = "proto3";
package token_transfer;
// Address message with oneof for different address types
message Address {
oneof address_type {
string smartContractAddress = 1; // Smart Contract address
string accountAddress = 2; // Account address
string liquidityPoolHash = 3; // Liquidity Pool hash
string claimableBalanceId = 4; // Claimable Balance ID
}
}
// EventMeta message
message EventMeta {
uint32 ledgerSequence = 1;
google.protobuf.Timestamp closedAt = 2;
optional uint32 operationIndex = 3; // Optional field
string txHash = 4; // Stellar transaction hash
}
message Transfer {
Address from = 1;
Address to = 2;
string asset = 3;
string amount = 4;
}
message Mint {
Address to = 1;
string asset = 2;
string amount = 3;
}
message Burn {
Address from = 1;
string asset = 2;
string amount = 3;
}
message Clawback {
Address from = 1;
Address admin = 2;
string asset = 3;
string amount = 4;
}
message Fee {
Address from = 1;
string amount = 2;
}
// TokenTransfer message
message TokenTransfer {
EventMeta meta = 1;
oneof event {
Transfer transfer = 1;
Mint mint = 2;
Burn burn = 3;
Clawback clawback = 4;
Fee fee = 5;
}
} |
I think the best approach would be to look at the operation which emitted produced the event. The ledger, transaction hash, and operation index will be part of the event so it will be possible to isolate the operation which produced the event. Once you have the operation you can look at the contract id and the invocation details (function name and function args). Etherscan displays this info in the Input Data section: 
https://etherscan.io/tx/0x4af64d6aa10a2cbab53bd8e9cc1c0c186dd714971b6db814a5103feae287caf4 Additionally you can look at the other soroban events which were emitted by that operation. In the example of depositing into blend I would guess that there is an event emitted by the blend smart contract when you complete that action. However you would need to add support for understanding events on an ad hoc basis. In other words, freighter could add support for Blend but if another smart contract comes along you will need to have more code to support that smart contract. As @karthikiyer56 mentioned, I think the best place for the code which parses the function invocation or parses additional events should be a separate function / library |
|
Re:
Updated the proto to reflect that. |
|
Hey Karthik, thanks once again for the writeup I have some follow up questions:
1. I see that for SEP-41 compliant custom tokens you emit events like:
Is there a reason the from and to are ContractIds and not both ContractIds and AccountIds like you have for SAC events? I’m thinking of use cases like someone transferring a certain amount of a custom token like USDL from one stellar account to another. How would these be modeled?
|
|
Re:
I changed the Re:
and
Information about custom tokens is unknowable at the ingestion level because of the very fact that they're custom, so they can store information however they want.
The answer to this is along the same lines as above. Re:
Yes to all your questions. |
Unfortunately, so would we 😉 which means the ingestion processor would now also need to run a simulation library and actively execute transactions as part of ingestion to glean this information. Like @karthikiyer56 said - impossible. A possible "fix" here could be to solve this underlying problem:
We could, theoretically, introduce an extension / breaking change to SEP-41 (the event standard for custom tokens) to ask nicely for custom tokens to shove those details into their events, but again that's not something we can do here at the ingestion layer. |
|
For SEP-41 custom tokens, If the InvokeHostFunction operation corresponding to the token transfer event (which we can get through a combination of ledger, txhash and operationIndex) can give us the contract address: https://github.com/stellar/go/blob/master/xdr/xdr_generated.go#L29459 |
|
@karthikiyer56 Just to clarify, every event in the list of events emitted by Mange Buy/Sell offer will have a corresponding ManageBuy/Sell offer operation (through a combination of txhash, ledger and operationIndex) right? And the same is true for all the events emitted by Path payment strict send/receive ? All those events will be tied to a PathPaymentStrictSend/Receive operations correct? |
If the contract is the source or destination of the transfer, it will be part of the event. Otherwise, if you wanted to identify a "participant" (e.g. intermediary contracts) then yes, you could either inspect the envelope (which would only give you the top-level contract address being invoked, no subcontract calls) or the invocation tree returned by simulation (which would show you all authorized contract invocations), but it wouldn't necessarily be present in the meta because the contract itself didn't "do" anything that had a side-effect on itself. But this begs a question: if a contract is "involved" in a transfer by happenstance (e.g. contract C transfers XLM from account A to account B, for some reason), do you even care about seeing C? You'd see a
Technically, they don't have to. There's nothing in SEP-41 explicitly preventing someone from writing the Rust equivalent of def name() -> str:
return random.choice("hey", "now")or just changing its name at any point after creation, but that'd be a really strange thing to do, so I think it's more than reasonable to assume those values won't change. |
I want to be able to identify the name/symbol of the custom token that is being transferred in the SEP-41 events, and I'm assuming that I can get that info by calling name/symbol on the contract corresponding to the contract address in the InvokeHost function: https://github.com/stellar/go/blob/master/xdr/xdr_generated.go#L29459
You're right, in this case we don't care about C |
|
@gouthamp-stellar good call out, we should be including the contract id of the stellar asset / sep-41 token which emitted the event. Once you have the contract id, you can obtain the symbol and name using the simulate transaction endpoint. As @Shaptic mentioned, technically speaking those values are not immutable but practically speaking I think you can cache them in the wallet backend for some period of time to reduce the amount of simulate transaction requests. You can also maintain a static list of token contract id -> name / symbol mapping for the token contracts that are most common and we know for a fact by looking at the source code the name / symbol do not change. @karthikiyer56 I think it makes sense to include an optional contract id field in the protobuf message: // TokenTransfer message
message TokenTransfer {
EventMeta meta = 1;
// Asset might not always be present so marking it as optional, especially in events emitted by SEP-41 custom token contracts
optional string asset = 2;
// Contract id of the SAC / SEP-41 token
optional string contract_id = 2;
oneof event {
Transfer transfer = 3;
Mint mint = 4;
Burn burn = 5;
Clawback clawback = 6;
Fee fee = 7;
}
} |
|
@tamirms if |
|
@Shaptic I was thinking that the contract id could be omitted in the case that the event corresponds to stellar classic activity (e.g. a stellar classic payment which emits a transfer event). But we could make the contract id always present even if the event is emitted by stellar classic because we can derive the stellar asset contract id from the asset name and issuer |
yes, that would be good, we could proactively setup for this in the IDL, and the asset string value currently proposed in IDL is somewhat opaque, it requires extra work/know-how for downstream consumers to decode if it's in classical, we could define a bit more schema to describe the asset encoding for more type safety, still preserve the recommend optional aspect: related to optional aspect, it uses the wrapper type |
|
I like @sreuland 's idea a lot. The CustomAsset.Name field can be null for now, but if/when the SEP-41 change were to happen, it can start getting filled and we wouldn't have to have a cache/make contract calls to get these values in the wallet backend. It's not a bad idea to have Asset be an actual type as opposed to a string, in case we want to add other fields to it. I do think we still need that contract_id field in the TokenTransfer event in the meantime though, so we can use that to cache/make contract calls to get the token's name etc. |
I think it's very unlikely that SEP-41 will change to include both the token name and symbol in the events payload. IMO it's better for the |
|
I understand from the meeting today that there isn't an immediate user of the protobuf, and so the product gains / impact to users of introducing protobufs are less clear. I noticed @karthikiyer56 mentioned today that they come with additional work to setup, and to maintain. There's also the cost to upskill folks on it as a new tool. Would not using protobufs today prevent the processor from adopting protobufs in the future if we needed to? I don't think so. Is that correct? If so we could adopt protobufs at a later date, or applications importing the processor could. Introducing other "official" serialisation formats is also problematic when it comes to fragmenting integration options, and accidentally creating bespoke data models that over time diverge from the protocol's data models and are difficult to change (e.g. Horizon's API). It should be possible for the wallet backend to use the RPC to stream events when CAP-67 is implemented, but the RPC's API doesn't use this protobuf format or data model. Does that mean the RPC API is insufficient, or need fixing, or are we building two different data models that serve the same purpose? |
|
I see GetTokenTransferEventsFromTransaction takes an ingest.LedgerTransaction as argument. Is there a way to convert/map either TransactionMetaXDR or any other field in RPC's getTransactions endpoint: https://developers.stellar.org/docs/data/rpc/api-reference/methods/getTransactions into an ingest.LedgerTransaction? I ask because getTransactions is how wallet backend ingests from rpc. If this cant be done, the way I see it, we have the following options:
|
|
@gouthamp-stellar - I am trying to address multiple comments in this one message Re a few comments about - "how to get name of the token from the contract Id"..... Re:
the answer is yes. Re addition of custom token asset in the
I want to clarify a few things.
|
could use the IDL from the beginning in go, follow similar pattern as the XDR compilation, i.e use the IDL compiler(protoc) locally with an optional new make target, commit the generated stub code and IDL file(.proto) for the application events defined in the IDL to the repo, this should not introduce a dependency on protobufs toolchain to existing targets in go repo. Using IDL up front in this way may avoid potential breaking change later on the proposed derived application event models which are emitted by these TokenTransferProcessor functions, if we hand roll the interfaces/structs in go for the transfer events now, the IDL gen code done later won't be the same and would get complex to try to adapt into the manual defined artifacts, it looks like cdp and rpc could be used equally as XDR input sources for go TokenTransferProcessor functions.
The IDL for token events can evolve to wider usage externally in other languages, it would entail packaging the transform knowhow from XDR to these transfer events which lives in the go TokenTransferProcessor functions into a process which can run multi-platform, then the serialization aspects of IDL get used in distributed messages, it's not required yet, but something like:
|
custom token contracts don't have to wait for SEP-41 spec to stipulate this to include their token name in events payload/topics correct? in which case if the event schema in IDL has provision for it, the transfer processors could optimistically check for token name existence from custom contracts in XDR side and if present, then populate it on the transformed IDL event. |
There is nothing trustable about the name of custom tokens, there's no guarantee the names will be unique or a custom token won't impersonate a SAC contract. SAC emits the name in the event because of prior ecosystem dependence on those old names that have the asset code and issuer. For any new system where-ever possible they should ingest based on the contract ID, not the "asset code and issuer" to reference an asset. if they ingest based on contract ID they'll be ingesting a common interface across both SAC and custom tokens. |
should asset be removed from these transfer event models then, to avoid potential foot-gun, promotes the better practice of using contract id for resolving asset/token identification more-so? |
I don't think so because it is not possible to derive the stellar classic asset from a contract id. I think we should only include the asset details when the transfer events involve stellar classic assets. In the case of custom sep-41 tokens we should omit the asset field |
|
Re this comment
Had a chat with @gouthamp-stellar about it. The |
|
@leighmcculloch : Addressing your comments here Re:
Yes, currently there is no consumer for the protobuf. The wallet team, for whom we are building this token transfer processor, will be building their backend in golang and even if this With protos, in a way we solve a big part of the problem. The The invoker of the CLI doesnt have to worry about golang code or deal with nuances of I believe @sreuland is also alluding to the same sentiment in this message Re:
By additional work, all I meant is that we would have to run a make target to generate Re:
The wallet-team's (the first users of this token transfer library) codebase is in golang itself. If we do (a), and then decide to pivot to use (b) on a later date, then changes will have to be done in 2 places - in the processor code we write and in the downstream code that wallet team uses. Re:
I dont agree with the notion of introducing another mode of serialization as being a problematic change. Re:
There are a few problems with the premise that you state here. |
I think we can/should enable this, and it interplays with work we already expect/plan to do: #5571 |
Let's get very specific here. What is the additional data they need? It wouldn't be unreasonable to bring the discussion of this over to CAP-67 and see if those things could be added to the underlying data model. Is it just the name/contractId from the earlier discussion on this issue?
Core committed to retroactively emitting events for older protocol versions on reingestion (see here), so that does solve the problem of historical reingestion.
CDP/RPC are not mutually exclusive, and in fact, we'd like to enable RPC as a LCM source for CDP (see #5571). This means that for example, the wallet backend could use the |
I think it will be possible, but through multiple mechanisms. RPC has Then CAP-67 will be implemented. At this point, |
|
Re:
The plan, by wallet, atleast for now, is to pass LCMs sourced from RPC's getLedger endpoint and call the functions in tokenTransferProcessor. Re this comment and this comment, I am summarizing what was discussed in an internal meeting
|
|
I am marking this ticket as closed, since all the work/events/model has been captured here. What now remains to be done is to actually implement said library, as as a part of MVP-1, to generate tokenTransferEvents from operation and operationResults. I will be generating separate tickets to track that work It has been decided by the platform team to use protos for the data modelling. |
Issue #5573 talks about building a generic token transfer processor that hat parses the
LedgerCloseMetato generate a list ofTokenTransferEventevents that detail how balances have been updated for an entity (account or smart contract)The event representation/modeling in #5573, while detailed and descriptive, suffered from the following issues:
It was too detailed.
for e.g, it adds meta information about an operation if the transfer event was caused by an an operation. See example
TokenTransferDetailsfield forManageOfferBuyoperation in #5573A more concise way to represent an event would be to record the fact that asset movement has happened as a part of an operation by including
operationIndexin the event. Downstream users of this library can use the combination ofLedgerSequence,txHashandoperationIndexwith the transaction to get more detals about theoperationandoperationResult.It was very prescriptive in terms of what it defines as a transfer event.
For e.g, In the previous modeling proposal, a simple payment operation from
AccountXtoAccountYwould have resulted in 2 events being generated - acreditevent forAccountX, and adebitevent forAccountY.Moreover, both these events would have the same
SimplePaymentDetailsstruct included, leading to duplication.We should not be making any assumptions on what the downstream invokers of the processor library might choose to do with the
TokenTransferEvent, and therefore, it is better to represent a transfer of asset as{from: X, to: Y, amount: 123.4, asset: <someAsset> }The biggest issue with the data modeling in #5573 was that it was not complaint with CAP-67 - the unified classic events model as defined by Stellar-Core.
This has the potential to cause maintenance problems as more and more events are added in the future by the core team.
It does not make sense for a downstream processor library to have a data model different from how the core team defines it.
To remedy the issues above, this ticket defines a data model that is more aligned with CAP-67, so as to reduce operational/maintenance work for future evolutions
Addresstypes in eventsCurrently the
Addressfield, as it appears in events emiited by core, can be either:The
Addresswill be be extended to include ClaimableBalanceIds and LiquidityPoolHashes as well, once CAP-67 is implemented.Henceforth, Address can mean any of the 4 entities -
Events emitted by core
Based on CAP-46-6, there are 4 events currently emitted by core for SAC - Transfer, Mint, Burn, Clawback
** IMPORTANT NOTE **
Currently these events are emitted only for SAC. The purpose of CAP-67 is to generate the same types (and potentially newer events) for classic operations as well
Additionally, As a part of CAP-67, a new
Feeevent will be added1. Transfer Event --- existing event for SAC - to be extended to include classic operations in CAP-67
{from: Address, to: Address, sep0011_asset: String, amount: i128}- This event indicates transfer of asset from a source Address to a destination Address.2. Mint Event --- existing event for SAC - to be extended to include classic operations in CAP-67
{to: Address, sep0011_asset: String, amount: i128}- This represents a mint operation by the issuer account to a recipient3. Burn Event --- existing event for SAC - to be extended to include classic operations in CAP-67
{from: Address, admin: Address sep0011_asset: String, amount: i128}- This represents a burn operation where an asset is sent from an address back to the issuer, effectively removing it from circulation** Note: As a part of CAP-67, the
adminfield will be removed from burn event4. Clawback Event --- existing event for SAC - to be extended to include classic operations in CAP-67
{from: Address, admin: Address, sep0011_asset: String, amount: i128}- This represents a clawback operation issued by the owner (Admin) to claim back and burn money from an address.** Note: As a part of CAP-67, the
adminfield will be removed from clawback event5. Fee Event --- new event to be included in CAP-67
{from: Address, amount: i128}- This represents the transaction fee (in XLM) paid by the source account for each transaction.Basic Event Model
It makes sense to categorize events at the top level based on the 5 types of money movement - Transfer, Burn, Mint, Clawback, Fee , instead of based on reason - Fee, SmartContract, ClassicOperation as described in #5573
Downstream consumers can filter by this
eventTypefield in the event structure, when requiredCommon Fields to all Events
All events will, at the very minimum, have the following fields, irrespective of
type.These fields can be clubbed in the category of
EventMetaledgerSequence- The ledger sequence number in which this event occuredclosedAt- closing time of the ledger. This field is to associate a temporal quality with the event. For e.g, upstream clients might want to get events between 2 timestampstxHash- The hash of the transaction in which the event happenedoperationIndex- if the event was generated in response to an operation, this field will indicate the operation index within the transaction (zero-indexed). Otherwise this field will be absent/null (for e.g for fee events or smart contract events)So far, the event model looks like:
A sample
EventMeta(in json) might look like** Note: The above representation is merely a Golang Struct representation of what the
TokenTransferEventwould look like. Field names might changeFee Event
fromis the account paying the fee - either the fee bump fee account or the transaction source account.Classic Operations and Events
This section highlights the kind of events that are generated for each operation, along with examples wherever required.
The relevant fields in the events are represented as a JSON for clarity and brevity.
The final section will attempt to accumulate all the different kinds of events in a single data model/proto definition
For all classic operations, the following 2 fields will be present in the event itself.
amount- the total amount of the asset being movedasset- the canonical name of the classic asset in the format<AssetCode>:<Issuer>ornativefor XLMImportant Things to note:
fromandtofields might be either an account address or aliquidityPoolHashor aClaimableBalanceIdfromortocould be present in the event - for e.g mint and burn events** The
EventMetasection is skipped for illustration purposes. It will however be present in all events as mentioned in the section above.fromis the account being debited (creator).tois the account being credited (created).amountis the starting native balance (in XLM).fromis the account being debited (merged).tois the account being credited (merged into).amountis the merged native balance.** Note: you can only call Merge Account when the source account does not have any trustlines. So the asset when MergeAccount is called will always be native XLM
Suppose
AccountAis sendingUSDCtoAccountBAccountAnorAccountBis the issuer ofUSDC, then a Transfer event will be generatedAccountAhappens to be the issuer ofUSDC, then a Mint event would be generatedAccountBhappens to be the issuer ofUSDC, then a Burn event would be generatedfromis the account being debited.tois the claimable balance being created.amountis the amount moved into the claimable balancefromis the claimable balance id.tois the account being credited.amountis the amount moved from the CB to the accountfromis the account or CBamountis the amount being moved out of the account or CB and burned.Please refer to this section in CAP-67. that talks about creation of a claimable balance when any of these operations revokes authorization from a trustline that deposited into a liquidity pool.
fromis the LiquidityPool Hashtois the Claimable Balance being createdamountis the amount moved in tho the CBEach
LiquidityPoolDepositoperation will generate 2 transfer events - one for each assetfromis the account being debited.tois the liquidity pool being credited.amountis the amount moved into the liquidity pool.If the
fromaccount happens to be the issuer of one of the assets being moved in the LP, then aMintevent will be emitted instead of aTransferEach
LiquidityPoolWithdrawoperation will generate 2 transfer events - one for each assetfromis the liquidity pool.tois the account being credited.amountis the amount moved out of the liquidity pool.If the
toaccount happens to be the issuer of one of the assets being moved in the LP, then aBurnevent will be emitted instead of aTransferFor all these offer operations, the number of transfer events emitted depends on the number of trades.
In general,
Total Events = Trades * 2, since each trade consists of 2 transfers.If there were zero trades, i.e if the offer was not marketable, then no events will be emitted.
Consider the following example:
Suppose 3 sell offers of BTC-USDC exist as follows:
offerId1, sellerId = accountXofferId2, sellerId = accountYofferId3, sellerId = accountXNote that accountX has 2 different offers that are resting for the same trading pair
Given a
ManageBuyOperationoperation for BTC-USDC byAccountP, that fills against the 3 existing offers, a total of 6 transfer events will be generated as follows:fromaccount happens to be the issuer of one of the assets being traded, then aMintevent will be emitted instead of aTransfertoaccount happens to be the issuer of one of the assets being traded, then aBurnevent will be emitted instead of aTransferA path payment from
AccountAtoAccountBcan cross through either several offers from the different trading pairs, or through different Liquidity Pools, depending on the path specified in the operation.The trades within a path payment, regardless of whether it is against an offer or liquidity pool, can be conceptually thought of as a a transfer between the source account and the seller in case of offers or against the Liquidity pool.
In addition, there will be one final event indicating a transfer of asset between the source and final destination.
In general,
Total Events = (Trades * 2) + 1For e.g, a path payment operation where
AccountAwants to send BTC and wantsAccountBto receive ETH with a path[USDC], and suppose it trades against 2 Liquidity pools -LP-BTC-USDCandLP-USDC-ETH, the following 5 events will be emitted:sendAsset == destAsset, then the operation is effectively a simple payment.fromaccount happens to be the issuer of one of the assets being traded, then aMintevent will be emitted instead of aTransfertoaccount happens to be the issuer of one of the assets being traded, then aBurnevent will be emitted instead of aTransferStellar Asset Contract (SAC) Events
CAP-46-6(https://github.com/stellar/stellar-protocol/blob/master/core/cap-0046-06.md) defines the interface followed by smart contracts that want to interact with classic tokens.
It also defines the events that will be emitted when asset movement happens.
SEP-41 Compliant Custom Tokens and Events
Smart contracts implementing SEP-41 and custom tokens will emit the following events that will be read by this processor library and surfaced to callers.
NOTE: The events generated by SEP-41 compliant custom contracts/tokens doesnt emit the asset. It is expected that downstream clients (for e.g Freighter) can identify the custom token being moved externally
Sample Proto Definition for events
Putting it all together, the protobuf definition for the
TokenTransferEventwill look something like this:EDIT: This protobuf definition has been updated to reflect this comment
TokenTransferProcessorand Downstream Usage by ClientsThe goal of the
TokenTransferProcessorlibrary/package is to provide a list ofTokenTransferEventsfrom a given ledger, transaction, operation.If downstream clients need further introspection/details, other than the fields present in the
TokenTransfer- for e.g more information about smart contracts/operation and result, the clients should be able to derive it themselves using the other helper libraries/helper functions from the ingest packageThe proposal is to provide implementation of 3 functions that can be used by downstream clients:
A downstream client, depending on usecase, can invoke these functions in this manner:
The text was updated successfully, but these errors were encountered: