-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement user API for wildcard indexing #171
Conversation
d0e88f3
to
6175f23
Compare
module SingleOrMultiple: { | ||
@genType.import(("./bindings/OpaqueTypes", "SingleOrMultiple")) | ||
type t<'a> | ||
let normalizeOrThrow: (t<'a>, ~nestedArrayDepth: int=?) => array<'a> | ||
let single: 'a => t<'a> | ||
let multiple: array<'a> => t<'a> | ||
} = { | ||
type t<'a> = Js.Json.t | ||
|
||
external single: 'a => t<'a> = "%identity" | ||
external multiple: array<'a> => t<'a> = "%identity" | ||
external castMultiple: t<'a> => array<'a> = "%identity" | ||
external castSingle: t<'a> => 'a = "%identity" | ||
|
||
exception AmbiguousEmptyNestedArray | ||
|
||
let rec isMultiple = (t: t<'a>, ~nestedArrayDepth): bool => | ||
switch t->Js.Json.decodeArray { | ||
| None => false | ||
| Some(_arr) if nestedArrayDepth == 0 => true | ||
| Some([]) if nestedArrayDepth > 0 => | ||
AmbiguousEmptyNestedArray->ErrorHandling.mkLogAndRaise( | ||
~msg="The given empty array could be interperated as a flat array (value) or nested array. Since it's ambiguous, | ||
please pass in a nested empty array if the intention is to provide an empty array as a value", | ||
) | ||
| Some(arr) => arr->Js.Array2.unsafe_get(0)->isMultiple(~nestedArrayDepth=nestedArrayDepth - 1) | ||
} | ||
|
||
let normalizeOrThrow = (t: t<'a>, ~nestedArrayDepth=0): array<'a> => { | ||
if t->isMultiple(~nestedArrayDepth) { | ||
t->castMultiple | ||
} else { | ||
[t->castSingle] | ||
} | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was the best way I could think of for generalising a type that can be an array or a value.
Unboxed types don't work with polymorphic base types in rescript and the runtime check for whether it is an array would not suffice for nested arrays.
type loaderHandler<'eventArgs, 'loaderReturn, 'eventFilter> = { | ||
loader: loader<'eventArgs, 'loaderReturn>, | ||
handler: handler<'eventArgs, 'loaderReturn>, | ||
wildcard?: bool, | ||
eventFilters?: SingleOrMultiple.t<'eventFilter>, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't able to use the optional second argument trick for gentype for the loaderWithHandler registration function. The problem was because 'loaderReturn remains a polymorphic parameter and using genType.import for the function signature wouldn't allow me to have a scoped polymorphic type. So basically I thought it's better to just extend the options into this structure.
Let me know if that makes sense.
let make = (~isWildcard, ~topicSelections: array<LogSelection.topicSelection>) => { | ||
let topic0sGrouped = [] | ||
let topicSelectionWithFilters = [] | ||
topicSelections->Belt.Array.forEach(ts => | ||
if ts->LogSelection.hasFilters { | ||
topicSelectionWithFilters->Js.Array2.push(ts)->ignore | ||
} else { | ||
ts.topic0->Belt.Array.forEach(topic0 => { | ||
topic0sGrouped->Js.Array2.push(topic0)->ignore | ||
}) | ||
} | ||
) | ||
let topicSelectionNoFilters = | ||
LogSelection.makeTopicSelection(~topic0=topic0sGrouped)->Utils.unwrapResultExn | ||
|
||
{ | ||
isWildcard, | ||
topicSelections: Belt.Array.concat(topicSelectionWithFilters, [topicSelectionNoFilters]), | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This constructor allows you to group topics that only have a topic0 together like they were before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to do this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to do this?
Yes, well it's better if it happens at the start on registration than if we build it in every query.
For instance instead of a query like:
[
{
"addresses": [...manyAddresses
],
"topics": [
[
"0x1"
],
[],
[],
[]
]
},
{
"addresses": [...manyAddresses
],
"topics": [
[
"0x2"
],
[],
[],
[]
]
}
]
it is flattened to this:
[
{
"addresses": [...manyAddresses
],
"topics": [
[
"0x1",
"0x2"
],
[],
[],
[]
]
},
]
@genType.import(("./bindings/OpaqueTypes.ts", "HandlerWithOptions")) | ||
type fnWithEventConfig<'fn, 'eventConfig> = ('fn, ~eventConfig: 'eventConfig=?) => unit | ||
|
||
@genType | ||
type handlerWithOptions<'eventArgs, 'loaderReturn, 'eventFilter> = fnWithEventConfig< | ||
HandlerTypes.handler<'eventArgs, 'loaderReturn>, | ||
HandlerTypes.eventConfig<'eventFilter>, | ||
> | ||
|
||
@genType | ||
type contractRegisterWithOptions<'eventArgs, 'eventFilter> = fnWithEventConfig< | ||
HandlerTypes.contractRegister<'eventArgs>, | ||
HandlerTypes.eventConfig<'eventFilter>, | ||
> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the trick I used to allow for an optional second argument in typescript. Only works for "handler" and "contractRegister"
module Hex = { | ||
type t | ||
external fromStringUnsafe: string => t = "%identity" | ||
external fromStringsUnsafe: array<string> => array<t> = "%identity" | ||
external toString: t => string = "%identity" | ||
external toStrings: array<t> => array<string> = "%identity" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought it might be nice to have a dedicated module for parsed types that can be shared between library bindings etc.
pub fn generate_get_topic_selection_code(params: &Vec<EventParam>) -> String { | ||
let indexed_params = params.iter().filter(|param| param.indexed); | ||
|
||
//Prefixed with underscore for cases where it is not used to avoid compiler warnings | ||
let event_filter_arg = "_eventFilter"; | ||
|
||
let topic_filter_calls = indexed_params | ||
.enumerate() | ||
.map(|(i, param)| { | ||
let param = EthereumEventParam::from(param); | ||
let topic_number = i + 1; | ||
let param_name = RescriptRecordField::to_valid_res_name(param.name); | ||
let topic_encoder = param.get_topic_encoder(); | ||
let nested_type_flags = match param.get_nested_type_depth() { | ||
depth if depth > 0 => format!("(~nestedArrayDepth={depth})"), | ||
_ => "".to_string(), | ||
}; | ||
format!( | ||
"~topic{topic_number}=?{event_filter_arg}.{param_name}->Belt.Option.\ | ||
map(topicFilters => \ | ||
topicFilters->SingleOrMultiple.normalizeOrThrow{nested_type_flags}->Belt.\ | ||
Array.map({topic_encoder})), " | ||
) | ||
}) | ||
.collect::<String>(); | ||
|
||
format!( | ||
"(eventFilters) => \ | ||
eventFilters->SingleOrMultiple.normalizeOrThrow->Belt.Array.map({event_filter_arg} \ | ||
=> LogSelection.makeTopicSelection(~topic0=[sighash->EvmTypes.Hex.fromStringUnsafe], \ | ||
{topic_filter_calls})->Utils.unwrapResultExn)" | ||
) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is completely covered in test_codegen test but I haven't written any unit tests on this directly in the rust codebase.
let toTwosComplement = (num: bigint, ~bytesLen: int) => { | ||
let maxValue = 1n->BigInt.Bitwise.shift_left(BigInt.fromInt(bytesLen * 8)) | ||
let mask = maxValue->BigInt.sub(1n) | ||
num->BigInt.add(maxValue)->BigInt.Bitwise.logand(mask) | ||
} | ||
|
||
let fromSignedBigInt = val => { | ||
let bytesLen = 32 | ||
let val = val >= 0n ? val : val->toTwosComplement(~bytesLen) | ||
val->Viem.bigintToHex(~options={size: bytesLen}) | ||
} | ||
|
||
type hex = EvmTypes.Hex.t | ||
//bytes currently does not work with genType and we also currently generate bytes as a string type | ||
type bytesHex = string | ||
let keccak256 = Viem.keccak256 | ||
let bytesToHex = Viem.bytesToHex | ||
let concat = Viem.concat | ||
let castToHexUnsafe: 'a => hex = val => val->Utils.magic | ||
let fromBigInt: bigint => hex = val => val->Viem.bigintToHex(~options={size: 32}) | ||
let fromDynamicString: string => hex = val => val->(Utils.magic: string => hex)->keccak256 | ||
let fromString: string => hex = val => val->Viem.stringToHex(~options={size: 32}) | ||
let fromAddress: Address.t => hex = addr => addr->(Utils.magic: Address.t => hex)->Viem.pad | ||
let fromDynamicBytes: bytesHex => hex = bytes => bytes->(Utils.magic: bytesHex => hex)->keccak256 | ||
let fromBytes: bytesHex => hex = bytes => | ||
bytes->(Utils.magic: bytesHex => bytes)->Viem.bytesToHex(~options={size: 32}) | ||
let fromBool: bool => hex = b => b->Viem.boolToHex(~options={size: 32}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A collection of TopicFilter functions that are combined when generating topic filters for different types
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can simplify it and do use something like this instead https://viem.sh/docs/contract/encodeEventTopics.html ?
Also, I'd like to move as much logic as possible from codegen. So doing encoding with Viem instead of selecting the right encoder in Rust, will make the envio closer to the library-like state
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not a bad approach. I'm not sure of it's limitations though. I would still need to generate each of those selectors. And it doesn't line up exactly with what we are supporting with our API. We're allowing multiple sets of topic1-3.
I personally prefer this as is right now. I can swap out my own hex encoding functions from viem or use another library and everthing works. No need to rely on viem to give as a full implementation.
I think for user API we still want to control that with codegen. For a library implementation I think they would just encode the topics themselves and pass it in.
let wildcardTopicSelection = T.contracts->Belt.Array.flatMap(contract => { | ||
contract.events->Belt.Array.keepMap(event => { | ||
let module(Event) = event | ||
let {isWildcard, topicSelections} = | ||
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions | ||
isWildcard ? Some(LogSelection.make(~addresses=[], ~topicSelections)) : None | ||
}) | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wildcard topic selections can created globally when instantiating a hypersync worker
T.contracts | ||
->Belt.Array.keepMap((contract): option<LogSelection.t> => { | ||
switch contractAddressMapping->ContractAddressingMap.getAddressesFromContractName( | ||
~contractName=contract.name, | ||
) { | ||
| [] => None | ||
| addresses => | ||
let topicSelection = | ||
LogSelection.makeTopicSelection(~topic0=contract.sighashes)->Utils.unwrapResultExn | ||
|
||
Some(LogSelection.make(~addresses, ~topicSelections=[topicSelection])) | ||
switch contract.events->Belt.Array.flatMap(event => { | ||
let module(Event) = event | ||
let {isWildcard, topicSelections} = | ||
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions | ||
|
||
isWildcard ? [] : topicSelections | ||
}) { | ||
| [] => None | ||
| topicSelections => Some(LogSelection.make(~addresses, ~topicSelections)) | ||
} | ||
} | ||
}) | ||
->Array.concat(wildcardTopicSelection) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non wildcard events need to generate their topic selections with the addresses passed in and wildcardTopicSelection is concatenated to these selections.
handleDecodeFailure( | ||
~eventMod, | ||
~decoder="hypersync-client", | ||
~logIndex=log.logIndex, | ||
~blockNumber=block.number, | ||
~chainId, | ||
~exn=UndefinedValue, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's now valid for a decode failure on wildcard events.
T.contracts->Belt.Array.forEach(contract => { | ||
contract.events->Belt.Array.forEach(event => { | ||
let module(Event) = event | ||
let {isWildcard, topicSelections} = | ||
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions | ||
|
||
let logger = Logging.createChild( | ||
~params={ | ||
"chainId": T.chain->ChainMap.Chain.toChainId, | ||
"contractName": contract.name, | ||
"eventName": Event.name, | ||
}, | ||
) | ||
if isWildcard { | ||
%raw(`null`)->ErrorHandling.mkLogAndRaise( | ||
~msg="RPC worker does not yet support wildcard events", | ||
~logger, | ||
) | ||
} | ||
|
||
topicSelections->Belt.Array.forEach( | ||
topicSelection => { | ||
if topicSelection->LogSelection.hasFilters { | ||
%raw(`null`)->ErrorHandling.mkLogAndRaise( | ||
~msg="RPC worker does not yet support event filters", | ||
~logger, | ||
) | ||
} | ||
}, | ||
) | ||
}) | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rpc doesn't support these filters yet.
type hex = string | ||
@unboxed | ||
type topicFilter = Single(hex) | Multiple(array<hex>) | @as(undefined) Undefined | ||
type topicQuery = array<topicFilter> | ||
let makeTopicQuery = (~topic0=[], ~topic1=[], ~topic2=[], ~topic3=[]) => { | ||
let topics = [topic0, topic1, topic2, topic3] | ||
|
||
let isLastTopicEmpty = () => | ||
switch topics->Utils.Array.last { | ||
| Some([]) => true | ||
| _ => false | ||
} | ||
|
||
//Remove all empty topics from the end of the array | ||
while isLastTopicEmpty() { | ||
topics->Js.Array2.pop->ignore | ||
} | ||
|
||
let toTopicFilter = topic => { | ||
switch topic { | ||
| [] => Undefined | ||
| [single] => Single(single->EvmTypes.Hex.toString) | ||
| multiple => Multiple(multiple->EvmTypes.Hex.toStrings) | ||
} | ||
} | ||
|
||
topics->Belt.Array.map(toTopicFilter) | ||
} | ||
|
||
let mapTopicQuery = ({topic0, topic1, topic2, topic3}: LogSelection.topicSelection): topicQuery => | ||
makeTopicQuery(~topic0, ~topic1, ~topic2, ~topic3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently just used in tests but I'll use this to implement the wildcard queries for RPC later on.
1bd7987
to
ffd02b9
Compare
describe("Single or Multiple", () => { | ||
it("Single nested", () => { | ||
let tupleWithArrays: tupleWithArrays = ([1n, 2n], ["test", "test2"]) | ||
let single: SingleOrMultiple.t<tupleWithArrays> = SingleOrMultiple.single(tupleWithArrays) | ||
let multiple: SingleOrMultiple.t<tupleWithArrays> = SingleOrMultiple.multiple([tupleWithArrays]) | ||
|
||
let expectedNormalized = [tupleWithArrays] | ||
|
||
let normalizedSingle = SingleOrMultiple.normalizeOrThrow(single, ~nestedArrayDepth=2) | ||
let normalizedMultiple = SingleOrMultiple.normalizeOrThrow(multiple, ~nestedArrayDepth=2) | ||
|
||
Assert.deepEqual( | ||
multiple->Utils.magic, | ||
expectedNormalized, | ||
~message="Multiple should be the same as normalized", | ||
) | ||
Assert.deepEqual(normalizedSingle, expectedNormalized, ~message="Single should be normalized") | ||
Assert.deepEqual( | ||
normalizedMultiple, | ||
expectedNormalized, | ||
~message="Multiple should be normalized", | ||
) | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've only tested one case here where I was initially getting a failure in my implementation. But it's tested in many ways with the integration test of the generated code.
const checkEventFilter = async (eventMod: any, filter: any) => { | ||
const topics = mapTopicQuery(eventMod.getTopicSelection(filter)[0]); | ||
const res = await hre.ethers.provider.getLogs({ | ||
address: await deployedTestEvents.getAddress(), | ||
topics, | ||
}); | ||
|
||
assert.equal(res.length, 1); | ||
checkedEvents[eventMod.name] = eventMod.sighash; | ||
}; | ||
|
||
it("Gets indexed uint topic with topic filter", async () => { | ||
const filter: TestEvents_IndexedUint_eventFilter = { num: [testParams.id] }; | ||
await checkEventFilter(GeneratedTestEvents.IndexedUint, filter); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test checks that the topic encoders work and correspond with the evm hashes of the deployed contract. It covers all cases of data structures that have different encoders.
298fcff
to
513fa6e
Compare
a6974e5
to
7c9f0a7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Amazing work 🔥 I'm just a little bit sad that it'll break the Fuel version a little bit, but I'll adjust it later. Also, I think wildcard indexing should be quite simple to implement for Fuel 🤔
One comment after the review is to move the encoder selection to the runtime code, if this is possible. And ideally even move it to the ChainWorker module
let make = (~isWildcard, ~topicSelections: array<LogSelection.topicSelection>) => { | ||
let topic0sGrouped = [] | ||
let topicSelectionWithFilters = [] | ||
topicSelections->Belt.Array.forEach(ts => | ||
if ts->LogSelection.hasFilters { | ||
topicSelectionWithFilters->Js.Array2.push(ts)->ignore | ||
} else { | ||
ts.topic0->Belt.Array.forEach(topic0 => { | ||
topic0sGrouped->Js.Array2.push(topic0)->ignore | ||
}) | ||
} | ||
) | ||
let topicSelectionNoFilters = | ||
LogSelection.makeTopicSelection(~topic0=topic0sGrouped)->Utils.unwrapResultExn | ||
|
||
{ | ||
isWildcard, | ||
topicSelections: Belt.Array.concat(topicSelectionWithFilters, [topicSelectionNoFilters]), | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to do this?
let toTwosComplement = (num: bigint, ~bytesLen: int) => { | ||
let maxValue = 1n->BigInt.Bitwise.shift_left(BigInt.fromInt(bytesLen * 8)) | ||
let mask = maxValue->BigInt.sub(1n) | ||
num->BigInt.add(maxValue)->BigInt.Bitwise.logand(mask) | ||
} | ||
|
||
let fromSignedBigInt = val => { | ||
let bytesLen = 32 | ||
let val = val >= 0n ? val : val->toTwosComplement(~bytesLen) | ||
val->Viem.bigintToHex(~options={size: bytesLen}) | ||
} | ||
|
||
type hex = EvmTypes.Hex.t | ||
//bytes currently does not work with genType and we also currently generate bytes as a string type | ||
type bytesHex = string | ||
let keccak256 = Viem.keccak256 | ||
let bytesToHex = Viem.bytesToHex | ||
let concat = Viem.concat | ||
let castToHexUnsafe: 'a => hex = val => val->Utils.magic | ||
let fromBigInt: bigint => hex = val => val->Viem.bigintToHex(~options={size: 32}) | ||
let fromDynamicString: string => hex = val => val->(Utils.magic: string => hex)->keccak256 | ||
let fromString: string => hex = val => val->Viem.stringToHex(~options={size: 32}) | ||
let fromAddress: Address.t => hex = addr => addr->(Utils.magic: Address.t => hex)->Viem.pad | ||
let fromDynamicBytes: bytesHex => hex = bytes => bytes->(Utils.magic: bytesHex => hex)->keccak256 | ||
let fromBytes: bytesHex => hex = bytes => | ||
bytes->(Utils.magic: bytesHex => bytes)->Viem.bytesToHex(~options={size: 32}) | ||
let fromBool: bool => hex = b => b->Viem.boolToHex(~options={size: 32}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can simplify it and do use something like this instead https://viem.sh/docs/contract/encodeEventTopics.html ?
Also, I'd like to move as much logic as possible from codegen. So doing encoding with Viem instead of selecting the right encoder in Rust, will make the envio closer to the library-like state
switch contract.events->Belt.Array.flatMap(event => { | ||
let module(Event) = event | ||
let {isWildcard, topicSelections} = | ||
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions | ||
|
||
isWildcard ? [] : topicSelections | ||
}) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A nitpick, but we can also precalculate the topicSelections on the ChainWorker creation and only join them with addresses here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes but we can only apply those based on the addresses in the contractAddressMapping. So we could create a premade lookup table essentially of contract->topic selection and on each contract in the contractAddressMapping make a LogSelection with the topics and addresses. That is not far off from what's happenening here but we could probably improve it slightly 👍🏼
@@ -0,0 +1,4 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like it should be ignored
TODO: