Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement user API for wildcard indexing #171

Merged
merged 23 commits into from
Sep 11, 2024
Merged

Conversation

JonoPrest
Copy link
Collaborator

@JonoPrest JonoPrest commented Sep 3, 2024

  • Adds types and api for user to configure event filters
  • Adds tests for all combination of indexed types and filtering

TODO:

  • Make each field optional for filters
  • Make a filter type able to take an array or a single item
  • Figure out encoding of a negative number for topic filter
  • Connect the configured user options to hypersync query

@JonoPrest JonoPrest requested review from DZakh and removed request for DZakh September 3, 2024 15:51
@JonoPrest JonoPrest force-pushed the jp/wildcard-user-api branch 2 times, most recently from d0e88f3 to 6175f23 Compare September 9, 2024 13:44
Comment on lines +205 to +241
module SingleOrMultiple: {
@genType.import(("./bindings/OpaqueTypes", "SingleOrMultiple"))
type t<'a>
let normalizeOrThrow: (t<'a>, ~nestedArrayDepth: int=?) => array<'a>
let single: 'a => t<'a>
let multiple: array<'a> => t<'a>
} = {
type t<'a> = Js.Json.t

external single: 'a => t<'a> = "%identity"
external multiple: array<'a> => t<'a> = "%identity"
external castMultiple: t<'a> => array<'a> = "%identity"
external castSingle: t<'a> => 'a = "%identity"

exception AmbiguousEmptyNestedArray

let rec isMultiple = (t: t<'a>, ~nestedArrayDepth): bool =>
switch t->Js.Json.decodeArray {
| None => false
| Some(_arr) if nestedArrayDepth == 0 => true
| Some([]) if nestedArrayDepth > 0 =>
AmbiguousEmptyNestedArray->ErrorHandling.mkLogAndRaise(
~msg="The given empty array could be interperated as a flat array (value) or nested array. Since it's ambiguous,
please pass in a nested empty array if the intention is to provide an empty array as a value",
)
| Some(arr) => arr->Js.Array2.unsafe_get(0)->isMultiple(~nestedArrayDepth=nestedArrayDepth - 1)
}

let normalizeOrThrow = (t: t<'a>, ~nestedArrayDepth=0): array<'a> => {
if t->isMultiple(~nestedArrayDepth) {
t->castMultiple
} else {
[t->castSingle]
}
}
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the best way I could think of for generalising a type that can be an array or a value.

Unboxed types don't work with polymorphic base types in rescript and the runtime check for whether it is an array would not suffice for nested arrays.

Comment on lines +270 to 275
type loaderHandler<'eventArgs, 'loaderReturn, 'eventFilter> = {
loader: loader<'eventArgs, 'loaderReturn>,
handler: handler<'eventArgs, 'loaderReturn>,
wildcard?: bool,
eventFilters?: SingleOrMultiple.t<'eventFilter>,
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to use the optional second argument trick for gentype for the loaderWithHandler registration function. The problem was because 'loaderReturn remains a polymorphic parameter and using genType.import for the function signature wouldn't allow me to have a scoped polymorphic type. So basically I thought it's better to just extend the options into this structure.

Let me know if that makes sense.

Comment on lines +294 to 314
let make = (~isWildcard, ~topicSelections: array<LogSelection.topicSelection>) => {
let topic0sGrouped = []
let topicSelectionWithFilters = []
topicSelections->Belt.Array.forEach(ts =>
if ts->LogSelection.hasFilters {
topicSelectionWithFilters->Js.Array2.push(ts)->ignore
} else {
ts.topic0->Belt.Array.forEach(topic0 => {
topic0sGrouped->Js.Array2.push(topic0)->ignore
})
}
)
let topicSelectionNoFilters =
LogSelection.makeTopicSelection(~topic0=topic0sGrouped)->Utils.unwrapResultExn

{
isWildcard,
topicSelections: Belt.Array.concat(topicSelectionWithFilters, [topicSelectionNoFilters]),
}
}
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This constructor allows you to group topics that only have a topic0 together like they were before.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do this?

Copy link
Collaborator Author

@JonoPrest JonoPrest Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do this?

Yes, well it's better if it happens at the start on registration than if we build it in every query.

For instance instead of a query like:

[
  {
    "addresses": [...manyAddresses
    ],
    "topics": [
      [
        "0x1"
      ],
      [],
      [],
      []
    ]
  },
  {
    "addresses": [...manyAddresses
    ],
    "topics": [
      [
        "0x2"
      ],
      [],
      [],
      []
    ]
  }
]

it is flattened to this:

[
  {
    "addresses": [...manyAddresses
    ],
    "topics": [
      [
        "0x1",
        "0x2"
      ],
      [],
      [],
      []
    ]
  },
]

Comment on lines +526 to +539
@genType.import(("./bindings/OpaqueTypes.ts", "HandlerWithOptions"))
type fnWithEventConfig<'fn, 'eventConfig> = ('fn, ~eventConfig: 'eventConfig=?) => unit

@genType
type handlerWithOptions<'eventArgs, 'loaderReturn, 'eventFilter> = fnWithEventConfig<
HandlerTypes.handler<'eventArgs, 'loaderReturn>,
HandlerTypes.eventConfig<'eventFilter>,
>

@genType
type contractRegisterWithOptions<'eventArgs, 'eventFilter> = fnWithEventConfig<
HandlerTypes.contractRegister<'eventArgs>,
HandlerTypes.eventConfig<'eventFilter>,
>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the trick I used to allow for an optional second argument in typescript. Only works for "handler" and "contractRegister"

Comment on lines +1 to +7
module Hex = {
type t
external fromStringUnsafe: string => t = "%identity"
external fromStringsUnsafe: array<string> => array<t> = "%identity"
external toString: t => string = "%identity"
external toStrings: array<t> => array<string> = "%identity"
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it might be nice to have a dedicated module for parsed types that can be shared between library bindings etc.

Comment on lines +378 to +410
pub fn generate_get_topic_selection_code(params: &Vec<EventParam>) -> String {
let indexed_params = params.iter().filter(|param| param.indexed);

//Prefixed with underscore for cases where it is not used to avoid compiler warnings
let event_filter_arg = "_eventFilter";

let topic_filter_calls = indexed_params
.enumerate()
.map(|(i, param)| {
let param = EthereumEventParam::from(param);
let topic_number = i + 1;
let param_name = RescriptRecordField::to_valid_res_name(param.name);
let topic_encoder = param.get_topic_encoder();
let nested_type_flags = match param.get_nested_type_depth() {
depth if depth > 0 => format!("(~nestedArrayDepth={depth})"),
_ => "".to_string(),
};
format!(
"~topic{topic_number}=?{event_filter_arg}.{param_name}->Belt.Option.\
map(topicFilters => \
topicFilters->SingleOrMultiple.normalizeOrThrow{nested_type_flags}->Belt.\
Array.map({topic_encoder})), "
)
})
.collect::<String>();

format!(
"(eventFilters) => \
eventFilters->SingleOrMultiple.normalizeOrThrow->Belt.Array.map({event_filter_arg} \
=> LogSelection.makeTopicSelection(~topic0=[sighash->EvmTypes.Hex.fromStringUnsafe], \
{topic_filter_calls})->Utils.unwrapResultExn)"
)
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is completely covered in test_codegen test but I haven't written any unit tests on this directly in the rust codebase.

Comment on lines +1 to +27
let toTwosComplement = (num: bigint, ~bytesLen: int) => {
let maxValue = 1n->BigInt.Bitwise.shift_left(BigInt.fromInt(bytesLen * 8))
let mask = maxValue->BigInt.sub(1n)
num->BigInt.add(maxValue)->BigInt.Bitwise.logand(mask)
}

let fromSignedBigInt = val => {
let bytesLen = 32
let val = val >= 0n ? val : val->toTwosComplement(~bytesLen)
val->Viem.bigintToHex(~options={size: bytesLen})
}

type hex = EvmTypes.Hex.t
//bytes currently does not work with genType and we also currently generate bytes as a string type
type bytesHex = string
let keccak256 = Viem.keccak256
let bytesToHex = Viem.bytesToHex
let concat = Viem.concat
let castToHexUnsafe: 'a => hex = val => val->Utils.magic
let fromBigInt: bigint => hex = val => val->Viem.bigintToHex(~options={size: 32})
let fromDynamicString: string => hex = val => val->(Utils.magic: string => hex)->keccak256
let fromString: string => hex = val => val->Viem.stringToHex(~options={size: 32})
let fromAddress: Address.t => hex = addr => addr->(Utils.magic: Address.t => hex)->Viem.pad
let fromDynamicBytes: bytesHex => hex = bytes => bytes->(Utils.magic: bytesHex => hex)->keccak256
let fromBytes: bytesHex => hex = bytes =>
bytes->(Utils.magic: bytesHex => bytes)->Viem.bytesToHex(~options={size: 32})
let fromBool: bool => hex = b => b->Viem.boolToHex(~options={size: 32})
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A collection of TopicFilter functions that are combined when generating topic filters for different types

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can simplify it and do use something like this instead https://viem.sh/docs/contract/encodeEventTopics.html ?

Also, I'd like to move as much logic as possible from codegen. So doing encoding with Viem instead of selecting the right encoder in Rust, will make the envio closer to the library-like state

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a bad approach. I'm not sure of it's limitations though. I would still need to generate each of those selectors. And it doesn't line up exactly with what we are supporting with our API. We're allowing multiple sets of topic1-3.

I personally prefer this as is right now. I can swap out my own hex encoding functions from viem or use another library and everthing works. No need to rely on viem to give as a full implementation.

I think for user API we still want to control that with codegen. For a library implementation I think they would just encode the topics themselves and pass it in.

Comment on lines +113 to +120
let wildcardTopicSelection = T.contracts->Belt.Array.flatMap(contract => {
contract.events->Belt.Array.keepMap(event => {
let module(Event) = event
let {isWildcard, topicSelections} =
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions
isWildcard ? Some(LogSelection.make(~addresses=[], ~topicSelections)) : None
})
})
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wildcard topic selections can created globally when instantiating a hypersync worker

Comment on lines +123 to +142
T.contracts
->Belt.Array.keepMap((contract): option<LogSelection.t> => {
switch contractAddressMapping->ContractAddressingMap.getAddressesFromContractName(
~contractName=contract.name,
) {
| [] => None
| addresses =>
let topicSelection =
LogSelection.makeTopicSelection(~topic0=contract.sighashes)->Utils.unwrapResultExn

Some(LogSelection.make(~addresses, ~topicSelections=[topicSelection]))
switch contract.events->Belt.Array.flatMap(event => {
let module(Event) = event
let {isWildcard, topicSelections} =
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions

isWildcard ? [] : topicSelections
}) {
| [] => None
| topicSelections => Some(LogSelection.make(~addresses, ~topicSelections))
}
}
})
->Array.concat(wildcardTopicSelection)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non wildcard events need to generate their topic selections with the addresses passed in and wildcardTopicSelection is concatenated to these selections.

Comment on lines +400 to +406
handleDecodeFailure(
~eventMod,
~decoder="hypersync-client",
~logIndex=log.logIndex,
~blockNumber=block.number,
~chainId,
~exn=UndefinedValue,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's now valid for a decode failure on wildcard events.

Comment on lines +12 to +43
T.contracts->Belt.Array.forEach(contract => {
contract.events->Belt.Array.forEach(event => {
let module(Event) = event
let {isWildcard, topicSelections} =
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions

let logger = Logging.createChild(
~params={
"chainId": T.chain->ChainMap.Chain.toChainId,
"contractName": contract.name,
"eventName": Event.name,
},
)
if isWildcard {
%raw(`null`)->ErrorHandling.mkLogAndRaise(
~msg="RPC worker does not yet support wildcard events",
~logger,
)
}

topicSelections->Belt.Array.forEach(
topicSelection => {
if topicSelection->LogSelection.hasFilters {
%raw(`null`)->ErrorHandling.mkLogAndRaise(
~msg="RPC worker does not yet support event filters",
~logger,
)
}
},
)
})
})
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rpc doesn't support these filters yet.

Comment on lines +1 to +31
type hex = string
@unboxed
type topicFilter = Single(hex) | Multiple(array<hex>) | @as(undefined) Undefined
type topicQuery = array<topicFilter>
let makeTopicQuery = (~topic0=[], ~topic1=[], ~topic2=[], ~topic3=[]) => {
let topics = [topic0, topic1, topic2, topic3]

let isLastTopicEmpty = () =>
switch topics->Utils.Array.last {
| Some([]) => true
| _ => false
}

//Remove all empty topics from the end of the array
while isLastTopicEmpty() {
topics->Js.Array2.pop->ignore
}

let toTopicFilter = topic => {
switch topic {
| [] => Undefined
| [single] => Single(single->EvmTypes.Hex.toString)
| multiple => Multiple(multiple->EvmTypes.Hex.toStrings)
}
}

topics->Belt.Array.map(toTopicFilter)
}

let mapTopicQuery = ({topic0, topic1, topic2, topic3}: LogSelection.topicSelection): topicQuery =>
makeTopicQuery(~topic0, ~topic1, ~topic2, ~topic3)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently just used in tests but I'll use this to implement the wildcard queries for RPC later on.

Comment on lines +5 to +27
describe("Single or Multiple", () => {
it("Single nested", () => {
let tupleWithArrays: tupleWithArrays = ([1n, 2n], ["test", "test2"])
let single: SingleOrMultiple.t<tupleWithArrays> = SingleOrMultiple.single(tupleWithArrays)
let multiple: SingleOrMultiple.t<tupleWithArrays> = SingleOrMultiple.multiple([tupleWithArrays])

let expectedNormalized = [tupleWithArrays]

let normalizedSingle = SingleOrMultiple.normalizeOrThrow(single, ~nestedArrayDepth=2)
let normalizedMultiple = SingleOrMultiple.normalizeOrThrow(multiple, ~nestedArrayDepth=2)

Assert.deepEqual(
multiple->Utils.magic,
expectedNormalized,
~message="Multiple should be the same as normalized",
)
Assert.deepEqual(normalizedSingle, expectedNormalized, ~message="Single should be normalized")
Assert.deepEqual(
normalizedMultiple,
expectedNormalized,
~message="Multiple should be normalized",
)
})
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only tested one case here where I was initially getting a failure in my implementation. But it's tested in many ways with the integration test of the generated code.

Comment on lines +71 to +85
const checkEventFilter = async (eventMod: any, filter: any) => {
const topics = mapTopicQuery(eventMod.getTopicSelection(filter)[0]);
const res = await hre.ethers.provider.getLogs({
address: await deployedTestEvents.getAddress(),
topics,
});

assert.equal(res.length, 1);
checkedEvents[eventMod.name] = eventMod.sighash;
};

it("Gets indexed uint topic with topic filter", async () => {
const filter: TestEvents_IndexedUint_eventFilter = { num: [testParams.id] };
await checkEventFilter(GeneratedTestEvents.IndexedUint, filter);
});
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test checks that the topic encoders work and correspond with the evm hashes of the deployed contract. It covers all cases of data structures that have different encoders.

@JonoPrest JonoPrest marked this pull request as ready for review September 9, 2024 16:41
@JonoPrest JonoPrest requested a review from DZakh September 9, 2024 16:49
Copy link
Member

@DZakh DZakh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work 🔥 I'm just a little bit sad that it'll break the Fuel version a little bit, but I'll adjust it later. Also, I think wildcard indexing should be quite simple to implement for Fuel 🤔

One comment after the review is to move the encoder selection to the runtime code, if this is possible. And ideally even move it to the ChainWorker module

Comment on lines +294 to 314
let make = (~isWildcard, ~topicSelections: array<LogSelection.topicSelection>) => {
let topic0sGrouped = []
let topicSelectionWithFilters = []
topicSelections->Belt.Array.forEach(ts =>
if ts->LogSelection.hasFilters {
topicSelectionWithFilters->Js.Array2.push(ts)->ignore
} else {
ts.topic0->Belt.Array.forEach(topic0 => {
topic0sGrouped->Js.Array2.push(topic0)->ignore
})
}
)
let topicSelectionNoFilters =
LogSelection.makeTopicSelection(~topic0=topic0sGrouped)->Utils.unwrapResultExn

{
isWildcard,
topicSelections: Belt.Array.concat(topicSelectionWithFilters, [topicSelectionNoFilters]),
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do this?

Comment on lines +1 to +27
let toTwosComplement = (num: bigint, ~bytesLen: int) => {
let maxValue = 1n->BigInt.Bitwise.shift_left(BigInt.fromInt(bytesLen * 8))
let mask = maxValue->BigInt.sub(1n)
num->BigInt.add(maxValue)->BigInt.Bitwise.logand(mask)
}

let fromSignedBigInt = val => {
let bytesLen = 32
let val = val >= 0n ? val : val->toTwosComplement(~bytesLen)
val->Viem.bigintToHex(~options={size: bytesLen})
}

type hex = EvmTypes.Hex.t
//bytes currently does not work with genType and we also currently generate bytes as a string type
type bytesHex = string
let keccak256 = Viem.keccak256
let bytesToHex = Viem.bytesToHex
let concat = Viem.concat
let castToHexUnsafe: 'a => hex = val => val->Utils.magic
let fromBigInt: bigint => hex = val => val->Viem.bigintToHex(~options={size: 32})
let fromDynamicString: string => hex = val => val->(Utils.magic: string => hex)->keccak256
let fromString: string => hex = val => val->Viem.stringToHex(~options={size: 32})
let fromAddress: Address.t => hex = addr => addr->(Utils.magic: Address.t => hex)->Viem.pad
let fromDynamicBytes: bytesHex => hex = bytes => bytes->(Utils.magic: bytesHex => hex)->keccak256
let fromBytes: bytesHex => hex = bytes =>
bytes->(Utils.magic: bytesHex => bytes)->Viem.bytesToHex(~options={size: 32})
let fromBool: bool => hex = b => b->Viem.boolToHex(~options={size: 32})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can simplify it and do use something like this instead https://viem.sh/docs/contract/encodeEventTopics.html ?

Also, I'd like to move as much logic as possible from codegen. So doing encoding with Viem instead of selecting the right encoder in Rust, will make the envio closer to the library-like state

Comment on lines +130 to +136
switch contract.events->Belt.Array.flatMap(event => {
let module(Event) = event
let {isWildcard, topicSelections} =
Event.handlerRegister->Types.HandlerTypes.Register.getEventOptions

isWildcard ? [] : topicSelections
}) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nitpick, but we can also precalculate the topicSelections on the ChainWorker creation and only join them with addresses here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but we can only apply those based on the addresses in the contractAddressMapping. So we could create a premade lookup table essentially of contract->topic selection and on each contract in the contractAddressMapping make a LogSelection with the topics and addresses. That is not far off from what's happenening here but we could probably improve it slightly 👍🏼

@@ -0,0 +1,4 @@
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it should be ignored

@JonoPrest JonoPrest enabled auto-merge (squash) September 11, 2024 07:45
@JonoPrest JonoPrest changed the title Implemt user API for wildcard indexing Implement user API for wildcard indexing Sep 11, 2024
@JonoPrest JonoPrest merged commit 7f6c504 into main Sep 11, 2024
1 check passed
@JonoPrest JonoPrest deleted the jp/wildcard-user-api branch September 11, 2024 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants