Can lily support third-party plugins for scraping metrics from Filecoin Network?

[placer14]

We've been talking about the idea of leveraging golang plugins to allow lily to support custom scraping logic. Lily has developed into a robust distributed worker pipeline for scraping data/metrics from the Filecoin network. Lily is a full Lotus node, the reference node implementation on the Filecoin network, and continually re-integrates new releases as they are made available.

The targets are prioritized for inclusion by the Sentinel team (albeit with generous room for community input). But we'd like to remove ourselves as gatekeepers and enable others to build custom scrapers which can leverage Lily's existing worker pipeline and accurate view into Filecoin network's state and behavior.

Here are some potential use cases:

• collect aggregates around behavior for specific classes of actors or messages before/after inclusion on-chain
• custom networked communication between multiple Lily nodes to survey Filecoin network communication efficiency
• capture correlated metrics at multiple layers of the Filecoin technology stack
• harness, observe, and rapidly test/deploy custom FVM contracts in a live node

Such use cases serve a very narrow audience and wouldn't make sense to directly include in releases, but still have value and users should be able to easily collect this information without re-inventing the wheel Filecoin network parsing machine.

Would this be useful? How would you use it? How important is this to you? Other thoughts welcome, too! 🙏

[f8-ptrk]

Would this be useful?

YES!

How would you use it?

Mpool surveillance in the first place

How important is this to you?

we'd build something anyways - with or without lily - piggybacking might be easier/cooler

Other thoughts welcome, too!

I think the other end of the pipeline - the output part - is equally important. We would love to get data directly into an influxdb, for example.

[f8-ptrk]

no. i didn't know. looks pretty much like what we use now :D

what options are there to write the CSV to right now?

[placer14]

what options are there to write the CSV to right now?

I can't think of what you're asking here other than "What can I import these CSVs into?" but that seems like something you'd know. Can you rephrase? But if that is your question, CSVs can be imported into all sorts of shapes and postgres can import CSVs directly. (But you almost certainly know this.) Worst case, you can pick up some SaaS like fivetran to fit the CSV into whatever shape you need on the other end. And I quickly found this article which explains a few more solutions native to Influx toolchain.

[placer14]

PS: Maybe you're asking when are we going to export CSVs to current heights? This is work currently in-progress which we are blocked on due the archival repos being extremely slow and painful to work with directly. I'm not sure what the timeline for this is at the moment, but our team is actively working on getting this pipeline automated so exports are up-to-date.

[f8-ptrk]

could i for example write the csv data via udp to a consumer?

for us it's more about how to consume the data properly into the transformation pipeline to make it read friendly/query'able

i neither would need the postgres nor the csv file if i could plugin consumers for the data - i rather cache the stream into ram than having it persisted in a read unfriendly way

[placer14]

That's basically what Telegraf is doing. It's an ETL agent which can input/output/process data into a time-series dataset. If you need to "stream" CSV data, I'm pretty sure telegraf can do that if your consumer can keep up (telegraf's operational pattern is to push until pressure and then drop data which might be a poor fit for your case) but writing something to parse CSVs on newlines and stream to a connection shouldn't be too much of a lift.

I think a plugin for Lily would end up being the solution you're looking for. The plugin would consume the Lily models directly or transforms/streams your data to another consumer. Then it just comes down to how you prefer to handle the Lily data within the plugin. (My best idea would be to pass in the raw models which match the DB schema as they are exported from the chain.)