This repository contains general data for the Filecoin Plus ecosystem.
This is work-in-progress and the documentation will soon be improved to cover more details about the project.
We're using GitHub workflows to handle data pipelines by leveraging GitHub's Flat Data toolkit. This way, we keep datasets public, versioned, and reusable.
Processing tasks and their outputs are separate from data fetched from the source, making the dataset lifecycle easier to understand while maintaining explainability, so stakeholders can safely and efficiently use the data and audit when necessary.
The repository has the following structure:
| Directory path | Description |
|---|---|
| /.github/workflows | Here's where we define how the data pipeline runs and set basic configuration. |
| /data | Holds directories for each data stage. |
| /data/raw | Raw datasets as fetched from source. |
| /data/processed | Datasets that've been processed by scripts in /postprocessing. |
| /data/generated | New datasets generated for data integration by scripts in /postprocessing. |
| /postprocessing | This is the home for all processing scripts. |
| /typings | This is work-in-progress and is expected to hold TypeScript definitions for the data we maintain. |
| /utils | Holds utility scripts used by processing scripts. |
| File path | Description |
|---|---|
| /data/generated/verifiers.json | Data of all Verifiers/Notaries pulled from multiple sources, processed, and combined. |
| /data/generated/address-mapping.json | Address mapping of on-chain addresses (Filecoin Address ID <-> Filecoin Address Key). Addresses are only added to this file if they exist on-chain (both address types are validated through Glif's APIs). |
| /data/processed/notary-governance-issues.json | Data extracted from the notary-governance issues and gone through filtering, cleansing, and normalization. |
| /data/raw/interplanetaryone-verifiers.json | Raw data fetched from the getVerifiers endpoint of InterPlanetary One's API. |
| /data/raw/interplanetaryone-verified-clients.json | Raw data fetched from the getVerifiedClients endpoint of InterPlanetary One's API. |
| /data/raw/notary-governance-issues.json | Raw data fetched from an endpoint running a script to get all issues from the notary-governance repository. |
Here are some recommended readings if you're interested in some of the concepts that have influenced this project.