New Tooling Contribution by the Data onboarding working group

[Stefaan-V]

The data onboarded to the Filecoin network has grown significantly in 2023, thanks to the many clients joining the network, and storage providers and data onboarders contributing to onboarding data to the network. Filecoin now stores over 1.4 EiB of data, up ~10x since mid-last year, and continues to onboard ~5 PiB of data per day!

A wide variety of data onboarding tools have been built to support the storing and retrieving data with Filecoin's network of Storage Providers, but always accessible to everyone and sometimes proprietary. For the Filecoin network to become a widely adopted storage platform for traditional applications we need to make it easy for users to use the Filecoin network.
I received multiple requests from the community, asking the onboarding team at PL how we are contributing to this effort and helping the community. In the last couple of months, the data onboarding team decided to double down on a subset of the data onboarding tools (e.g. Singularity, Project Motion, Spade) that are aimed to help with this effort. We believe these efforts will help the community drive better client onboarding experiences and expand the feature sets required to make the Filecoin network a competitive storage platform.

Problems we are trying to solve for:

1. Web2 apps still rely on REST and File System interfaces

There are successful examples of S3 gateways and other onboarding tools that have been built by the ecosystem on top of Filecoin, and we see more coming online daily. Yet builders face a steep learning curve when integrating with Filecoin and the development of these onramps is not moving fast enough to keep up with the storage supply. One of the main reasons is that there is no easy way to integrate Filecoin with these widely used data movers (archival, backup+recovery, content management, etc.) that use S3, SMB, NFS, POSIX or a restful API to connect to a storage layer. Having a RESTful API would make it much easier for Web2 apps to integrate as it is native to how they store data today.

2. Lack of standards slows the time to market to a wider audience and use case functionality

To achieve a wider reach of users and support a wider set of use cases, data onboarding services on the network need to integrate with as many existing independent software vendors (ISV) as possible.

Second, to drive differentiation and achieve data composability, applications need to expose and integrate the unique strengths of Filecoin into the application stack. I’m referring to the CIDs and the daily data timestamps when the data verification was run by the providers. To make this happen, we need an easy and standard method to pull this on-chain data from the network into the application UX. Only then will we be able to expose the benefits of verifiability and content addressable storage into the traditional Web2 workflows and increase the value differentiation beyond cost.

3. Lack of engineering resources limits the speed to data onboarding

Even though around the world, there are successful examples of customer onboarding tools built by the ecosystem, there are a limited number of teams building these tools today and most of the 1000s of storage providers hosting storage capacity don’t have the software teams to build these tools. Yet there are a huge amount of sellers, integrators and providers that have access to users that are looking to store data on the network. Those members need easy integrated solutions and vetted architectures that plug-into the Filecoin network.

4. A lack of automated user to storage provider matchmaking

Filecoin is a permissionless market place which allows everyone to participate in the Filecoin network. This permissionless feature makes it harder for users to find the storage provider of choice with the right functionality to support the user request. The network needs market places to find and transact deals between users and providers and make a transaction with the network easy and automated.

How the data onboarding team is contributing to the network:

I’m sure there is nothing new here that you haven’t heard already, though I thought it was important to highlight what the onboarding team is building to support the ecosystem.
These are 3 main projects the onboarding teams are focused on now:

1. Simplified Filecoin API

The onboarding working group is building an automation and orchestration layer with a REST API interface to bridge and integrate Web2 apps more easily with the Filecoin network. This open source project (“Project Motion”) is still under development and can be followed on Github here. This orchestration layer abstracts the complexity of how data is prepared and deals are made with SPs, while providing tracking and reporting status of the dataset’s integrity and the unique identifier it is referenced by.

Existing applications can easily be integrated through these APIs without having to understand the full Filecoin ecosystem. In this way new application builders no longer need to build an entire stack and can focus on feature differentiation while maintaining data composability for their users.
You can help by downloading the docker container, test the API and provide feedback on Github.

2. Data Preparation tools - “Singularity v2”

The popular Singularity tool has been used to onboard close to 275PB of data onto the network. After receiving a ton of user feedback and requests from the ecosystem, the data onboarding team is expanding its resources and adding new functionality to v2 of Singularity.
Check out the list of new features added to Singularity v2 here. New features include encryption, wallet management, remote signing capabilities etc. and more features are being worked on, including a lightweight UX interface. The new orchestration tool, “Project Motion”, is being tested and integrated with Singularity, but not exclusively dependent on it. Other data preparation tools can be integrated and used with the orchestration tool.

3. Data onboarding marketplace - “Spade”

Spade matches storage clients to storage providers and facilitates high-throughput low-friction data onboarding. Storage clients get to specify how and where their data should be stored, including what type of functionality (e.g. SOC2 compliance) should be supported. Storage providers that comply with these requirements can take advantage of this marketplace and accept storage deals in a peer-to-peer way.
Spade is currently in a pilot phase, where the onboarding team is only onboarding a select set of tenants. If you are interested in participating in the pilot or are generally interested in using Spade to onboard data to the Filecoin network, see further details here!

Below is an architectural view on how these components integrate

What’s next and how can you contribute?
We ask the community to help provide feedback on the above projects and bring more ISVs to the network. In the upcoming weeks we will be releasing more updates on the roadmap, new landing pages and share reference architectures on how these tools can be deployed.