The first milestone is to have a well thought out engineering design and plan for building the project. This includes a detailed engineering design doc and development plan, which includes but is not limited to: system diagrams, document/repo with engineering design, specific roadmap on project phases and how project will progress. This will be reviewed by the PL team, and based on the proposal, we will determine checkpoints for the project progress for the duration of the project. Together we will discuss a schedule of progression for functionality buildout, and estimates for each phase/checkpoint. Only one team will be funded for this milestone.

Engineering design and project plan

The second milestone is to have the full implementation and working solution. Based on the previous milestone, there will also be checkpoints between Milestone 1 and Milestone 2 for progress updates and phased project functionality checks. Only a worker that passed Milestone 1 will be eligible to participate in Milestone 2.

Full implementation and working solution

The third milestone is to have the solution fully tested by real users and be fully documented. There should be a full user testing plan (which was reviewed in the prior milestone) executed, with results and learnings that are to be shared with the PL team. The solution needs to be fully documented - each modular component, the end to end solution design, the setup process, user guides, and troubleshooting guides. Only a worker that passed Milestone 2 will be eligible to participate in Milestone 3.

End to end user testing and documentation

The fourth milestone is for ongoing maintenance and support for this solution. This includes bug fixes, responding to questions (by PL and clients), and client support. Based on client input, this will also include implementation of new feature requests. Only a worker that passed Milestone 3 will be eligible to participate in Milestone 4.

Ongoing maintenance

Data onboarding to Filecoin currently is a highly technical, complex multi step process. For most clients with small datasets, they are able to upload data to Filecoin by using hosted services such as <a href="https://estuary.tech/">Estuary</a>, or a combination of data onboarding tools like <a href="https://github.com/filedrive-team/go-graphsplit">https://github.com/filedrive-team/go-graphsplit</a>, <a href="https://github.com/filecoin-project/go-fil-commp-hashhash/tree/master/cmd/stream-commp">stream-commp</a>, and . However, for clients with large, PB-scale datasets, there are gaps with existing tooling. It is difficult to handle PB scale data with the existing tooling and services available. This makes the data onboarding process for large-scale clients challenging, and is a barrier to using Filecoin as a storage solution. This grant should address PB scale data onboarding to Filecoin. The goal is to build a tool or set of tools that can help fill the gaps that PB scale clients deal with (scalability for multiple TiBs to multiple PiBs of data, one stop shop for simple onboarding).

Build a tool or set of tools that can help fill the gaps with data onboarding that PB-scale clients deal with, and conduct user testing for onboarding PB-scale datasets

Please login to apply for this grant!