Proof of Data Possession (PDP) [was PoCD]: Efficient Retrieval Provably Enabled

[lucaniz]

Motivation

One key feature Filecoin should enable is efficient retrievals. A key insight from the growing data-owner base of Filecoin, is that this efficiency is of utmost importance. This holds both for the clients themselves and for SPs who are faced with difficult choices on how to optimize their hardware for the most common use-cases.
In essence, given the way our sealing process is designed today, the easiest way to ensure such a feature is via asking providers to arrange reliable access to the clear data (ie, unsealed data). One key question is how to assure clients that, once they hand their data to an SP, this reliable access to their unsealed data is continuously maintained.

We propose to introduce a new proof which is specifically checking access to a clear copy of the data in order to enable efficient retrieval capabilities.

This represents a step forward in terms of Filecoin protocol guarantees. Even if it would not guarantee data delivery (impossibility results from classical cryptography prove that this is impossible without a third party), it would provably guarantee that efficient retrieval is at least possible.

Protocol Specification

A new concept of (unsealed) data container is introduced, which is independent from sealed sectors.
A new proof type "PDP" is introduced, which is independent from PoRep.
We also introduce two new methods to the existing miner actor:

• PDPProveCommit for adding data containers (the analogous of ProveCommit for sectors)
• SubmitPDP for proving access to data into the containers over time (the analogous of SubmitWindowPost for sectors)

Anyone willing to provide proof of data access on containers basically follows the Filecoin existing storage flow.

• Sends onchain a message PDPProveCommit with Comm(D) and (possibly) a proof of correctness.
• Periodically submit PDPSubmit to prove clear data access guarantees over time

Note that the maximum size of data container being proven via a single SubmitPDP, similarly to what happens with SubmitWindowPost partitions is fixed and dictated by the snark circuit.

Impact on Filecoin

We propose to introduce a new proof which aims to give evidence of clear data access. This proof should be decoupled from the concept of sectors, and follows the principle of full flexibility.
Indeed, we identify different potential applications:

• At the protocol level (L1): we envision the possibility of providing Proof of Data Possession for data which is either encoded into sector or stored in the clear.
  This would open the door to new players in the protocol which can serve as Data Providers without being rewarded for persistent storage, as well as giving current SPs the possibility of proving evidence of fast retrievability capabilities to their clients.
• On top of Filecoin (L2): many applications would benefit from being able to prove Clear Data Access independently from the core protocol, via (F)EVM contract

Note that, in order to have a significant impact, this new proof must be smoothly usable also outside the Filecoin core protocol. This means that on one hand, it is important to decouple Proof of Data Possession from the concept of sector. On the other hand, having full compatibility with the Filecoin core protocol is also crucial. and on the other hand is the reason why having this proof decoupled from the concept of sector is crucial.

On Retrievability Enablement Vs Retrievability Guarantees

On one hand, in the current shape, the new proof we aim to introduce can not provide retrievability guarantees. What we can provably ensure is efficient retrievability capabilities.
On the other hand, having such a new tool would allow any builder in the ecosystem to design his own system of incentives in order to make efficient retrievability rational and plug in that incentive system directly on PDP.

Note that this can happen on L1 (and we are planning to work on it) and on top of Filecoin, in any L2 protocol. Note also that given the nature of PDP and its independence from the notion of sector, we envision PDP to be perfectly usable by other chains and protocols as well.

Security Considerations

In the current stage we are not proposing protocol incentives for PDP.
As PDP-proven data are not considered to be persistent nor have a role in consensus, no security threat is introduced by this new proof.

A detailed security analysis will be done when we'll be ready to introduce an incentivized L1 version of PDP.

Note that the fact that we are not currently proposing incentives at L1 does not prevent anyone willing to use PDP in its own protocol to reward it. In this latter case any protocol willing to incentivize PDP should run a security analysis its own incentive system.
Gas and Proving Overhead Implications
At L1, Given PoDC follow the same flow as current Filecoin sectors (apart from being "sealing free"), the gas and proving costs are the known ones. In particular

• At PDPProveCommit the gas cost is

  • The one of posting a commitment onchain (similar to current CommR)
  • The one related to prove verification (if applicable)

• At PDPSubmit the gas cost is comparable with the one associated with the current SubmitWindowPost

[jaychoucc]

support

[Fatman13]

We propose to introduce a new proof which is specifically checking access to a clear copy of the data in order to enable efficient retrieval capabilities.

Why PeRep+PoST was chosen over PoCD in the first place when protocol was designed, if retrieval is considered this important to be on-chain? It sounds like PoCD can just replace PoRep+PoST?

[lucaniz]

PoRep + PoSt ensures persistent storage. PDP does not give guarantees on persistent storage and thus in principle is not fully suited for full storage based consensus. If you want storage based consensus you need something in line with PoRep + PoSt as you have the guarantee that if you prove storage of 32GiB, you are actually storing 32GiB (no chance of compressing/sharing/etc)

Indeed, data under PDP can be compressible/shared and still allow for passing the proving step. PDP proves accessibity to a copy of the data, not storing of a unique piece of storage of a given size. Hope this helps.

[15012700225]

PoCD equal double disk used for a sector? should change implement PoRep+PoST to that keep unsealed file and remove or
reduce the size of sealed file. Balancing the needs of spatiotemporal proof and fast access.
This is a good direction to separate user file access from computing power, as the priority of the storage system is to ensure user access, and computing power calculation can be delayed.
Overconsideration of generating attacks is the reason why it takes several hours for the seal to be too long. If PoCD like proofs are added, it can reduce dependence on PoST and shorten the sealing process. Therefore, sealing and proofs are a whole and need to be considered together. Perhaps paid storage is not far away, otherwise paid storage is far away, and filecoin is unlikely to become a storage option for formal projects.

[lucaniz]

Data proven via PDP can be shared, so the storage overhead can be amortized.

Keep in mind we are suggesting to introduce a new proof which is independent from sectors. This means that

• one ould go w/ PDP without having sectors encoding the same data
• one having a sector and storing ax extra copy can give additional provable guarantees
• one having a sector can decide to pay the (amortized) cost of accessing a copy in the clear for provably showing he is actually enabled to provide efficient retrieval.

[jennijuju]

This would open the door to new players in the protocol which can serve as Data Providers without being rewarded for persistent storage, as well as giving current SPs the possibility of proving evidence of fast retrievability capabilities to their clients.

Is the Data Provider role necessary in Filecoin L1?oversial, but in my opinion, L1 needs to focus only on SPs serve retrieval. Other roles and incentives for those roles can be built on top of It.

With that thought, which assumes the primary data provider is SP, have we considered to

• add "PoCDRequired" flag to PieceMainifest (introduced in FIP0076), which is sent by SPs to the chain, tho value should be set to the retrieval SLA agreement made between the client & SP offchain (i.e: in deal protocol)
  Note: Understood PoCD doesn't provide a retrieval guarantee. SLA here only means those who provide PoCD have a higher chance to serve the first byte upon receiving a retrieval request without the client need to wait for unsealing to complete.
• (skipping incentive design of PoCD for now, but it needs to exist in some form so that SPs are willing to store extra copy, i.e: paid deals)
• SPs will be slashed if fail to submit PoCD for those sectors, includes PoCD-required pieces
  • question: can PoCD be piece-based instead of sector-based?

[momack2]

I'm excited about adding this new proof primitive to Filecoin for a number of reasons. Some of the benefits I think this can unlock for Filecoin:

1. Fast and Economical Cache Space: The new proof type unlocks a cost-effective "cache" space for data storage on Filecoin that doesn't require sealing and unsealing. This allows for rapid onboarding and retrieval of data, helping Filecoin better support applications that require frequent access to stored information.
2. Interactivity with L2s & Smart Contracts: By creating a primitive that L2 solutions and smart contracts can interact with directly on the L1 layer, Filecoin can facilitate more complex and responsive data management strategies. This interaction can enable the implementation of dynamic incentives and other operational logics directly tied to the data storage and retrieval processes.
3. Expanding the meme of Filecoin for Hot Data: Filecoin has traditionally viewed as a solution for long-term, cold data storage, Filecoin’s addition of hot storage capabilities challenges this narrative. It makes the platform suitable for a broader range of data storage needs, including frequently accessed data, thus broadening its appeal and utility.
4. FIL+ Integration and Data Onboarding: With the FIL+ mechanism, the platform can mandate the maintenance of a hot copy of data, leveraging the existing incentives to subsidize this kind of data storage. This ensures that critical or frequently accessed data remains readily available, enhancing the service level for applications relying on quick data access.
5. New type of Filecoin SPs: This new proof type opens the design space for a new type of Filecoin storage provider to emerge who differentiates by offering hot/fast data storage on Filecoin - and getting payment from clients for making this data highly available.

Curious if other folks see these same opportunities - or other ways of achieving these benefits even faster?

[tmellan]

From an economic perspective the development is exciting because the apparent combination of lower protocol costs across several dimensions (data prep, sealing, extra copy, gas), plus potential for much higher deal fees, since hot storage is more valuable than cold.

Kiran and I were curious and took a pass seeing how different SP profiles look, comparing PDP to CC, Fil+ and so forth across a range of cost/revenue variables which can be changed with sliders: https://pdpeconomics.streamlit.app/Cost_Breakdown

A few observations:

• Even with 0 incentives, PDP appears viable at deal income that is relatively low (check sliders ^)
• If incentives were considered for PDP, just as a thought experiment within the current block rewards paradigm with a multiplier >=1 (move sliders), then the space where PDP is viable, and even the rational choice, starts from a very low deal income, and so the space where this approach can succeed is wide. Of course there are many considerations and potential design space, but I thought it was nonetheless encouraging.

Would be great to hear more about a timeline for getting a version of PDP online, either L2 or L1.

Thanks Luca and team for sharing the exciting work!

[Fatman13]

Building Something that Nobody Want

From product perspective, not sure if the motivation is enough to warrant the resources dedicated. "Building something that nobody wants" would be wasteful to tech startups / networks. Snapdeal, for example, has seen insignificant usage for almost 3 years and no one is taking up ownership to improve the UX around snapdeal to increase its usage. A typical case of building something that nobody wants.

To PDP motivation, i would ask the following basic questions, focusing on why people would want to use the product feature...

• If you build it, will they come?
• Why would SP want to send extra messages on-chain?
• Why a client would be relatively comfortable to see PDP as retrievability?
• Are there real distributed storage demand in scale in the real world?
• Are there real distributed retrieval demand in scale in the real world?

There should be clear metrics to valuate any hypothesis which PDP may have for the above questions to validate the resources committed. Or it would be just another snapdeal, a failed product.

[steven004]

Firstly, I would say PDP is a brilliant idea within the Filecoin ecosystem for verifying data retrievability and incentivizing retrieval services.

With DDO (Direct Data Onboarding) and other improvements we've implemented to remove built-in market privileges, PDP presents a great user scenario that can be built within a user-defined market. This market would feature its own incentivizing mechanisms, real data handling, and significantly better service for data clients compared to the original deal system.

I would be more than happy to support this if it is built on FVM.

However, I am not in favor of making any changes to the core protocol to support this, as it would complicate Layer 1. We are trying to elevate Fil+ to foster more and faster innovation within the ecosystem. PDP is another example of this, and we would like to see more innovation at higher layers.

Some might argue that the project would be more likely to succeed if it could use the native token ($FIL) to incentivize honest retrieval servers. While this looks probably true, the crucial question is how much is appropriate for this? There is no definitive answer. Why not let the market decide? If implemented as a user-defined market and utilizing its project token for incentivization, the market will automatically value it. Additionally, the tokenomics will absorb good amount of provers to join, helping to bootstrap the system, and also attract users to the ecosystem just like DePin projects.

Again, in this way, this would be a great web3 project in Filecoin ecosystem.