Increased Terminations #691
Replies: 15 comments 25 replies
-
I'm not sure what the right solution would look like, but I agree this is something we want to solve. If at all possible, I'd be interested in trying to find a structure where one-off terminations can be discounted. We want to penalize intentional termination and/or faulty storage practices, not one-off errors. E.g., for every M sectors sealed and/or "naturally expired", an SP gets a discounted termination. |
Beta Was this translation helpful? Give feedback.
-
If we do something about the dipping block reward, I guess the termination fee concern would be solved automatically.Why don't we pursuit that end? |
Beta Was this translation helpful? Give feedback.
-
I'm not sure if this is the right solution. I suggest being very cautious about this matter. In principle, increasing the sector termination fee will increase the risk for storage providers and also increase the entry threshold for them, which will indirectly have a negative impact on total power growth, which may be contrary to the original intention of this idea. There are many reasons for terminating sectors, but there are mainly two: one is force majeure, such as private key loss or natural disasters; the other is that the storage providers consider it unprofitable or less profitable than expected, and is willing to exit the market. For the first case, increasing the termination fee will make storage providers feel that the risk is higher, and they may hesitate more when entering the ecosystem; for the second case, some storage providers will still consider the high risk of not being able to exit, and may be hesitant to enter. That is, this proposal may set a higher barrier for existing SP to early leave, but set a higher barrier for new comers. From the perspective of network and data security, adopting a more flexible strategy may be more helpful. For CC sectors, which do not contain valid data, if the threshold is lowered (including reducing the termination fee), a large amount of storage power may be added when there is a profit opportunity; when there is not enough profit opportunity, some poorly managed service providers will exit, thus leaving better profit space for the storage providers who continue to serve. This is a virtuous balance. For DC data, it is best to let the market evaluate the terms. A better way is for clients and storage providers to negotiate the collateral rate and termination fee, because different data has different values, and the agreements they reach will be different, which can better meet market demands. It can be understood that in the initial design of Filecoin, we used a fixed collateral ratio and sector termination fee ratio because we couldn't establish a user-defined market at that time. After FVM goes online, we can consider leaving these to the market. When Filecoin can better meet market needs, its value can be better reflected. |
Beta Was this translation helpful? Give feedback.
-
No. The motivation behind increasing termination is not properly addressed in the FIP discussion.
The above statement is very counter-intuitive. If a storage deal is profitable, why would storage providers prematurely exit? Plus storage clients are already making replicas anyways. |
Beta Was this translation helpful? Give feedback.
-
Do people think a 0.5y sector and 3.5y sector should have the same termination fees? |
Beta Was this translation helpful? Give feedback.
-
1, since termination fee is derived directly from block reward, why don't you consider add block reward to increase the termination cost? 2, Why do you think add termination fee would remediate future terminations? Did you do any research on the termination events happened in the last 12 months? And if the penalty've been raised, would they changed their mind and choose not to terminate? If no research has been done on that, your proposal is just pre-mature. |
Beta Was this translation helpful? Give feedback.
-
Firstly, thanks for opening this discussion. While I can sense the implied proposal is an increase to termination fees, I'm glad the discussion has been opened to prompt reflection on the principles and goals around which such mechanisms might be based. I have a few large clusters of thoughts, and I will make separate comments about each as discussion-starters. Doubt over the projectionsI have questions/gripes about the forecasts given as context. I followed the link to (undocumented) R code that produced them and the also skimmed the paper describing the Prophet forecasting system used. I don't think I and others should have to do this, though – an explanation of the basics of the model used is necessary for interpreting it. It appears to be pure trend-fitting, with no knowledge or structure of the underlying system. From the paper, "We are, in effect, framing the forecasting problem as a curve fitting exercise". Is this appropriate for a system where we have so much more knowledge about the environment? We know things like block reward, token price, pledge distribution, interest rates ... almost all the inputs. With a model of such factors, we might address them directly rather than hacking around them with increased fees. What reason do we have to expect the "trend" discovered by this regression to continue? On what basis would we make the hypothesis that a meaningful trend dominates here, rather than specific structural or environmental factors? Maybe we just shifted mode to a new normal? Did the model use non-linear saturating growth, or linear trend-fitting? How many changepoints were modelled? I can't tell this from the code. The big red boxes on the chart give the visual impression of increasing badness due to their size and colour. It takes some thought to realise that the visual impact should in fact be interpreted as large uncertainty. Bounded below by zero, higher is the only available direction for that uncertainty, but that doesn't make it a good forecast. I thus find the charts a bit misleading, even if I thought the underlying model was any good for forecasting this phenomenon. I'm certainly not asserting that terminations won't increase – I don't know. But I'm not confident this projection knows much better. A principled model based on the economic drivers that we understand well would be much more convincing. |
Beta Was this translation helpful? Give feedback.
-
Motivation for termination feesI think it's worth describing why we might want termination fees at all. We first need to reach some alignment here if we want to later align on a mechanism or change. I'll list some motivations, in rough decreasing order of importance to me.
Client assurance
This is a legitimate goal, but as @steven004 points out above, a sector termination fee is not a good way of providing client assurance. Big reasons why are (a) it's not negotiable by the client, either higher or lower, (b) it has big impacts on sectors that aren't carrying deals, and (c) we already have deal collateral as a somewhat negotiable (higher only) disincentive in the built-in storage market actor. With client interests in mind, surely it would also be much preferred to pay the fee to the client in the event of a deal agreement cut short. All up, client assurance is quite important, but applies only to data and would be far better left to the parties to negotiate. The network could assert a level of minimum client assurance, a minimum termination fee that must apply to all arrangements with all clients. Perhaps there is some social good to be gained from this that can't be reached by the market (compare with a minimum wage law, and all associated uncertainty and controversy). I'm not convinced, but if relevant, it shouldn't apply to CC sectors that provably have no data. Stability of storage How valuable is stability of the storage commitments? Bitcoin has no exit fees, consequently volatile hashpower, but ... does this matter? I'll grant there is probably some value here for the image of a storage network, even if it only affected CC. Stability will reassure potential clients. Some disincentive to exit is probably justified (but there might be alternatives to termination fee). This motivation does not justify any differential in termination fees for different sectors of the same size – if an SP is going to terminate a sector, it may as well be the least-profitable one to them, regardless of duration. There might be a security angle to this stability too, raising the cost of a short-duration attack based on flash onboarding then exit of storage. Stability of pledge Pure staking networks typically address this with mandatory lock-up times or un-staking delays. Filecoin can't mimic these exactly because the un-staking is coupled to a real-world random event of proving storage (or failing to). A termination fee might be second best. Stake is fungible, so this would motivate termination fees proportional to pledge (which, unfortunately, varies between sectors). Enforcing commitments I am unmoved by this motivation. This is a moralistic stance, very hard to argue without using words like "should". I think blockchain protocol design is far better served by a dispassionate analysis of the incentives that are introduced, and we should (!) attempt to design those incentives to achieve mutually beneficial outcomes. There is no promise, there is only a cold business decision made in light of the protocol rules. Scaling termination fees with duration is a disincentive to long-duration sectors. An alternative moralistic take on the situation is that a sector that has been online for a long time has done great service to the network. This demonstrated commitment to the network ought to be rewarded. There is zero damage done to the network if it terminates the epoch prior to meeting its "commitment" – why should we charge the highest possible termination fee? We should reduce it with duration instead! OTOH, an SP constantly onboarding and terminating sectors for very short terms is causing state churn, pledge and storage volatility, and probably has ill-intent – why should they get low termination fees? I don't think that the above take should carry any weight either. I'm just showing that a moral take invites debate about "should" that can never be resolved by data or analysis. The incentives are what counts. Network revenue Earning revenue by increasing the costs of participation seems like a self-defeating strategy this early in the network's life. SummaryThere may be a security need for some termination fee, which is important to understand and preserve if so. Beyond that, stability of storage and pledge may motivate some level of termination fee, trading off maximum participation levels for decreased volatility. Neither of those justify charging different fees based on duration committed or served, but maybe should include reference to heterogenous pledges. Are there other motivations I have missed? |
Beta Was this translation helpful? Give feedback.
-
Healthy termination levelsThe global economy, the web3 ecosystem, and Filecoin have all suffered from harsh economic conditions for a significant period now. In such conditions, it shouldn't be surprising if some participants (the least-efficient and/or least-aligned) are seeking to exit. Is it not healthy to let them? An SP who wants to exit because they are losing money (but would lose more by paying termination fees) is not an aligned participant. The value of any rewards they earn will surely leave Filecoin's ecosystem. Economic value is being destroyed by keeping them running. If they have investors who are otherwise aligned with the ecosystem but treat this SP as a loss, operating the unprofitable operation is only reducing their ability to re-invest elsewhere. Larger termination fees lock in unaligned participants for longer. This prevents their block rewards flowing to more efficient or optimistic participants. This in turn moves the most marginal of them closer to a decision to exit, and limits reinvestment potential of others, compared with the original SP being able to rationally exit. This may perversely cause a larger exit than otherwise, if the conditions persist. The fragility of individual sectors or SPs is a source of antifragility for the network as a whole. I think this is a motivation against termination fees (but there are motivations for, too). The healthy termination level is not zero. Perhaps its higher than 1/2e-05 per day (0.73% per year, higher than almost all of history) or even 1/3e-05 per day (1.1% pa, middle of the projected scary red bars). Is there a principled reason to believe it should be lower? If those projections are in fact prophetic, is that actually worse than a system design that bottles-in smaller stresses but then suffers more badly with a larger shock? This is orthogonal to client assurance. Yes 1.1% pa deal failure rate would be too much (except for clients who want cheap and dirty storage). I'm curious whether the sectors exiting carry deals. But as discussed above deal assurance should be a matter for negotiation between the parties. Note: this musing is outside my core areas of competence, I hold it lightly and am very open to being persuaded otherwise. |
Beta Was this translation helpful? Give feedback.
-
One under-discussed element to this FIP thus far is what increased termination fees may do to lenders in the ecosystem. If I'm reading this correctly, an increase to the termination fee would dramatically change a lender's risk model towards being significantly under-collateralized. We've flagged this discussion for lenders to provide additional insights/impact. I could easily see how a rise in the termination fees could have a dramatic impact on lending risk models. Thus, I'm not sure if the community would be in support of anything that made lending more challenging than it is today or a change that would results in a larger upfront down payment than currently exists (~10%). |
Beta Was this translation helpful? Give feedback.
-
Design goalsI do think there's room to significantly improve things here. I propose two design goals to consider if we're going to change anything. SimplicityThe current termination fee mechanism is complicated. The calculations are complex, and the inputs required for those calculations consume chain state. Complexity makes this hard for participants to model, and hard for implementers to ensure correctness. As a data point on this complexity, the description of the fee calculation in the original post is incorrect. The The inputs to this calculation require three No protocol bias between sectorsDifferent sectors have different termination fees depending on the network conditions at they time they were onboarded. This distorts SP incentives about which sectors to terminate, if they are going to do so intentionally. All sectors with the same QAP earn the same share of block rewards, and contribute the same storage capacity. An SP should be free to terminate whichever sector is least valuable to them. When other incentive systems are appropriately designed, this will also be the sector least valuable to the network. For example, if the SP has a client-paying deal in a sector, it is more valuable to them and the network, and should be least preferred to terminate. A higher-priced deal is in turn more valuable and less preferable to terminate. The current mechanism of indexing termination fees to past network conditions means that sector termination fees vary independently of sector content. Sometimes a deal sector will have a lower termination fee than a CC sector, giving an SP a preference to terminate the deal. There are mechanisms like mandatory deal collateral to counter this, but the sector termination fee can work in opposition to all parties' interests. It is arbitrary with respect to sector utility. The protocol should treat all currently active power as equal. The activation/upgrade epoch or age of a sector are not factors in the value of a sector to the SP or to the network. (Note: a second protocol bias between sectors is varying pledge. The termination fee isn't indexed to pledge, though, so the termination-fee-to-pledge ratio also varies.) A termination fee based only on the current network state would be simpler, cheaper, and remove an arbitrary protocol bias between sectors. |
Beta Was this translation helpful? Give feedback.
-
Thanks for putting together this discussion and to the community for all the great contributions so far. I'll respond to this from a few different perspectives.
It would make more sense to address the primary issues at hand, which I understand to be:
Look forward to further comments. |
Beta Was this translation helpful? Give feedback.
-
Philosophically I think we should optimize for flexibility and minimize barriers to entry and exit. Miners ability to join and leave the network at will is a core feature of BTC/ETH and part of their value proposition. The right goal is to make the network resilient and adaptable so that when miners drop out others pick up the slack and the quality of service isn't affected. |
Beta Was this translation helpful? Give feedback.
-
Quality Adjusted Power is up about 25% YoY while raw power is down almost 30%. That tells me only FIL+ deals are profitable. I suggest slowly decreasing the block reward multiplier for FIL+ deals. |
Beta Was this translation helpful? Give feedback.
-
Hi all, thank you for your continued contributions to this conversation. CEL has spent some time analyzing the temrination fee structure in more detail and would like to present our report to the public - https://hackmd.io/@1dR0N2W7SQyZWg7DGB8Vfw/HyKRs2AD3 The TL:DR is:
Please note that this is just one example of a possible change. This structure should further iterated on to consider structural features such as:
Modifications to the existing strucutre can be brough out only after rigourous simulation and investigation. To that end, we would like to hear any potential ideas that the commmunity has, building on our proposal or otherwise, and extend the convrsation to design experiments through agent-based-models and simulations that can be used to investiage what the effect of these changes would look like. |
Beta Was this translation helpful? Give feedback.
-
TL;DR
The network is seeing an increased number of sectors being terminated. In the months between July-2022 to March-2023, the network has recorded higher values for the daily fraction of raw byte power terminating than the past year and a half.
The present termination fee structure was designed in the context of 6 month sectors. The principle that it follows is to cap the termination fee to half the lifetime earnings of a 180 day sector. Since then, the network has also onboarded sectors with durations up to 1.5 years, and the termination fees for such sectors is still capped at 90 days worth of block rewards. This means that the current fees may not be a sufficient disincentive to prevent storage providers from prematurely exiting storage deals.
Terminations in the network
Historical and projected termination are shown in figure below. The terminations are shown both in raw form (TiB/day), and as a fraction of network raw byte power. Details on the forecast method can be found here.
Fig 1. A) Boxplots drawn to capture the distribution of data relating to the daily fraction of RBP terminated. The bands are drawn on the 25th and 75th percentile values. Projections are obtained for the same confidence interval. In the months between July-2022 to March-2023, the network has recorded higher values for the daily fraction of raw byte power terminating than the past year and a half. Current projections indicate a strong potential for it to go even higher in the coming year. B) A similar trend is noticed on the daily raw byte power (TiB) terminating. Projections for the coming few months show a potential for increase in the daily terminations of the network.
The above analysis indicates a relative increase in both the daily terminations (measured in Raw Byte Power) as well as the daily fraction of Raw Byte Power Terminating in the past 6 months. Projections also indicate a possibility for this trend to continue for the coming year as well.
If this trend continues it could affect client experience and the stability of the network.
Current termination fee structure
Definitions:
TERMINATION_LIFETIME_CAP
, i.e the maximum number of daily block rewards penalized from the sector’s start epoch forward, currently set to a value ofCurrently, the termination fee is implemented as follows:
Note, that in this equation 2880 represents the number of epochs in one day. Therefore, the first part of the equation$SP(t)$ computes the daily block rewards aggregated over a period of 20 days since the sector was initially sealed/last upgraded. The second part of the equation, $B(t)$ , scales the daily block rewards aggregated over a 1 day period since the sector was initially sealed/last upgraded, with a constant weighting factor that is defined in the code, and the minimum between the number of days the sector has been active for, and a constant
TERMINATION_LIFETIME_CAP
which is also defined in the code.The current value for$w$ was set to 0.5 and $TLC$ 140 after the community arrived at consensus on what an acceptable value would be. This was done keeping in mind that the maximum sector duration at the time was 6 months. Therefore, with the combined values of the two parameters, according to the current fee structure, the maximum penalty that an SP would be charged for terminations would be 90 days worth of block rewards. This is equivalent to the block rewards that an SP with a 6-month sector would earn for half the lifetime of their sector.
It is important to note that since then the maximum sector length was increased to 1.5 years, which means that under the present structure a sector of length 1.5 years and a sector of length 6 months pay the same termination fees.
Upgrading the termination fees
Given the present rate of increased terminations in the network, and the outdated assumptions under which the present fee structure was derived, we would like to open up the conversation on changing and updating the termination fees. This is particularly relevant in the context of the upcoming FIP-0052 which will increase the maximum sector commitment duration to 3.5 years.
Given this, we’d like to suggest that as a community we:
Beta Was this translation helpful? Give feedback.
All reactions