Rhizome database space patrol

[quixotique]

The Rhizome database currently grows without limit. The following features must be implemented:

• remove old payloads when a manifest is updated with a new one
• evict old content to stay within size limit setting

Mantis severity: minor
Mantis priority: normal
Mantis project: Core Project
Product version: 0.09 Shiny
Target version: 0.09 Shiny
Can reproduce: always

[quixotique]

Originally reported as mantis:0000213 at Wed May 23 03:45:54 2012.

[quixotique]

Issue migrated from Mantis to GitHub with Mantis status feedback and resolution reopened at Tue Sep 11 17:26:58 2012 using sp-mantis2github version 0.91

[quixotique]

At Tue Sep 11 17:56:04 2012:

This issue is now being tracked on GitHub: servalproject/batphone#8

[quixotique]

I propose the following approach to expiring and fetching bundles.

1. Each Rhizome node keeps an ordered list of all the manifests known to it, including the manifests of advertised bundles whose payloads have not yet been fetched, and the manifests of expired bundles whose payloads have been evicted. This list is called the BUNDLE RANK.
2. The ordering of the bundle rank will depend on the manifest contents and information known to the node itself, such as current date/time, available payload storage space, routing state (recently encountered SIDs), etc.
3. When a new bundle advertisement is received, its manifest is fetched and inserted into the rank. Rhizome then iterates down the rank, counting the space required by each bundle until the storage limit is reached. The payloads of all bundles beyond the storage limit are evicted to reclaim the space. The missing payloads of any bundles before the storage limit are fetched to occupy the space.
4. It would be nice to be able to sort the rhizome list output in rank order. This would allow the construction of a diagnostic display in the Batphone app showing the Rhizome store, which could dynamically update showing incoming bundles and bundle expiry.

[lakeman]

The rhizome storage layer will now evict older and/or larger payloads when space is required to insert a new payload. Only the space of a single new payload is considered, we do not track the total set of all transfers in progress.
Two configuration options have been created to limit the size of the store; rhizome.database_size & rhizome.min_free_space.
None of the other suggested changes have been made to the way bundles are fetched.
See 47f0519, 5df5a33