Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VOM should track recognizable vrefs (conservatively), inhibit syscall.retireImport #3161

Closed
warner opened this issue May 24, 2021 · 0 comments · Fixed by #3409
Closed

VOM should track recognizable vrefs (conservatively), inhibit syscall.retireImport #3161

warner opened this issue May 24, 2021 · 0 comments · Fixed by #3409
Assignees
Labels
enhancement New feature or request SwingSet package: SwingSet

Comments

@warner
Copy link
Member

warner commented May 24, 2021

What is the Problem Being Solved?

#3106 is about reducing RAM usage for unreachable objects. The next phase after that is to reduce disk usage by pruning unnecessary DB entries, starting with c-list entries for objects that no longer need their identity tracked.

The low-hanging fruit is the vref for a Presence which was imported into a vat, never used as the key of a WeakMap or WeakSet, and is then released. This will trigger a syscall.dropImport when the Presence is collected, but it could also trigger syscall.retireImport. That would allow the kernel to delete the importing vat's c-list entry and reduce the recognizability count. Once that count drops to zero, the kernel could notify the exporting vat with dispatch.retireExport and then delete the export-side c-list entry.

To do this reliably, the virtual object manager must be able to tell liveslots whether a given unreachable vref is recognizeable or not. We'll start with a conservative approach that simply remembers (in a strong Set) whether a given vref has ever been used as the key of a WeakMap/WeakSet. If so, the vat never retires the import. This conservative approach would never remove the item, even if the entry was manually deleted from every WeakMap/WeakSet, and/or all those maps/sets were collected themselves.

This should let us retire the vast majority of vrefs (we don't use any Presences as weak-collection keys right now, only Representatives, in the ERTP paymentLedger). However it would create a RAM burden of one vref string per Payment object, because the Set lives in RAM.

The second step is to move the flag onto disk, removing the RAM burden and replacing it with a (somewhat larger) disk burden. Something like vatstoreSet('${vref}.recognizable', 1). I think we're using more Presences overall than Payments, so expect it would be a net improvement, but we should have a way to measure it.

A much later stage would be to replace the conservative "once added, never removed" approach with more detailed refcounting. This might take the form of a list of weak-thing tables that use the key: vatstoreSet('${vref}.recognizable', JSON.stringify(['weakmap1', 'weakset3'])), which would be most efficient if userspace tends to have a small number of weak tables (and it's ok if they hold a very large number of keys). Or we flip it around, use one vatstore key per weak table, and give each one a list of vrefs used as keys (better if userspace has a large number of weak tables, but bad if they have a large number of keys). Or we find some more efficient scheme. Maybe a Bloom Filter?

We won't need to implement any specific cycle-handling code to make this less conservative, however it won't retire as much as it possibly can until we do handle cycles in the virtualized-data reference graph, because otherwise the entries can't be deleted and their keys will remain recognizable.

Test Plan

Normal unit tests: send a Presence into a vat, use it (or not) as a WeakMap and WeakSet key (test both!), tell the vat to drop its strong reference to the Presence, then watch to see whether the vat does both syscall.dropImport and syscall.retireImport, or only just syscall.dropImport.

@warner warner added enhancement New feature or request SwingSet package: SwingSet labels May 24, 2021
@warner warner self-assigned this May 24, 2021
warner added a commit that referenced this issue May 24, 2021
Change liveslots to provoke GC at mid-crank, then process the "dead set" and
emit `syscall.dropImports` for everything we can.

For now, we conservatively assume that everything remains recognizable, so we
do *not* emit `syscall.retireImport` for anything. The vref might be used as
a WeakMap/WeakSet key, and the VOM isn't yet tracking those. When #3161 is
done, we'll change liveslots to ask the VOM about each vref, and retire the
ones it does not know how to recognize (which will hopefully be the majority
of them).

closes #3147
refs #2615
refs #2660
warner added a commit that referenced this issue May 25, 2021
Change liveslots to provoke GC at mid-crank, then process the "dead set" and
emit `syscall.dropImports` for everything we can.

For now, we conservatively assume that everything remains recognizable, so we
do *not* emit `syscall.retireImport` for anything. The vref might be used as
a WeakMap/WeakSet key, and the VOM isn't yet tracking those. When #3161 is
done, we'll change liveslots to ask the VOM about each vref, and retire the
ones it does not know how to recognize (which will hopefully be the majority
of them).

closes #3147
refs #2615
refs #2660
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request SwingSet package: SwingSet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants