You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#3106 is about reducing RAM usage for unreachable objects. The next phase after that is to reduce disk usage by pruning unnecessary DB entries, starting with c-list entries for objects that no longer need their identity tracked.
The low-hanging fruit is the vref for a Presence which was imported into a vat, never used as the key of a WeakMap or WeakSet, and is then released. This will trigger a syscall.dropImport when the Presence is collected, but it could also trigger syscall.retireImport. That would allow the kernel to delete the importing vat's c-list entry and reduce the recognizability count. Once that count drops to zero, the kernel could notify the exporting vat with dispatch.retireExport and then delete the export-side c-list entry.
To do this reliably, the virtual object manager must be able to tell liveslots whether a given unreachable vref is recognizeable or not. We'll start with a conservative approach that simply remembers (in a strong Set) whether a given vref has ever been used as the key of a WeakMap/WeakSet. If so, the vat never retires the import. This conservative approach would never remove the item, even if the entry was manually deleted from every WeakMap/WeakSet, and/or all those maps/sets were collected themselves.
This should let us retire the vast majority of vrefs (we don't use any Presences as weak-collection keys right now, only Representatives, in the ERTP paymentLedger). However it would create a RAM burden of one vref string per Payment object, because the Set lives in RAM.
The second step is to move the flag onto disk, removing the RAM burden and replacing it with a (somewhat larger) disk burden. Something like vatstoreSet('${vref}.recognizable', 1). I think we're using more Presences overall than Payments, so expect it would be a net improvement, but we should have a way to measure it.
A much later stage would be to replace the conservative "once added, never removed" approach with more detailed refcounting. This might take the form of a list of weak-thing tables that use the key: vatstoreSet('${vref}.recognizable', JSON.stringify(['weakmap1', 'weakset3'])), which would be most efficient if userspace tends to have a small number of weak tables (and it's ok if they hold a very large number of keys). Or we flip it around, use one vatstore key per weak table, and give each one a list of vrefs used as keys (better if userspace has a large number of weak tables, but bad if they have a large number of keys). Or we find some more efficient scheme. Maybe a Bloom Filter?
We won't need to implement any specific cycle-handling code to make this less conservative, however it won't retire as much as it possibly can until we do handle cycles in the virtualized-data reference graph, because otherwise the entries can't be deleted and their keys will remain recognizable.
Test Plan
Normal unit tests: send a Presence into a vat, use it (or not) as a WeakMap and WeakSet key (test both!), tell the vat to drop its strong reference to the Presence, then watch to see whether the vat does both syscall.dropImport and syscall.retireImport, or only just syscall.dropImport.
The text was updated successfully, but these errors were encountered:
Change liveslots to provoke GC at mid-crank, then process the "dead set" and
emit `syscall.dropImports` for everything we can.
For now, we conservatively assume that everything remains recognizable, so we
do *not* emit `syscall.retireImport` for anything. The vref might be used as
a WeakMap/WeakSet key, and the VOM isn't yet tracking those. When #3161 is
done, we'll change liveslots to ask the VOM about each vref, and retire the
ones it does not know how to recognize (which will hopefully be the majority
of them).
closes#3147
refs #2615
refs #2660
Change liveslots to provoke GC at mid-crank, then process the "dead set" and
emit `syscall.dropImports` for everything we can.
For now, we conservatively assume that everything remains recognizable, so we
do *not* emit `syscall.retireImport` for anything. The vref might be used as
a WeakMap/WeakSet key, and the VOM isn't yet tracking those. When #3161 is
done, we'll change liveslots to ask the VOM about each vref, and retire the
ones it does not know how to recognize (which will hopefully be the majority
of them).
closes#3147
refs #2615
refs #2660
What is the Problem Being Solved?
#3106 is about reducing RAM usage for unreachable objects. The next phase after that is to reduce disk usage by pruning unnecessary DB entries, starting with c-list entries for objects that no longer need their identity tracked.
The low-hanging fruit is the vref for a Presence which was imported into a vat, never used as the key of a WeakMap or WeakSet, and is then released. This will trigger a
syscall.dropImport
when the Presence is collected, but it could also triggersyscall.retireImport
. That would allow the kernel to delete the importing vat's c-list entry and reduce the recognizability count. Once that count drops to zero, the kernel could notify the exporting vat withdispatch.retireExport
and then delete the export-side c-list entry.To do this reliably, the virtual object manager must be able to tell liveslots whether a given unreachable vref is recognizeable or not. We'll start with a conservative approach that simply remembers (in a strong Set) whether a given vref has ever been used as the key of a WeakMap/WeakSet. If so, the vat never retires the import. This conservative approach would never remove the item, even if the entry was manually deleted from every WeakMap/WeakSet, and/or all those maps/sets were collected themselves.
This should let us retire the vast majority of vrefs (we don't use any Presences as weak-collection keys right now, only Representatives, in the ERTP
paymentLedger
). However it would create a RAM burden of one vref string perPayment
object, because theSet
lives in RAM.The second step is to move the flag onto disk, removing the RAM burden and replacing it with a (somewhat larger) disk burden. Something like
vatstoreSet('${vref}.recognizable', 1)
. I think we're using more Presences overall than Payments, so expect it would be a net improvement, but we should have a way to measure it.A much later stage would be to replace the conservative "once added, never removed" approach with more detailed refcounting. This might take the form of a list of weak-thing tables that use the key:
vatstoreSet('${vref}.recognizable', JSON.stringify(['weakmap1', 'weakset3']))
, which would be most efficient if userspace tends to have a small number of weak tables (and it's ok if they hold a very large number of keys). Or we flip it around, use one vatstore key per weak table, and give each one a list of vrefs used as keys (better if userspace has a large number of weak tables, but bad if they have a large number of keys). Or we find some more efficient scheme. Maybe a Bloom Filter?We won't need to implement any specific cycle-handling code to make this less conservative, however it won't retire as much as it possibly can until we do handle cycles in the virtualized-data reference graph, because otherwise the entries can't be deleted and their keys will remain recognizable.
Test Plan
Normal unit tests: send a Presence into a vat, use it (or not) as a WeakMap and WeakSet key (test both!), tell the vat to drop its strong reference to the Presence, then watch to see whether the vat does both
syscall.dropImport
andsyscall.retireImport
, or only justsyscall.dropImport
.The text was updated successfully, but these errors were encountered: