pageserver: "could not find data for key" in test_scrubber_physical_gc_ancestors

[jcsp]

https://neon-github-public-dev.s3.amazonaws.com/reports/main/13196464234/index.html#testresult/628b11d330ec56c6/retries

 CRITICAL: could not ingest record at 0/15C4FF8\n')
Hint: use scripts/check_allowed_errors.sh to test any new allowed_error you add

Full error:

timeline_id=bddb5abb9b9209fd64bf7b9bba3d9c08}:connection{node_id=1}: CRITICAL: could not ingest record at 0/15C4FF8

Caused by:
    could not find data for key 020000000000000000000000000000000000 (shard ShardNumber(0)) at LSN 0/15C4FF9, request LSN 0/15C4FF8, ancestor 0/0

Tasks

Preview Give feedback

1. Fix pageserver: getpage requests sometimes skip reading recently written image layers #9185
2. Revert tests: temporarily permit a log error #10752

[jcsp]

Also seen for key 020000000000000000000000000000000000 in test_sharding_gc in #10741

[jcsp]

Very strangely, I can see the TWOPHASEDIR_KEY being included in image layer generation on non-zero shards as expected, but then subsequently being not found for read during ingest.

[jcsp]

On a failure of test_sharding_gc I can see that there is an image layer covering the key: 000000068000000000000017720000000001-030000000000000000000000000000000002__00000000016A8E98-v1-00000001

and with pagectl I can see that the key is indeed in the file.

However, the ingest code is trying to read at an older LSN than the image layer was created at.

[jcsp]

I think this is quite... niche. You need a shard that:

1. passes through some WAL region where there is no data for it
2. then gets a freeze_and_flush that advances its disk_consistent_lsn up to the highest LSN it has seen (the Advancing disk_consistent_lsn past WAL ingest gap code path)
3. then it does ingest some data, and opens an in-memory layer, but that in-memory layer starts from OpenLayerManager.next_open_layer_at, which is lower than where we actually ingested up to
4. then an image layer generation at disk_consistent_lsn, which now falls within the ephemeral layer range
5. then GC with pitr=0, which may remove layers below the image layer just generated
6. Finally, an ingest message type which tries to read a key that is not present in the ephemeral layer.

The reason this happens at all in certain tests is:

• they're hammering the pageserver with cycles of write,checkpoint,write,checkpoint -- so steps 1+2+3 are quite likely.
• then to exercise GC they intentionally do image layer generation and GC in quick succession

[jcsp]

disk_consistent_lsn is weird:

• We use it for backpressure (~10GB threshold)
• It doesn't literally mean what it used to mean: it can advance without flushing an L0 for shards.
• There's no such thing as "disk consistent" any more, we always load from remote storage.