PS/Prefetch: Use a timeout for reading data from TCP

[MMeent]

This reduces pressure on OS TCP buffers, reducing flush times in other systems like PageServer.

Problem

Summary of changes

[github-actions]

7744 tests run: 7366 passed, 0 failed, 378 skipped (full report)

---

Code coverage* (full report)

• functions: 32.8% (8642 of 26359 functions)
• lines: 48.7% (73209 of 150476 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
e456cc9 at 2025-02-27T13:31:47.712Z :recycle:}

[github-actions]

If this PR added a GUC in the Postgres fork or neon extension,
please regenerate the Postgres settings in the cloud repo:

make NEON_WORKDIR=path/to/neon/checkout \
  -C goapp/internal/shareddomain/postgres generate

If you're an external contributor, a Neon employee will assist in
making sure this step is done.

[skyzh]

LGTM from the storage side

[hlinnaka]

This still makes me pretty squirmish. Are you sure you might not "re-enter" some parts of the prefetching code? The readpage_reentrant_guard is supposed to prevent that, but I can't convince myself that it's enough and that it's used in the right places. I don't see any direct bug or issue either though.

For example, can an interrupt happen while we're in a prefetch_register_bufferv() call? Is it OK? I don't see any CHECK_FOR_INTERRUPTS() calls there, but it's a long function, it's hard to tell at a quick glance.

Does the guard protect some specific fields of MyPState?

[hlinnaka]

prefetch_pump_state has this comment:

 * Note that this works because we don't pipeline non-getPage requests.

Took me a while to understand, but I got it now:

smgrnblocks()
 -> neon_nblocks()
   -> page_server_request()
     -> page_server->receive()
       -> pageserver_receive()
         -> call_PQgetCopyData()
           -> CHECK_FOR_INTERRUPTS()
             -> pagestore_smgr_processinterrupts()
               -> prefetch_pump_state()
                 -> page_server->try_receive()
                   -> PQgetCopyData()

The above cannot happen, because prefetch_pump_state() would find that there are no in-flight requests in the ring.

So that's OK, although feels a bit fragile.

[hlinnaka]

I don't see a concrete problem with this even though I'm a bit squirmish.

Let's add a GUC for this though. I'd love to have this in staging and pre-prod for a while, and perhaps do a slow roll out to production too, starting with the endpoints that have experienced problems with the socket buffers filling up.

I think a simple check in reconfigure_timeout_if_needed to do nothing if the GUC is not set would do the trick. Or since we're adding a GUC, perhaps make it an integer GUC to replace the PS_BACKGROUND_DELAY constant. Then we can also experiment making it more or less aggressive.

[MMeent]

@hlinnaka Did you mean something like this latest? (i.e. commit 2 of 3)