for #10993: cancellation sensitivity for buffered writer

[problame]

In

• buffered writer: handle write errors by retrying all write IO errors indefinitely #10993 (comment)

I added infinite retries for buffered writer flush IOs, primarily to gracefully handle ENOSPC but more generally so that the buffered writer is not left in a state where reads from the surrounding InMemoryLayer cause panics, but stall instead until the read succeeds. See the #10993 description for details.

However, I didn't add cancellation sensitivity, which is concerning because then there is no way to detach a timeline/tenant tenant that is encountering the problem.

This PR

• addresses a TODO that highlights that flush loop is in fact infallible with the infinite retries
• adds sensitivity to Timeline::cancel to the flush loop
• and fixes the InMemoryLayer/EphemeralFile/BufferedWriter almagamate to remain read-available after flush loop is cancelled.

The support for read-availability after cancellation is is necessary so that reads from the InMemoryLayer that are already queued up behind the RwLock that wraps the BufferedWriter won't panic because of the mutable=None that we leave behind in case the flush loop gets cancelled.

Alternatives

One might think that we can drop #10993 + the cancellation sensitivity added in this PR, using only the changes that add read-availability after flush error.

The problem here is that read-availability sounds good but is really quite useless, because we cannot ingest new WAL and hence very soon, reads from compute are going to wait indefinitely on wait_lsn, because ingest isn't progressing.

Thus, having the infinite flush retry still retries still make more sense because they're just "slowness" to the user

This avoids any sort of panic when detaching a
Remaining read-available allows in-flight getpage

One might think

The problem with cancellation sensitivity is the same as with handling IO errors.
is that it's basically