Panic when releasing write lock

[kayleg]

From reading the oneshot channel code, this appears to happen when the waiting lock is dropped before it can be notified. (This is just a guess, but the best I could gleam even though I don't believe I have a case where this would happen in my project) The current error message and stack trace makes it impossible to find out which was the offending lock.

thread 'tokio-runtime-worker-5' panicked at 'Sender::send: ()', src/libcore/result.rs:1084:5

futures_locks::rwlock::RwLock<T>::unlock_writer 
at ./home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-locks-0.4.0/src/rwlock.rs:470

I also question if this is a good behavior as there is no way (without catching a panic) for the crate user to handle that error.

[kayleg]

Upon digging into the source, it looks like when any future returned by either .write() or .read() is dropped before gaining the lock, it will close the oneshot receiver, but does not remove the tx from the list of waiters. Then the tx makes its way to the front of the queue and send is invoked but it has already been closed.

[asomers]

Ooh, interesting. I suppose it would be too much to ask for the error to be reproducible?

[kayleg]

It's occurs in a massive codebase, randomly too. What I think is going on in my case is a gRPC client is dropping the connection to my tower-grpc server which then cancels the future that was waiting on the lock. If I get some down time, I'll try come up with a small reproducible case.

[asomers]

I've got a test case! Thanks for reporting this. I'll have a fix soon, probably by Monday at the latest.

[kayleg]

Thanks!! In my fork I tweaked the unlock to loop until it finds a viable receiver. It works, but definitely not the most optimum.