Couldn't receive txs expected value at line 1 column 1

[h10086733]

When ord executes index run, Couldn't receive txs expected value at line 1 column 1 often occurs, which often occurs when the full amount of data is synchronized. In the mode of index-sats index run, the amount of data has almost reached 100G, and every It takes a lot of time to load index.rdb into memory for the first time, so more retries are needed to reduce reloading when exception handling occurs. And every time the database is upgraded, index.rdb needs to be reloaded. Whether to consider splitting Optimizing processing in the form of historical rdb files.

[nick07002]

I have the same issue:
[indexing blocks] ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████░ 795724/799050[2023-07-17T11:19:28Z ERROR ord::index::updater] Couldn't receive txs connection closed before message completed
error: channel closed

[victorkirov]

Same here. Mine happened on block 790593 on a full index with SATS.

[raphjaph]

That error occurs when fetching transactions but don't think it impacts performance in a large way.

And every time the database is upgraded, index.rdb needs to be reloaded. Whether to consider splitting Optimizing processing in the form of historical rdb files.

Every time the database schema changes we have to rebuild the database. Would be great to implement database migration but that could be quite involved.

Generally there is a lot of optimizations that can still be done for indexing. Will try to enumerate in the coming weeks so people can take a stab at it.

[victorkirov]

I traced the error to the fetcher. I didn't put in a log, so not 100% sure what bitcoin core is returning, but when transactions are fetched from the Bitcoin core, it seems to return some sort of error string instead of json. When ord tries to deserialize the response, it returns the "expected value at line 1 column 1" error.

The Bitcoin node seems to be overloaded maybe, as the Electrs api running off the same node has issues around the same time.

I wrapped the transaction fetch logic in a retry loop and the error seemed to stop 🎉

[nick07002]

I wrapped the transaction fetch logic in a retry loop and the error seemed to stop 🎉

that's really nice @victorkirov mind sharing the location where you made the change? I probably need to do the same.
appreciate it.

[victorkirov]

Sorry, I'm on my phone 🙈 It's line 79 here: /~https://github.com/ordinals/ord/blob/master/src/index/fetcher.rs

The response is most likely an error string instead of json.

I stuck lines 68 to 79 in a method and called in in a retry loop that retries 3 times.

If you do it, please add a log of the response body so we can see what it is 🙏

I can post the code a bit later today if you want.