data: Replace custom formats with msgpack #374
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have previously attempted to refactor data converters used in data.py in this branch. In the end I wasn't happy with the result, because I believe that writing custom parsers for each database is the wrong approach.
This PR replaces all data.py parsers with msgpack. Plain python objects can be serialized and deserialized into the databases.
The main advantage is convenience - values can be manipulated like normal Python objects, no string parsing is required anywhere in the codebase that interacts with the database.
From what I remember, larger databases were also a bit smaller, mainly because large ints take less space in msgpack than in base10 representation. But to be fair, there is some storage overhead for other datatypes.
I also wouldn't be surprised if average serialization/deserialization times were a bit smaller, although I don't have numbers on that and I doubt it's a major bottleneck anywhere.
Leaving this as a draft - I tested it only a little bit.