go-kvfile stores key/value pairs to a file with O(log N) (binary search) lookup.
The values are concatenated together at the beginning of the file, followed by a set of length-suffixed entries containing each key and the offset of the associated value, followed by a list of positions of index entries.
The compress package supports seekable-zstd compressed kvfiles.
The kvfile CLI can be used to read/write a kvfile on the command line:
NAME:
kvfile - A CLI tool for working with key-value files
COMMANDS:
count Print the number of keys in a k/v file.
keys Print all keys in a k/v file in sorted order.
values Print all key-value pairs in a k/v file.
get Get the value for a specific key.
write Write a new kvfile from JSON input.
GLOBAL OPTIONS:
--binary-keys read and log keys as binary (base58) (default: false)
--binary-values read and log values as binary (base58) (default: true)
--file value, -f value path to the kvfile to read
--compress use kvfile compression (default: false)
The Reader reads values from the kvfile and can search for specific keys using a binary search on the key index:
// Read keys from the file.
Get(): Looks up the value for the given key.
ReadTo(): Reads the value for the given key to the writer.
Exists(): Checks if the given key exists in the store.
// Iterate over keys in the file.
ScanPrefix(): iterates over key/value pairs with a prefix.
ScanPrefixKeys(): iterates over key/value pairs with a prefix, returning keys only.
// Utilities for reading the file structure.
ReadIndexEntry(): Reads the index entry at the given index.
SearchIndexEntry(): Looks up an index entry for the given key.
GetValuePosition(): Determines the position and length of the value for the key.
Write() writes the given key-value pairs to the file with the writer.
var buf bytes.Buffer
keys := [][]byte{
[]byte("test-1"),
[]byte("test-2"),
[]byte("test-3"),
}
vals := [][]byte{
[]byte("val-1"),
[]byte("val-2"),
[]byte("val-3"),
}
var index int
err := Write(&buf, keys, func(wr io.Writer, key []byte) (uint64, error) {
nw, err := wr.Write(vals[index])
if err != nil {
return 0, err
}
index++
return uint64(nw), nil
})The Writer can be used to incrementally write keys and values to a file.
var buf bytes.Buffer
keys := [][]byte{
[]byte("test-2"),
[]byte("test-3"),
[]byte("test-1"),
}
vals := [][]byte{
[]byte("val-2"),
[]byte("val-3"),
[]byte("val-1"),
}
wr := NewWriter(&buf)
for i, key := range keys {
if err := wr.WriteValue(key, bytes.NewReader(vals[i])); err != nil {
t.Fatal(err.Error())
}
}
if err := wr.Close(); err != nil {
t.Fatal(err.Error())
}Please open a GitHub issue with any questions / issues.
... or feel free to reach out on Matrix Chat or Discord.
MIT