Make changelog polling signals pluggable #3
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These were previously calculated where needed but they ended up being needed in multiple locations.
The current CDC engine implementation embeds the file watching logic. This change extracts that logic, along with some alternative implementations, into a new interface that can be used by any engine implementation.
This fixes a timing condition where the watch loop of a signal could still be waiting to write an event when the channel is closed. This condition resulted in a panic due to the closed channel write. Now, the channels are never closed but the goroutines are guaranteed to exit once the signals are stopped by selecting all writes along with reads of the closer channel.
The trigger CDC engine implementation previously embedded the FS watcher logic. This is factored out and replaced with customizable signals. The default signal is now set to a combination of the FS watcher and a timer after discovering that some resource constrained or containerized environments don't always emit FS events consistently. I didn't get to an exact root cause but the tests would intermittently pass or fail on some, but not all, machines.
The bootstrap events are useful for systems that accumulate state based on the CDC stream and then later encounter a critical fault. The standard recovery is to bootstrap but bootstrap records are otherwise identical to INSERT change records. This leaves the consuming system with no way of knowing whether any currently persisted state is invalid such as a record that was deleted but the system failed to correctly process the delete. In this case, a bootstrap event can be a signal to purge any persisted state relevant to a table before handling the bootstrap events.
The CLI for setup/teardown of triggers is now a different exectuable from the CLI for processing changes. This is to help limit the scope of the CLI when running change processing as a side-car process to some other system that stores data in SQLite.
kevinconway
changed the title
|
I think there's some kind of caching issue with the build results. I could reproduce the linter issue locally, fixed the issue, and can now no longer reproduce. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Test failures in some environments suggested that either the file watcher or the signal handling code is fallible in some way that results in the system not polling after the SQLite file changes. To account for this, I've refactored the file watcher into a pluggable interface and added a time based polling implementation to use alongside file watching or in environments where file watching isn't possible.
Additionally, this change adds a suite of benchmarks to measure the overhead of the triggers on insert, update, and delete operations. A summary of the results is included in the README.