Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: (core) Add circuit breaker pattern for database operations #719

Closed

Conversation

augchan42
Copy link
Contributor

Implements circuit breaker pattern to handle database failures gracefully and prevent cascading failures. Fixes #712.

Changes:

  • Adds CircuitBreaker class with CLOSED, OPEN, and HALF-OPEN states
  • Introduces BaseCircuitBreakerAdapter for database adapters
  • Configurable failure thresholds and recovery timeouts
  • Automatic recovery attempts in HALF-OPEN state
  • Detailed logging of circuit breaker state changes

Circuit breaker configuration:

  • Opens after 5 consecutive failures (configurable)
  • Resets after 60 seconds in OPEN state
  • Requires 3 successful operations in HALF-OPEN state to close

This helps prevent overwhelming failed database connections and provides graceful degradation during outages.

Discord username

hosermage

@shakkernerd
Copy link
Member

shakkernerd commented Nov 30, 2024

Hi, I like the idea but it changes the overall use of DatabaseAdapter. Hence, I think might cause breaking changes.
Is there anyway you could implement this into the DatabaseAdapter class itself?

@augchan42
Copy link
Contributor Author

augchan42 commented Dec 1, 2024 via email

@shakkernerd
Copy link
Member

Great, now its better.
I will test locally and then merge in if no issue.

@lalalune
Copy link
Member

lalalune commented Dec 2, 2024

lgtm, will leave for @shakkernerd to merge
We're moving fast, so there are some small merge conflicts in package lock, whoever is maintaining can just fix and merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Add circuit breaker pattern for database operations
3 participants