Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared polling topic listener #496

Closed
steven-sheehy opened this issue Jan 14, 2020 · 0 comments · Fixed by #515
Closed

Shared polling topic listener #496

steven-sheehy opened this issue Jan 14, 2020 · 0 comments · Fixed by #515
Assignees
Labels
enhancement Type: New feature grpc Area: GRPC API P2
Milestone

Comments

@steven-sheehy
Copy link
Member

Problem
If there's a large number of subscribers (especially for the same topic), then polling each topic separately will be very inefficient and cause database contention. Also, the existing polling topic listener can't poll as quickly since there are so many of them, increasing message latency.

Solution
Use a single, shared polling topic listener.

  • Implement a SharedPollingTopicListener that uses the existing logic in PollingTopicListener except moves most of the Flux definition to the constructor and calls Flux.share().
  • Instead of calling repository.findByFilter(), a new method repository.findByConsensusTimestampGreaterThan(long consensusTimestamp) will be created to return results for all topics.
  • Ensure it uses an index or create one if necessary.
  • Filtering will be done in memory.
  • The interval can be set to occur more often like 500ms.

Alternatives
Replace the existing PollingTopicListener with the new approach.

Additional Context

@steven-sheehy steven-sheehy added enhancement Type: New feature P2 grpc Area: GRPC API labels Jan 14, 2020
@steven-sheehy steven-sheehy self-assigned this Jan 29, 2020
@steven-sheehy steven-sheehy added this to the Mirror 0.5.3 milestone Jan 29, 2020
@steven-sheehy steven-sheehy mentioned this issue Jan 30, 2020
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Type: New feature grpc Area: GRPC API P2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant