Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: stdcm: optimize incremental conflict detection #9980

Merged
merged 3 commits into from
Jan 7, 2025

Conversation

eckter
Copy link
Contributor

@eckter eckter commented Dec 6, 2024

Fix #9801
Helps with #10034 without entirely fixing it

Possible review by commit (though it's possible to just ignore the previous version and read the new one on its own):

  1. Split "static" and "incremental" conflict detection classes (the uses and possible optimizations differ quite a lot)
  2. Remove incremental routing conflict detection (wasn't used and brought lots of complexity, we can add it back later)
  3. Use range sets / tree maps for zone uses instead of a list of requirements

I haven't run a proper benchmark as it requires a pretty specific setup (thousands of trains in a realistic timetable), but I do have some metrics:

  • In the profiler, this section of the code goes from ~60% of execution time down to 2-5%
  • On the slowest request I've logged (uncapped), execution time goes from 330s down to 190s. Still a timeout but not by much (180s threshold)

This PR also reduces code complexity by quite a lot, and may fix some of the "mismatch" crashes.


edit: hold on, the new version still takes up a lot of time when it timeout (a lot more than 2-5% execution time). On one massive request, the time we spend in the conflict detection methods goes from 71% to 45% execution time: it's much better but still quite a lot. edit2: partially fixed by #10126.

@github-actions github-actions bot added the area:core Work on Core Service label Dec 6, 2024
@codecov-commenter
Copy link

codecov-commenter commented Dec 6, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.47%. Comparing base (a4190de) to head (1493c83).
Report is 4 commits behind head on dev.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #9980      +/-   ##
==========================================
- Coverage   81.48%   81.47%   -0.02%     
==========================================
  Files        1058     1058              
  Lines      104270   104270              
  Branches      722      722              
==========================================
- Hits        84969    84956      -13     
- Misses      19260    19273      +13     
  Partials       41       41              
Flag Coverage Δ
editoast 73.68% <ø> (-0.05%) ⬇️
front 89.21% <ø> (ø)
gateway 2.18% <ø> (ø)
osrdyne 3.28% <ø> (ø)
railjson_generator 87.50% <ø> (ø)
tests 87.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@eckter eckter force-pushed the ech/optimize-incr-conflict-detection branch 5 times, most recently from 7e3019f to 3287ab9 Compare December 17, 2024 13:45
@eckter eckter marked this pull request as ready for review December 17, 2024 14:30
@eckter eckter requested a review from a team as a code owner December 17, 2024 14:30
@eckter eckter requested a review from Khoyo December 17, 2024 14:30
@eckter eckter marked this pull request as draft December 18, 2024 09:01
@eckter eckter force-pushed the ech/optimize-incr-conflict-detection branch from 9883462 to 3287ab9 Compare December 18, 2024 13:17
@eckter eckter marked this pull request as ready for review December 18, 2024 13:18
Copy link
Contributor

@bougue-pe bougue-pe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this works, found the new code quite easy to read indeed 🙏
(some of my suggestions may tend to old code intricacy...)

Didn't review the old code and what might be missing from it.

As I understand it, we just don't check the routing requirements since STDCM v2, and this PR removes the check from v1. Which is fine by me (trusting you on a late non-incremental check for routing conflicts 🙈).

Copy link
Contributor

@Erashin Erashin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the routingRequirements + higherEntry really simplifies everything. Maybe we'll be able to do things more easily with this structure as well once/if we have to add the routing requirements back (it was really bloated with the previous way it was coded).
I'll let you deal with the other comments, but this looks pretty decent already.

@eckter eckter force-pushed the ech/optimize-incr-conflict-detection branch 2 times, most recently from 04e7379 to d9a0f84 Compare January 6, 2025 13:59
Copy link
Contributor

@bougue-pe bougue-pe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improved perf and much clearer to me too, thanks for all this, really helps to dive in this part 🙏

@eckter eckter force-pushed the ech/optimize-incr-conflict-detection branch from 66bad79 to 77d5703 Compare January 7, 2025 15:00
eckter added 3 commits January 7, 2025 16:21
Not used for now and duplicates code, but it enables optimizations

Signed-off-by: Eloi Charpentier <eloi.charpentier.42@gmail.com>
Signed-off-by: Eloi Charpentier <eloi.charpentier.42@gmail.com>
Signed-off-by: Eloi Charpentier <eloi.charpentier.42@gmail.com>
@eckter eckter force-pushed the ech/optimize-incr-conflict-detection branch from 77d5703 to 1493c83 Compare January 7, 2025 15:21
@eckter eckter enabled auto-merge January 7, 2025 15:23
@eckter eckter requested a review from Erashin January 7, 2025 16:16
Copy link
Contributor

@Erashin Erashin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gj!

@eckter eckter added this pull request to the merge queue Jan 7, 2025
Merged via the queue into dev with commit f441176 Jan 7, 2025
28 checks passed
@eckter eckter deleted the ech/optimize-incr-conflict-detection branch January 7, 2025 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core Work on Core Service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

core: stdcm: improve conflict detection performances
4 participants