Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[columnar] Raise the maximums for row counts #186

Merged
merged 1 commit into from
Nov 1, 2023
Merged

[columnar] Raise the maximums for row counts #186

merged 1 commit into from
Nov 1, 2023

Conversation

JerrySievert
Copy link
Contributor

implements #185.

What's changed?

columnar.chunk_group_row_limit and stripe_row_limit maximums set to 10000000.

@JerrySievert JerrySievert added the enhancement New feature or request label Oct 27, 2023
@JerrySievert JerrySievert requested a review from wuputah October 27, 2023 20:39
@JerrySievert JerrySievert self-assigned this Oct 27, 2023
@wuputah
Copy link
Member

wuputah commented Oct 30, 2023

is enforcing chunk_group_row_limit >= stripe_row_limit already implemented? just want to make sure we aren't opening up potential weird behavior.

@JerrySievert
Copy link
Contributor Author

just want to make sure we aren't opening up potential weird behavior.

we can always try and see what weird behavior it ends up allowing. we had the ability to mismatch these before this PR.

@wuputah
Copy link
Member

wuputah commented Nov 1, 2023

Yes, for sure; can you please test it?

we had the ability to mismatch these before this PR.

That's true, but I'd rather not replace a smaller foot-gun with a larger foot-gun, though both foot-guns would likely have the same result.

@JerrySievert
Copy link
Contributor Author

clickbench=# set columnar.chunk_group_row_limit = 1000000;
SET
clickbench=# set columnar.stripe_row_limit = 1000;
SET
clickbench=# create table hits using columnar as select * from hits_orig limit 5000000;
SELECT 5000000
clickbench=# \timing on
Timing is on.
clickbench=# select count(*) from hits;
  count
---------
 5000000
(1 row)

Time: 25.338 ms

alternately, with defaults:

clickbench=# create table hits using columnar as select * from hits_orig limit 5000000;
SELECT 5000000
clickbench=# \timing on
Timing is on.
clickbench=# select count(*) from hits;
  count
---------
 5000000
(1 row)

Time: 26.616 ms

@wuputah
Copy link
Member

wuputah commented Nov 1, 2023

thanks, so I guess it clamps the values at write time?

@JerrySievert JerrySievert merged commit 648b709 into main Nov 1, 2023
@JerrySievert JerrySievert deleted the limits branch November 1, 2023 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants