Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

addBatch write timed out #991

Closed
imyoubo opened this issue Jul 11, 2022 · 6 comments
Closed

addBatch write timed out #991

imyoubo opened this issue Jul 11, 2022 · 6 comments
Labels
Milestone

Comments

@imyoubo
Copy link

imyoubo commented Jul 11, 2022

image
image

I batch write clickhouse through jdbc client.
But I ran into a problem while writing it.
When addBatch is executed about 2000 times, it is blocked, stuck there and cannot continue, and after exceeding the set socket_timeout, an exception of Write timed out is reported.
It's weird that the same code ru.yandex.clickhouse.ClickHouseDriver doesn't have this problem

The version is 0.3.2-patch10
Please help with this issue, thanks!

我通过jdbc client 批量写clickhouse。
但是在写的时候遇到了一个问题。
当addBatch 执行了大约2000次左右时,就阻塞住了,卡在那里无法继续,然后超过设置的socket_timeout后,报Write timed out 异常了。
很奇怪,同样的代码 ru.yandex.clickhouse.ClickHouseDriver 没有出现这个问题

版本是 0.3.2-patch10
请帮忙看一下这个问题,谢谢!

@imyoubo
Copy link
Author

imyoubo commented Jul 11, 2022

My program is executeBatch every 25w records
But every time after executing addBatch about 2000 times, it blocks, let alone executeBatch

我的程序是每25w条记录执行一次executeBatch
但是每次都是执行了2000次左右addBatch后,它就阻塞了,更没有执行过 executeBatch

@zhicwu
Copy link
Contributor

zhicwu commented Jul 11, 2022

Hi @imyoubo, could you change max_queued_buffers to zero(unlimited) and try again? For example: jdbc:ch://localhost?max_queued_buffers=0.

@zhicwu zhicwu added the bug label Jul 11, 2022
@zhicwu
Copy link
Contributor

zhicwu commented Jul 11, 2022

I'll remove the size limit in JDBC driver, so that nobody has to set the parameter explicitly.

@imyoubo
Copy link
Author

imyoubo commented Jul 11, 2022

Hi @imyoubo, could you change max_queued_buffers to zero(unlimited) and try again? For example: jdbc:ch://localhost?max_queued_buffers=0.

oh I tried it just now, and it really solved it. It seems to be the configuration problem of max_queued_buffers, thank you very much

@imyoubo
Copy link
Author

imyoubo commented Jul 12, 2022

Hi @zhicwu There seems to be a new problem, When executing executeBatch, an error is reported, "Please call addBatch method at least once before batch execution"
image

@imyoubo
Copy link
Author

imyoubo commented Jul 12, 2022

Is #977

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants