Larger Files

[elcarn23]

Does cockpit provide a way to parse or handle files that are around 20MB or so from the local file system? Or should we use other methods? We're currently trying to read an XML file that contains a lot of meta information and reporting data and we get the error that there is "Too much data".

[martinpitt]

Can you please be a little more specific what you are trying to do, and where? Is that some standard cockpit page, or a custom one of yours? cockpit.file(...).read() should work fine with 20 MB, provided that you set an appropriate max_read_size (the default is 16 MiB, see docs).

But indeed things get increasingly inefficient if you download that much data into JS and the page's working memory. At some point it's probably better to process the XML data on the server side, by running some program/script through cockpit.spawn().

[elcarn23]

Thanks for the response. I had missed the part where you could set a max_read_size. We are just using the cockpit.file().read() to open it and extract some information needed for the first case and the other is when our plugin generates an XML file on the host we wanted a straightforward way for the end-user to download that file from the plugin. I will raise the limit for now. We aren't going to be going much beyond that and generally we'll be below that threshold. Thank you again for your answer.

[elcarn23]

One last thing on that max read size. The value it takes, is it in bytes, kb, or mb?

[elcarn23]

Never mind figured it out. Thanks for all your help. My apologies again for missing the flag in the documentation.

[martinpitt]

No worries, I'm glad that it's working now! Indeed the documentation could be a bit clearer on the unit. FTR, it's plain bytes. I'll send a PR.

[martinpitt]

I sent PR #17119 to improve the documentation, comments welcome.