Explanation on BOTS file needed

[gethvi]

I believe I understand why BOTS file exists. However I do not understand why does it need to be exposed to the user in the intelmq/etc directory with other configuration files. User is not supposed to touch this file, however he can use it as a template for runtime.conf which is fine. But why does for instance intelmqctl check considers missing intelmq/etc/BOTS file as a Fatal error?!

I went through the codebase looking for code that uses intelmq/etc/BOTS file. I couldn't find any case where it would be actually preferable (in my opinion) to use it instead of the /usr/local/lib/python3.5/dist-packages/intelmq/bots/BOTS file which is part of the python package and thus "hidden" from the user. I believe that every occurence of code using intelmq/etc/BOTS actually doesn't expect the file to be modified which supports my claim for hiding it from the user.

I also tested running IntelMQ with intemq/etc/BOTS file having content of only empty json {} which satisfies checks and does not seem to break any runtime functionality.
(EDIT: It actually broke intelmq-manager from adding new bots. But again, this is a case where it is not expected that the file will be modified and therefore shouldn't be accessible to the user.)

But perhaps I am just missing something? If so, please enlighten me, I really don't understand the purpose of it being in intelmq/etc. Otherwise I would like to suggest the removal of the BOTS file from intelmq/etc and keeping it only inside the python package directory.

Thank you!

[ghost]

I believe I understand why BOTS file exists. However I do not understand why does it need to be exposed to the user in the intelmq/etc directory with other configuration files.

That file is topic for many discussions, a short list of them:

#1069
#972
#757
#668
#644
#552
#368

tldr: We should get rid of it.

As all components evolved over the last years, but that file did not really. It's kind of a central "registry", which is stored in the wrong location and can be substituted by some code. It also blocks several use-cases.

I agree with your conlusions.

User is not supposed to touch this file, however he can use it as a template for runtime.conf which is fine. But why does for instance intelmqctl check considers missing intelmq/etc/BOTS file as a Fatal error?!

It's not a hard requirement for the core, but necessary for the IntelMQ Manager and intelmqctl check (to check if the installation has errors). Thus, I would consider it (currently) a requirement.

Otherwise I would like to suggest the removal of the BOTS file from intelmq/etc and keeping it only inside the python package directory.

Instead of changing the file location, I'd prefer resolving the issues linked above all together.

Personally, I would like to see that changed in 3.0, but it's currently not part of the 3.0 plan (https://lists.cert.at/pipermail/intelmq-users/2020-May/000162.html / /~https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md)

[gethvi]

Thanks for your answer.

I really like the approach suggested by @sykaeh in #757. I could try to follow up on her ideas and her code for 3.0? Maybe even 2.x, it could be included side by side with the current solution until 3.0.

[aaronkaplan]

Personally, I would like to see that changed in 3.0, but it's currently not part of the 3.0 plan (https://lists.cert.at/pipermail/intelmq-users/2020-May/000162.html / /~https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md)

Then let's add it...

[aaronkaplan]

It is in there.