Fixes certtools/intelmq#668.

[sykaeh]

Reorganization of BOTS file and more documentation

[sykaeh]

The build failed because the order of the collectors is not alphabetical in the BOTS failed. I changed it to be generic collectors and then specific collectors (as discussed in #668). Should I change it to be alphabetical again? Or do we want to change the test?

[dmth]

Thanks for your work!
From my point of view the test is not useful. On the contrary: It prohibits a user freindly ordering of fields, for instance: First: Username, Second: Password

[sykaeh]

Yes, I agree that the ordering can be done more user friendly when it is not alphabetical. However, I do think that a test to check whether the BOTS file is valid JSON should be done.

Are there any other checks that would be useful? Possibly a check to see whether there is a section for all of the bot types (i.e. collectors, parsers, outputs, experts)?

[sebix]

Yes, I agree that the ordering can be done more user friendly when it is not alphabetical. However, I do think that a test to check whether the BOTS file is valid JSON should be done.

Previously (before I enforced indentation, sorting etc) the file was a mess. I just don't want to get back there. If you think it's not useful anymore (the list is much smaller now too), then we can drop the sorting for the BOTS-file.

Are there any other checks that would be useful? Possibly a check to see whether there is a section for all of the bot types (i.e. collectors, parsers, outputs, experts)?

A check if the module actually exists and the the description exists and is not empty. For collectors: Parameter feed exists.

[sykaeh]

A check if the module actually exists and the the description exists and is not empty. For collectors: Parameter feed exists.

Where exactly is the description used? Most of the parsers have the same nondescript text anyways: " is the bot responsible to parse the report and sanitize the information." Wouldn't it make more sense to move the description of the bots to the documentation instead of this file (like the docs/Bots.md file)?

[sebix]

Where exactly is the description used? Most of the parsers have the same nondescript text anyways: " is the bot responsible to parse the report and sanitize the information."

Yes, it's in mostly pretty useless unfortunately.

Wouldn't it make more sense to move the description of the bots to the documentation instead of this file (like the docs/Bots.md file)?

The docs/Bots.md file is also not a good place. We had the idea to save the following data per bot into a readme or as docstring:

• parameters+descriptions+default value
• description (i.e. readme)

Then we can always generate a combined documentation (sphinx et al.) and the bots always have their default values and they can easily be upgraded.

Also see #368 and #644

I will have time to address these issues as recently as in October.

[aaronkaplan]

---

Mobile

On 22.09.2016, at 20:02, Sybil Ehrensberger notifications@github.com wrote:

A check if the module actually exists and the the description exists and is not empty. For collectors: Parameter feed exists.

Where exactly is the description used? Most of the parsers have the same nondescript text anyways: " is the bot responsible to parse the report and sanitize the information." Wouldn't it make more sense to move the description of the bots to the documentation instead of this file (like the docs/Bots.md file)?

+1

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[aaronkaplan]

---

Mobile

On 22.09.2016, at 17:15, Sybil Ehrensberger notifications@github.com wrote:

Yes, I agree that the ordering can be done more user friendly when it is not alphabetical. However, I do think that a test to check whether the BOTS file is valid JSON should be done.

Are there any other checks that would be useful? Possibly a check to see whether there is a section for all of the bot types (i.e. collectors, parsers, outputs, experts)?

• Valid JSON syntax check

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[sebix]

• Valid JSON syntax check
  Once we parse it to check anything (indentation, existence of fields),
  we always check the syntax implicitly.

[aaronkaplan]

---

Mobile

On 23.09.2016, at 11:20, Sebastian notifications@github.com wrote:

• Valid JSON syntax check
  Once we parse it to check anything (indentation, existence of fields),
  we always check the syntax implicitly.

Excellent.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[bernhardreiter]

@sebix, you wrote:

The docs/Bots.md file is also not a good place. We had the idea to save the following data per bot [..]

If we separate the documentation, we should also separate the rest of the bot infos and complete #552. However this may be too much for the 1.0 release and you did already tag #552 for 1.1, so my suggestion for 1.0 and this issue at hand is: put documentation either into docs/Bots.md (to be moved later when everything is moved with #552) or link stuff from docs/Bots.md to the README.md in the directory (I think I will do so for for two files while fixing #711).

[coveralls]

Coverage increased (+0.01%) to 71.873% when pulling 933595b on sykaeh:issue-long-bots-file into 37a3db9 on certtools:master.

[codecov-io]

Current coverage is 71.87% (diff: 100%)

Merging #706 into master will increase coverage by 0.01%

@@             master       #706   diff @@
==========================================
  Files           209        209          
  Lines          7673       7676     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           5514       5517     +3   
  Misses         2159       2159          
  Partials          0          0          

Powered by Codecov. Last update 37a3db9...933595b