Errors when already configured bots gain additional options through upgrade

[gsiv]

It appears that when bots receive additional options through an upgrade,
previously configured bots produce errors because of those new, unconfigured
options.

At least, this is what happened after the mail collector bots gained the two
new chunking options:

2016-08-05 17:38:13,111 - Shadowserver-Open-SNMP - INFO - Reading email report
2016-08-05 17:38:13,112 - Shadowserver-Open-SNMP - INFO - Downloading report from https://[…]
2016-08-05 17:38:15,883 - Shadowserver-Open-SNMP - INFO - Report downloaded.
2016-08-05 17:38:15,884 - Shadowserver-Open-SNMP - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 121, in start
    self.process()
  File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/mail/collector_mail_url.py", line 89, in process
    for report in generate_reports(template, io.BytesIO(resp.content), self.parameters.chunk_size,
AttributeError: 'Parameters' object has no attribute 'chunk_size'
2016-08-05 17:38:15,884 - Shadowserver-Open-SNMP - INFO - Current Message(event): None.
2016-08-05 17:38:15,885 - Shadowserver-Open-SNMP - INFO - Bot will restart in 15 seconds.
2016-08-05 17:38:18,144 - Shadowserver-Open-SNMP - ERROR - Received KeyboardInterrupt.
2016-08-05 17:38:18,145 - Shadowserver-Open-SNMP - INFO - Bot stopped.
2016-08-05 17:38:18,145 - Shadowserver-Open-SNMP - ERROR - Exiting.
2016-08-05 18:02:27,511 - Shadowserver-Open-SNMP - INFO - MailURLCollectorBot initialized with id BSI-Shadowse
ver-Open-SNMP and version 3.4.3 (default, Oct 14 2015, 20:28:29) as process 11604.
2016-08-05 18:02:27,512 - Shadowserver-Open-SNMP - INFO - Bot is starting.
2016-08-05 18:02:27,515 - Shadowserver-Open-SNMP - INFO - Bot starts processings.
2016-08-05 18:02:27,516 - Shadowserver-Open-SNMP - INFO - Pipeline ready.
2016-08-05 18:02:27,516 - Shadowserver-Open-SNMP - INFO - Start processing.
2016-08-05 18:02:27,611 - Shadowserver-Open-SNMP - INFO - Reading email report
2016-08-05 18:02:27,611 - Shadowserver-Open-SNMP - INFO - Downloading report from https://[…]
2016-08-05 18:02:30,578 - Shadowserver-Open-SNMP - INFO - Report downloaded.
2016-08-05 18:02:30,581 - Shadowserver-Open-SNMP - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 121, in start
    self.process()
  File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/mail/collector_mail_url.py", line 90, in process
    self.parameters.chunk_replicate_header):
AttributeError: 'Parameters' object has no attribute 'chunk_replicate_header'
2016-08-05 18:02:30,581 - Shadowserver-Open-SNMP - INFO - Current Message(event): None.

I don't know whether this is a problem specific to this bot or possibly a generic
issue.

[sebix]

Yes, it's totally a generic issue and I'd solve it in conjunction with #267. However should we fix it prior to 1.1 too?

[sebix]

These options came to my mind:

1. Each bot has a defaults.conf in its dir providing the defaults values. This is shipped directly in the program itself, thus always exists. bots/BOTS can then be constructed using these single files. Also a first step towards a better configuration system.
2. We of course need documentation too for these values, YAML looks good for this. We could use 1) with yaml-files. Should the docs with yaml-header be in the module itself or in a separate documentation file (readme)?
3. Each bot carries the default parameters in it's python-code itself.

All of them are backwards-compatible, as BOTS is not removed with these options yet. (but that's definitely the target).
I tend towards the second option.

Also ping @dmth @aaronkaplan

[sebix]

postponed