Configurations - Hierarchy configurations

[SYNchroACK]

• generic
• bot type
• bots

[sebix]

Could you please expand the description of this issue? It is totally unclear what is meant here.

[SYNchroACK]

The idea is remove the defaults.conf file and have the runtime.conf in the following format:

Current config:

{
    "malware-domain-list-collector": {
        "error_procedure": "retry",
        "error_retry_delay": 30,
        "error_log_message": true,
        "error_dump_message": true,
        "url": "http://www.malwaredomainlist.com/updatescsv.php",
        "feed": "Malware Domain List",
        "rate_limit": 3600
    },
    "malware-domain-list-parser": {
        "error_procedure": "pass",
        "error_retry_delay": 30,
        "error_log_message": true,
        "error_dump_message": true
    },
    "file-output": {
        "error_procedure": "retry",
        "error_retry_delay": 30,
        "error_log_message": true,
        "error_dump_message": true,
        "file": "/opt/intelmq/var/lib/bots/file-output/events.txt"
    }
}

Proposal:

{
    "__default__": {
        "broker": "redis",
        "error_procedure": "retry",
        "error_max_retries": 3,
        "error_log_message": true,
        "error_log_exception": true,
        "error_dump_message": true,
        "error_retry_delay": 15,
        "exit_on_stop": true,
        "rate_limit": 0,
        "load_balance": false,
        "source_pipeline_host": "127.0.0.1",
        "source_pipeline_port": "6379",
        "source_pipeline_db": "2",
        "destination_pipeline_host": "127.0.0.1",
        "destination_pipeline_port": "6379",
        "destination_pipeline_db": "2",
        "http_proxy": null,
        "https_proxy": null,
        "http_user_agent": null
    },
    "Collectors": {
        "__default__": {
            "error_procedure": "pass",
            "error_retry_delay": 1000,
            "error_log_message": false,
            "error_dump_message": false,
        },
        "malware-domain-list-collector": {
            "error_procedure": "retry",
            "error_retry_delay": 30,
            "error_log_message": true,
            "error_dump_message": true,
            "url": "http://www.malwaredomainlist.com/updatescsv.php",
            "feed": "Malware Domain List",
            "rate_limit": 3600
        }
    },
    "Parsers": {
        "__default__": {
            "error_procedure": "retry",
            "error_retry_delay": 2323,
            "error_log_message": false,
            "error_dump_message": true,
        },
        "malware-domain-list-parser": {
            "error_procedure": "pass",
            "error_retry_delay": 30,
            "error_log_message": true,
            "error_dump_message": true
        }
    },
    "Outputs": {
        "__default__": {
            "error_procedure": "pass",
            "error_retry_delay": 9999,
            "error_log_message": true,
            "error_dump_message": false,
        },
        "file-output": {
            "error_procedure": "retry",
            "error_retry_delay": 30,
            "error_log_message": true,
            "error_dump_message": true,
            "file": "/opt/intelmq/var/lib/bots/file-output/events.txt"
        }
    }
}

[sebix]

Sounds reasonable for me. +1

[SYNchroACK]

A little more context information...

Explanation:

• Generic parameters values will be overwritten by the specific parameters.

Example:

• If a bot starts and load the parameter 'rate_limit: 0' in the 'default' global section, but that bot is the 'malware-domain-list-collector', the 'rate_limit: 0' will be overwritten by 'rate_limit: 3600'.

[sebix]

Will be postponed as this needs also changes in intelmq-manager and @mauroasilva is not available currently.

[sebix]

@robcza @aaronkaplan @dmth

I want to bring this up again. Do we want this change for 1.0, 1.1 or 2.0? It does make sense to apply it now, because doing the change in 1.1, we must allow both configurations for backwards-compatibility. This would be error-prone for both the users and in code.

Working on 1.0 we can still break compatibility.

[aaronkaplan]

Basically we are talking about inheritance (object orient programming inheritance) here.

[sebix]

For 1.0 we could easily remove system.conf and move the 2 values to defaults.conf. Then we get at least rid of one of the files.

Migration path: Until 1.0 we still read the system.conf, (if it exists) but emit a warning. In 1.0 we remove it entirely.

[sebix]

Next step before 1.0 could be: remove startup.conf and move the values into runtime.conf. On intelmq-side this is trivial, for the manager (I guess) too. What do you think?

The bots themselves do not depend on it, it's only the ctl-tool which uses/reads it.

[aaronkaplan]

On 14 Sep 2016, at 12:53, Sebastian notifications@github.com wrote:

Next step before 1.0 could be: remove startup.conf and move the values into runtime.conf. On intelmq-side this is trivial, for the manager (I guess) too. What do you think?

+1

[aaronkaplan]

Revisiting this issue : how does it play together with our idea of splitting up the config into snippets for each bot in a conf.d/* directory structure? Does that play into this issue or do you want to keep it separate? For me this thread is not very clearly defined right now.

Are we talking about only inheritance of values of a config?
Are we talking about the conf.d/* changes as well?
Are we talking only about moving startup.conf into runtime.conf ?

It's not 100% clear how the config part of intelmq shall look like in the future? (by which deadline now)?

[vince-nayal]

Hi,
Is @SYNchroACK config proposal usable in the current realease ?

[SYNchroACK]

@vince-nayal still not implemented, unfortunately

[ghost]

The configuration has been revisited since the creation of the issue (removal of system.conf and startup.conf). Issue is staled since 3 years, so closing now.