[Security Solution] Improve rule diff algorithm performance by splitting diff calculation into async chunks

[jpdjere]

Epics: /~https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

• The calculation of rule fields diffs in the /upgrade/_review endpoint is a computationally expensive synchronous process that blocks the event loop during its calculation.
• In order to avoid blocking the event loop and improve server-side performance, we can split diff calculation into async chunks that yield control back to the event loop after a number of them are executed.
• A solution could look like so (PSEUDO):

        for (let i = 0; i < upgradeableRules.length; i += chunkSize) {
            const chunk = upgradeableRules.slice(i, i + chunkSize);

            chunk.forEach(({ current }) => {
                const ruleVersions = ruleVersionsMap.get(current.rule_id);
                invariant(ruleVersions != null, 'ruleVersions not found');
                const diffResult = calculateRuleDiff(ruleVersions);
                ruleDiffCalculationResults.push(diffResult);
            });

            // Yield control back to the event loop
            await setImmediatePromise();
        }

Background

The above solution is based on:

1. Answer from stack overflow
2. Node.js guide: Complex calculations without blocking the Event Loop

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)