-
Notifications
You must be signed in to change notification settings - Fork 2
Evaluation Activity Reform
I propose a new structure for Evaluation Activities. The purpose of the change is to impose more rigor and structure on testing through enforcement of a tighter schema.
There will be a new tag <eactivity> to replace the current <aactivity> tag. This way both tags could co-exist if necessary.
Also, EAs will located at the component level, rather than the current element level. They will be mandatory at the component level and optional at the element level. And also optional for management functions. They probably should be required for rows of "tabularized" SFRs.
<eactivity id="ea-cp-fcs-cop-1-aescbc">
<ea-objective><words></words></ea-objective>
<zeroOrMore><ea-applies-to-ext ref="id-oftarget"><ref to document contianing the target component, element, or management function.
</ea-applies-to-ext></zerOrMore>
<zerOrMore><ea-depends on-ea="id-ofEA"/></zeroOrMore>
<choice>
<ea-None>Statement that there are no EAs for this component/element/management function.</ea-None>
<group>
<ea-TSS>Information about this SFR that isn't already required by a SAR or something</ea-TSS>
<ea-KMD>(optional)Information abut key management. This part of the TSS, but is usually not public</ea-KMD>
<ea-Guidance>aka AGD. Guidance about this SFR that is already not required by a SAR or something.</ea-Guidance>
<ea-Tests>
<ea-Tests-objective>(optional)
<ea-test>
<test-objective>(optional--but should be here or above)
<ea-input>(format and content of inputs)
<ea-tools>(tools needed)(optional)
<ea-competencies>(optional)
<ea-test-setup>(i necessary)
<ea-steps>
<ea-criteria>(for pass-fail)
<ea-reporting>Content and Format of test evidence</ea-reporting>
</ea-test>
<ea-criteria>(for pass-fail)(optional at this level)
</ea-Tests>
</group>
<rationale>Gonna leave this out</ea-rationale>
</choice>
</eactivity>CC:2022 says, "Evaluation activities shall be uniquely identified within their source document."
It's not clear whether they need a name, but if we give them an id, then at least they can be referenced. I recommend the convention:
<eactivity id="ea-cp-componentId"> for EAs attached to components (e.g. <eactivity id="ea-cp-fcs-ckm-1-ak">)
<eactivity id="ea-el-elementId"> for EAs attached to elements (e.g. <eactivity id="ea-el-fcs-ckm-1-ak-1">)
<eactivity id="ea-el-mfId"> for EAs attached to management functions (e.g. <eactivity id="ea-mf-radio-toggle">)CC:2022 says, "The objective of performing the evaluation activity shall be stated."
It seems like this should be mandatory, but perhaps if it is omitted the framework can generate some boilerplate text such as, this EA (somehow using the Id) is intended to test the TOE's conformance to <Component/Element/Management function>. Although this is pretty lame.
If the EA applies to SFRs or SARs in another document, the relationship should be specified here. For the normal NIAP case in which the EA appears with the SFR, there is no need to do anything. Even if the EAs appear in a separate document, the framework should be able to handle the linkage. So, this would be most useful for Evaluation Methods documents or maybe catalogs.
If the success of the EA depends on another EA, then reference the other EA here. Not sure what to do with this other than to explicitly note it. I suppose this could be in another document.
This section is mandatory. If it is empty, the framework outputs something like "There are no additional TSS evaluation activities for this component/element/management function."
This section is optional. If it is present, but empty, the framework outputs "There are no additional KMD evaluation activities for this component/element/management function."
This section is mandatory. If it is empty, the framework outputs "There are no additional Guidance evaluation activities for this component/element/management function."
This section is mandatory. If it is empty the framework outputs "There are no test activities for this component/element/management function."
<Tests>
General description of the tests and purpose.
<testlist>
The evaluator shall perform the following tests:
<test>
Describe the purpose of this test.
Describe test set up.
<steplist>
each step in the test
</steplist>
Describe the expected results.
Describe the test evidence and its expected format.
Explain anything a validator might need to know to
determine if the evidence shows that the test passed.
</test>
<test>
.
.
</test>
</testlist>
</Tests>There are a few other elements that may appear within a test. They are all optional (and seldom used). They will likely be deprecated in the near future.
-
<test-obj>: Describe the objective of the test -
<evidence>: Describes the expected test evidence. -
<applies-if>: Another way of doing conditional requirements that no one has ever used.
This structure is intended to impose some rigor on the development of tests by forcing adherence to a strict schema.
In this proposal, all tests must be defined within a <test> element. This may be within a test list or standalone.
<test-strict id="tst-unique-id" test-id="FCS-CKM-EXT-4.1-#" name="Name of Test">
<!-- Name is optional, but some Crypto SFR tests have names, e.g. "Short Messages Test: Bit-oriented Mode" -->
<!-- test-id should be mandatory. It could, for example, refer to a test from a Evaluation Methods document.
Straight up sequential numbers are terrible -->
<!-- In the future, it might be possible to reference tests from external documents rather than include them in the PP -->
<depends on="some conditions"/> <!-- there can be more than one of these -->
<!-- For conditional tests (must be based on a selection, use case, platform, feature, SFR, or package (rigor!) -->
<!-- if omitted, the test is unconditional. -->
<!-- There should also be a way to express conditions in only words, e.g. a test depends on the failure/success of a previous test -->
<!-- This makes more sense than using the <h:div>, though that can still be used for arbitrary text.
<test-purpose>
A short explanation of what the test intends to accomplish. Empty not allowed.
</test-purpose>
<test-setup>
Description or steps needed to prepare the TOE for testing. Default is TOE in evaluated condition from operational guidance.
Can be free text or a <steplist> or both. This can reference another test (<test-ref ref="id">?). But not conditional tests?
This could be allowed at the <Tests> level if the same setup applies to more than one test.
</test-setup>
<test-steps>
A <steplist>.
A step list could have have an id in case another test needs to reference it rather than repeat it. Once again, not for a conditional test
Probably want an expected result for each step, as well as for the test as a whole.
</test-steps>
<test-expected-results>
The test passes if...
The test fails if...
</test-expected-results>
<test-evidence>
Expected format/form/contents for results and evidence. Maybe this should point at some other document? CEM?
</test-evidence>
<test-validator-guidance>
????
</test-validator-guidance>
</test-strict>