Tests

10 January 2024

The Current "Structure" for <Tests>

The <Tests> section describes tests that are intended to show that the TOE meets the associated SFR. This section has traditionally been structured fairly loosely in order to allow a wide variety of ways to describe testing. As a result, the quality of tests in NIAP protection profiles has been quite uneven. In the near future, we hope to have a more more structured format for the <Tests> section to try to force some rigor on test development, but for now the section looks something like this:

If there are no tests for the component, element, or management function, then you can say as much:

	<Tests>
     	    There are no test evaluation activities for this component.
	</Tests>			

or you can simply use an empty element <Tests/> and the appropriate text will appear in the document.

Test Contents

If there are tests, the <Tests> section should include:

• The purpose of the test
• Any setup required for the test
• Each step of the testing process
• The expected results of the test
• The format for expressing the results and reporting evidence
• Validation steps (if necessary)

The <Tests> element is very flexible (probably too flexible) in allowing tests to be specified in just about any manner the PP Author would like.

For example, it can be nothing but free text:

	<Tests>
		The evaluator must perform the following tests:
                Here are some words describing the test or tests for this SFR. I can go on and on forever about these
                tests without any structure. So blah, blah, blah.
        </Tests>

If there are several tests for the same SFR, the <testlist> element can be used to add some structure.

	<testlist>
		<test>
			For each authentication mechanism selected in FIA_UAU.5.1, the evaluator shall
			enable that mechanism and verify that it can be used to authenticate the user at 
			the specified authentication factor interfaces.
		</test>
		<test>
			For each authentication mechanism rule, the evaluator shall ensure that the 
			authentication mechanism behaves accordingly.
		</test>
	</testlist>

The <testlist> element can be be combined with free text.

	<testlist>
                Let me explain something about these tests. There's not a lot of detail here. And nothing about test evidence. Ugh!
		<test>
			For each authentication mechanism selected in FIA_UAU.5.1, the evaluator shall
			enable that mechanism and verify that it can be used to authenticate the user at 
			the specified authentication factor interfaces.
		</test>
		<test>
			For each authentication mechanism rule, the evaluator shall ensure that the 
			authentication mechanism behaves accordingly.
		</test>
	</testlist>

Conditional Tests

Many PPs contain conditional tests. Often these are simply indicated by sticking "[conditional]" in front of the description of the test. Ideally, conditional tests should depend on automatable conditions, for example, a selection, use case, platform, package inclusion, or feature. This way, the conditional tests can be discarded during ST development so no evaluator or validator ever has to look at them.

Conditional tests are currently expressed by using the <depends> tag at the beginning of the element that encloses the test. In this case, we use the <h:div> tag as a way to group the dependency together with the text of the test. The condition in the dependency references a platform that is declared elsewhere. See Platforms Declaration. Currently, this construct is used only in the AppPP.

	<Tests>
		If a container image is claimed, the evaluator shall verify that application updates are distributed as container images.
		If the format of the platform-supported package manager is claimed,
		the evaluator shall verify that application updates are distributed in the
		format supported by the platform. This varies per platform:               
		<h:div><depends ref="android"/> 
                    The evaluator shall ensure that the application is packaged in the Android application package (APK) format. 
                </h:div>
		<h:div><depends ref="ios"/> 
                    The evaluator shall ensure that the application is packaged in the IPA format. 
                </h:div>
		<h:div><depends ref="Solaris"/> 
                       The evaluator shall ensure that the application is packaged in the PKG format. 
                </h:div>
		<h:div><depends ref="mac"/> 
                    The evaluator shall ensure that the application is packaged in the DMG format, the PKG format, or the MPKG format. 
                </h:div>
	</Tests>