ORAS OCI store index race conditions

[akashsinghal]

What happened in your environment?

• ORAS referrer store can fail to initialize for external verifiers with error: Original Error: (Original Error: (create store from input config failed with error Original Error: (invalid OCI Image Index: failed to decode index file: EOF), Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Component Type: referrerStore, Detail: could not create local ORAS cache at path: /home/runner/.ratify/local_oras_cache)
• This occurs when multiple external verifiers are executed for the same subject in parallel.
• Intermittent issue affecting e2e tests

What did you expect to happen?

No response

What version of Kubernetes are you running?

No response

What version of Ratify are you running?

v1.0.0

Anything else you would like to add?

No response

Are you willing to submit PRs to contribute to this bug fix?

• Yes, I am willing to implement it.

[akashsinghal]

This issue is not high priority and intermittently affects the ratify e2e tests. Here's a sample verifier report with the failures:

{
  "verifierReports": [
    {
      "subject": "localhost:5000/all:v0",
      "isSuccess": true,
      "name": "notation",
      "message": "signature verification success",
      "extensions": {
        "Issuer": "CN=ratify-bats-test,O=Notary,L=Seattle,ST=WA,C=US",
        "SN": "CN=ratify-bats-test,O=Notary,L=Seattle,ST=WA,C=US"
      },
      "artifactType": "application/vnd.cncf.notary.signature"
    },
    {
      "isSuccess": false,
      "name": "cosign",
      "message": "Original Error: (Original Error: (create store from input config failed with error Original Error: (invalid OCI Image Index: failed to decode index file: EOF), Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Component Type: referrerStore, Detail: could not create local oras cache at path: /home/runner/.ratify/local_oras_cache), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: cosign, Component Type: verifier), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: cosign, Component Type: verifier",
      "artifactType": "application/vnd.dev.cosign.artifact.sig.v1+json"
    },
    {
      "subject": "localhost:5000/all:v0",
      "isSuccess": true,
      "name": "licensechecker",
      "message": "License Check: SUCCESS. All packages have allowed licenses",
      "artifactType": "application/vnd.ratify.spdx.v0"
    },
    {
      "subject": "localhost:5000/all:v0",
      "isSuccess": true,
      "name": "schemavalidator",
      "message": "schema validation passed for configured media types",
      "artifactType": "vnd.aquasecurity.trivy.report.sarif.v1"
    },
    {
      "isSuccess": false,
      "name": "sbom",
      "message": "Original Error: (Original Error: (create store from input config failed with error Original Error: (invalid OCI Image Index: failed to decode index file: EOF), Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Component Type: referrerStore, Detail: could not create local oras cache at path: /home/runner/.ratify/local_oras_cache), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: sbom, Component Type: verifier), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: sbom, Component Type: verifier",
      "nestedResults": [
        {
          "subject": "localhost:5000/all@sha256:b71c1f874fbc92173278bcb7bb44c785b167f7efa3c44b52eb48e20d540741b5",
          "isSuccess": true,
          "name": "notation",
          "message": "signature verification success",
          "extensions": {
            "Issuer": "CN=ratify-bats-test,O=Notary,L=Seattle,ST=WA,C=US",
            "SN": "CN=ratify-bats-test,O=Notary,L=Seattle,ST=WA,C=US"
          },
          "artifactType": "application/vnd.cncf.notary.signature"
        }
      ],
      "artifactType": "org.example.sbom.v0"
    }
  ]
}

[akashsinghal]

Investigation notes:

• The real error is the EOF error returned by the ORAS OCI store's New method. The oras-go implementation attempts to load the index.json file. When Ratify is initialized and the ORAS Referrer store is first created, the index.json file does not exist and is thus created and populated with an empty OCI Index manifest json. On verification start, each verifier spawns a separate routine. And each routine invokes the registered verifier. Each verifier is running in parallel.
• Each external verifier invokes a SEPERATE process which creates a new OCI store instance. index.json already exists so the loadIndex functionality in oras-go will attempt to decode the file contents as json. This decoding is what is throwing the EOF errors.
• ORAS OCI store is not process safe and OCIStore is not concurrency safe at a process-level oras-project/oras-go#286 acknowledges this gap
• Current hypothesis is that the concurrent file writing/reading by different processes is leading to EOF errors.

[akashsinghal]

Maybe a more straightforward solution here is to see if the OCI store's index can leverage a lightweight local db like boltdb which containerd already used for it's content store. I'm not sure if ORAS's interface is open enough to support this