Too much parrallelism creates issues with Grant: tuple concurrently updated

[awoimbee]

I'm using my pulumi to configure a local postgreSQL instance, it's really fast !
Maybe it's a bit too fast. I get error: could not execute revoke query: pq: tuple concurrently updated.
To fix the issue, I need to run pulumi with low parallelism (e.g. pulumi up -p 2).

Steps to reproduce

My stack is quite complex, it seems like is happens when it creates multiple Grant (one for a database, another for a schema, ...) to one Role. (to improve)

Expected: everything works
Actual: error is created, pulumi stops creating resources

Environment

• postgreSQL: latest version, in docker
• CPU: 8 cores, 16 threads (R7 4800H)
• pulumi: v3.28.0
• @pulumi/postgresql: 3.4.0

=> upstream issue: cyrilgdn/terraform-provider-postgresql#178

[stack72]

Hi @awoimbee

Please can you try and give us some minimal reproducible code that will help us here?

Paul

[awoimbee]

Hi @stack72,
Here are steps to reproduce:

docker run --rm --name postgres -e POSTGRES_PASSWORD=toto -p 5432:5432 -d postgres
export PULUMI_CONFIG_PASSPHRASE=""
pulumi new typescript -y -s poc-tuples-concurrent
cat package.json | jq '.dependencies *= {"@pulumi/postgresql": "^3.1.0"}' > p2 && mv p2 package.json
npm i
pulumi config set postgresql:sslmode disable
pulumi config set postgresql:password toto

Contents of index.ts: lots of grants, lots of roles:

import * as pg from "@pulumi/postgresql";

const NB_SCHEMAS = 1;
const NB_GRANTS = 2;

let roles: pg.Role[] = [];
for (let i = 0; i < NB_GRANTS + 1; i += 1) {
  roles.push(new pg.Role(`r-${i}`));
}

let schemas: pg.Schema[] = [];
for (let i = 0; i < NB_SCHEMAS; i += 1) {
  schemas.push(
    new pg.Schema(
      `s${i}`,
      {
        database: "postgres",
        name: `s${i}`,
        owner: roles[0].name
      }
    )
  );
}

schemas.forEach((s, i) => {
  for (let grant = 0; grant < NB_GRANTS; grant += 1) {
    new pg.Grant(
      `r${grant+1}-${i}`,
      {
        database: "postgres",
        schema: s.name,
        objectType: "schema",
        privileges: ["USAGE"],
        role: roles[grant + 1].name
      }
    );
  }
});

On my machine, 3 roles, 1 schema and 2 grants are enough to trigger the error:

import * as pg from "@pulumi/postgresql";
const r0 = new pg.Role("r0");
const r1 = new pg.Role("r1");
const r2 = new pg.Role("r2");
const s = new pg.Schema(`s0`, {
    database: "postgres",
    name: `s0`,
    owner: r0.name
});
const baseGrant = {
  database: "postgres",
  schema: s.name,
  objectType: "schema",
  privileges: ["USAGE"]
};
new pg.Grant("g0", {...baseGrant, role: r1.name});
new pg.Grant("g1", {...baseGrant, role: r2.name});

Then, to be 100% sure to trigger the bug, use: while 1; do pulumi up --yes --skip-preview --non-interactive; sleep 0.5; pulumi destroy --yes --skip-preview --non-interactive; done

[shousper]

Looks like this issue still hasn't been resolved downstream: cyrilgdn/terraform-provider-postgresql#178

[AaronFriel]

It looks like there is an upstream fix:

• Fix concurrency issue when granting privileges on databases cyrilgdn/terraform-provider-postgresql#224

However it breaks compatibility with CockroachDB, one of our partners:

• pg_advisory_xact_lock breaks Cockroach Compatibility cyrilgdn/terraform-provider-postgresql#277

I've requested advice from Cockroach here:

• sql: fill out pg_advisory_lock stubs cockroachdb/cockroach#13546

[AaronFriel]

Hey @awoimbee, @shousper, we've made the decision to merge this despite #178, in order to bring this fix to users. Please let us know if you still encounter this with default parallelism.

[awoimbee]

I just tested against a local PG DB, a case where I would get tons of failures, it's working great ! Thanks !