Skip to content

Commit

Permalink
Move neon_pgstat_file_size_limit to the extension (#10959)
Browse files Browse the repository at this point in the history
## Problem

PG14 uses separate backend for stats collector having no access to
shaerd memory.
As far as AUX mechanism requires access to shared memory, persisting
pgstat.stat file
is not supported at pg14. And so there is no definition of
`neon_pgstat_file_size_limit`
variable. It makes it impossible to provide same config for all Postgres
version.

## Summary of changes

Move neon_pgstat_file_size_limit to Neon extension.

Postgres submodules PR:
neondatabase/postgres#587
neondatabase/postgres#588
neondatabase/postgres#589

---------

Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>
Co-authored-by: Tristan Partin <tristan@neon.tech>
  • Loading branch information
3 people authored Feb 25, 2025
1 parent 758f597 commit 8f82c66
Show file tree
Hide file tree
Showing 6 changed files with 27 additions and 7 deletions.
20 changes: 20 additions & 0 deletions pgxn/neon/neon.c
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
#include "fmgr.h"

#include "miscadmin.h"
#include "pgstat.h"
#include "access/subtrans.h"
#include "access/twophase.h"
#include "access/xlog.h"
Expand Down Expand Up @@ -410,6 +411,16 @@ ReportSearchPath(void)
}
}

#if PG_VERSION_NUM < 150000
/*
* PG14 uses separate backend for stats collector having no access to shared memory.
* As far as AUX mechanism requires access to shared memory, persisting pgstat.stat file
* is not supported in PG14. And so there is no definition of neon_pgstat_file_size_limit
* variable, so we have to declare it here.
*/
static int neon_pgstat_file_size_limit;
#endif

void
_PG_init(void)
{
Expand Down Expand Up @@ -467,6 +478,15 @@ _PG_init(void)
0,
NULL, NULL, NULL);

DefineCustomIntVariable("neon.pgstat_file_size_limit",
"Maximal size of pgstat.stat file saved in Neon storage",
"Zero value disables persisting pgstat.stat file",
&neon_pgstat_file_size_limit,
0, 0, 1000000, /* disabled by default */
PGC_SIGHUP,
GUC_UNIT_KB,
NULL, NULL, NULL);

/*
* Important: This must happen after other parts of the extension are
* loaded, otherwise any settings to GUCs that were set before the
Expand Down
2 changes: 1 addition & 1 deletion test_runner/regress/test_pgstat.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ def test_pgstat(neon_simple_env: NeonEnv):

n = 10000
endpoint = env.endpoints.create_start(
"main", config_lines=["neon_pgstat_file_size_limit=100kB", "autovacuum=off"]
"main", config_lines=["neon.pgstat_file_size_limit=100kB", "autovacuum=off"]
)

con = endpoint.connect()
Expand Down
2 changes: 1 addition & 1 deletion vendor/postgres-v15
2 changes: 1 addition & 1 deletion vendor/postgres-v16
2 changes: 1 addition & 1 deletion vendor/postgres-v17
6 changes: 3 additions & 3 deletions vendor/revisions.json
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
{
"v17": [
"17.4",
"59b2fe851f8e0595f6c830b90ee766f4f1c17a0f"
"517b8dc244abf3e56f0089849e464af76f70b94e"
],
"v16": [
"16.8",
"261ed10e9b8c8dda01ad7aefb18e944e30aa161d"
"799e7a08dd171aa06a7395dd326f4243aaeb9f93"
],
"v15": [
"15.12",
"6ff50443773b69749e16da6db9d4f4b19064b4b7"
"9b118b1cffa6e4ca0d63389b57b54d11e207e9a8"
],
"v14": [
"14.17",
Expand Down

1 comment on commit 8f82c66

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7884 tests run: 7485 passed, 13 failed, 386 skipped (full report)


Failures on Postgres 16

  • test_throughput[github-actions-selfhosted-50-pipelining_config7-30-100-128-batchable {'max_batch_size': 1, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config8-30-100-128-batchable {'max_batch_size': 2, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config9-30-100-128-batchable {'max_batch_size': 2, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config10-30-100-128-batchable {'max_batch_size': 4, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config11-30-100-128-batchable {'max_batch_size': 4, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config12-30-100-128-batchable {'max_batch_size': 8, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config13-30-100-128-batchable {'max_batch_size': 8, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config14-30-100-128-batchable {'max_batch_size': 16, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config15-30-100-128-batchable {'max_batch_size': 16, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config16-30-100-128-batchable {'max_batch_size': 32, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config17-30-100-128-batchable {'max_batch_size': 32, 'execution': 'tasks', 'mode': 'pipelined'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config5-30-100-128-batchable {'mode': 'serial'}]: release-x86-64-with-lfc
  • test_throughput[github-actions-selfhosted-50-pipelining_config6-30-100-128-batchable {'max_batch_size': 1, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]: release-x86-64-with-lfc
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config7-30-100-128-batchable {'max_batch_size': 1, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config8-30-100-128-batchable {'max_batch_size': 2, 'execution': 'concurrent-futures', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config9-30-100-128-batchable {'max_batch_size': 2, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config10-30-100-128-batchable {'max_batch_size': 4, 'execution': 'concurrent-futures', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config11-30-100-128-batchable {'max_batch_size': 4, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config12-30-100-128-batchable {'max_batch_size': 8, 'execution': 'concurrent-futures', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config13-30-100-128-batchable {'max_batch_size': 8, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config14-30-100-128-batchable {'max_batch_size': 16, 'execution': 'concurrent-futures', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config15-30-100-128-batchable {'max_batch_size': 16, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config16-30-100-128-batchable {'max_batch_size': 32, 'execution': 'concurrent-futures', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config17-30-100-128-batchable {'max_batch_size': 32, 'execution': 'tasks', 'mode': 'pipelined'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config5-30-100-128-batchable {'mode': 'serial'}] or test_throughput[release-pg16-github-actions-selfhosted-50-pipelining_config6-30-100-128-batchable {'max_batch_size': 1, 'execution': 'concurrent-futures', 'mode': 'pipelined'}]"
Flaky tests (1)

Postgres 17

Code coverage* (full report)

  • functions: 32.8% (8636 of 26346 functions)
  • lines: 48.6% (72909 of 149878 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
8f82c66 at 2025-02-25T15:49:54.433Z :recycle:

Please sign in to comment.