Skip to content

Commit

Permalink
transaction deferable
Browse files Browse the repository at this point in the history
  • Loading branch information
insani7y committed Mar 5, 2024
1 parent 24709de commit 0f53dbe
Show file tree
Hide file tree
Showing 11 changed files with 155 additions and 66 deletions.
98 changes: 64 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,39 @@
[![PyPI](https://img.shields.io/pypi/v/psqlpy?style=for-the-badge)](https://pypi.org/project/psqlpy/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/psqlpy?style=for-the-badge)](https://pypistats.org/packages/psqlpy)


# PSQLPy - Async PostgreSQL driver for Python written in Rust.

Driver for PostgreSQL written fully in Rust and exposed to Python.
The project is under active development and _**we cannot confirm that it's ready for production**_. Anyway, We will be grateful for the bugs found and open issues. Stay tuned.
*Normal documentation is in development.*
_Normal documentation is in development._

## Installation

You can install package with `pip` or `poetry`.

poetry:

```bash
> poetry add psqlpy
```

pip:

```bash
> pip install psqlpy
```

Or you can build it by yourself. To do it, install stable rust and [maturin](/~https://github.com/PyO3/maturin).

```
> maturin develop --release
```

## Usage

Usage is as easy as possible.
Create new instance of PSQLPool, startup it and start querying.

```python
from typing import Any

Expand Down Expand Up @@ -57,11 +62,14 @@ async def main() -> None:
# rust does it instead.

```

Please take into account that each new execute gets new connection from connection pool.

### DSN support

You can separate specify `host`, `port`, `username`, etc or specify everything in one `DSN`.
**Please note that if you specify DSN any other argument doesn't take into account.**

```py
from typing import Any

Expand All @@ -86,31 +94,33 @@ async def main() -> None:
```

### Control connection recycling

There are 3 available options to control how a connection is recycled - `Fast`, `Verified` and `Clean`.
As connection can be closed in different situations on various sides you can select preferable behavior of how a connection is recycled.

- `Fast`: Only run `is_closed()` when recycling existing connections.
- `Verified`: Run `is_closed()` and execute a test query. This is slower, but guarantees that the database connection is ready to
be used. Normally, `is_closed()` should be enough to filter
out bad connections, but under some circumstances (i.e. hard-closed
network connections) it's possible that `is_closed()`
returns `false` while the connection is dead. You will receive an error
on your first query then.
be used. Normally, `is_closed()` should be enough to filter
out bad connections, but under some circumstances (i.e. hard-closed
network connections) it's possible that `is_closed()`
returns `false` while the connection is dead. You will receive an error
on your first query then.
- `Clean`: Like [`Verified`] query method, but instead use the following sequence of statements which guarantees a pristine connection:
```sql
CLOSE ALL;
SET SESSION AUTHORIZATION DEFAULT;
RESET ALL;
UNLISTEN *;
SELECT pg_advisory_unlock_all();
DISCARD TEMP;
DISCARD SEQUENCES;
```
This is similar to calling `DISCARD ALL`. but doesn't call
`DEALLOCATE ALL` and `DISCARD PLAN`, so that the statement cache is not
rendered ineffective.
```sql
CLOSE ALL;
SET SESSION AUTHORIZATION DEFAULT;
RESET ALL;
UNLISTEN *;
SELECT pg_advisory_unlock_all();
DISCARD TEMP;
DISCARD SEQUENCES;
```
This is similar to calling `DISCARD ALL`. but doesn't call
`DEALLOCATE ALL` and `DISCARD PLAN`, so that the statement cache is not
rendered ineffective.

## Query parameters

You can pass parameters into queries.
Parameters can be passed in any `execute` method as the second parameter, it must be a list.
Any placeholder must be marked with `$< num>`.
Expand All @@ -123,7 +133,9 @@ Any placeholder must be marked with `$< num>`.
```

## Connection

You can work with connection instead of DatabasePool.

```python
from typing import Any

Expand Down Expand Up @@ -154,17 +166,22 @@ async def main() -> None:
```

## Transactions

Of course it's possible to use transactions with this driver.
It's as easy as possible and sometimes it copies common functionality from PsycoPG and AsyncPG.

### Transaction parameters

In process of transaction creation it is possible to specify some arguments to configure transaction.

- `isolation_level`: level of the isolation. By default - `None`.
- `read_variant`: read option. By default - `None`.
- `deferable`: deferable option. By default - `None`.

### You can use transactions as async context managers

By default async context manager only begins and commits transaction automatically.

```python
from typing import Any

Expand All @@ -188,6 +205,7 @@ async def main() -> None:
```

### Or you can control transaction fully on your own.

```python
from typing import Any

Expand Down Expand Up @@ -219,9 +237,11 @@ async def main() -> None:
```

### Transactions can be rolled back

You must understand that rollback can be executed only once per transaction.
After it's execution transaction state changes to `done`.
If you want to use `ROLLBACK TO SAVEPOINT`, see below.

```python
from typing import Any

Expand All @@ -247,6 +267,7 @@ async def main() -> None:
```

### Transaction ROLLBACK TO SAVEPOINT

You can rollback your transaction to the specified savepoint, but before it you must create it.

```python
Expand Down Expand Up @@ -280,6 +301,7 @@ async def main() -> None:
```

### Transaction RELEASE SAVEPOINT

It's possible to release savepoint

```python
Expand Down Expand Up @@ -308,12 +330,15 @@ async def main() -> None:
```

## Cursors

Library supports PostgreSQL cursors.

Cursors can be created only in transaction. In addition, cursor supports async iteration.

### Cursor parameters

In process of cursor creation you can specify some configuration parameters.

- `querystring`: query for the cursor. Required.
- `parameters`: parameters for the query. Not Required.
- `fetch_number`: number of records per fetch if cursor is used as an async iterator. If you are using `.fetch()` method you can pass different fetch number. Not required. Default - 10.
Expand Down Expand Up @@ -357,7 +382,9 @@ async def main() -> None:
```

### Cursor operations

Available cursor operations:

- FETCH count - `cursor.fetch(fetch_number=)`
- FETCH NEXT - `cursor.fetch_next()`
- FETCH PRIOR - `cursor.fetch_prior()`
Expand All @@ -370,15 +397,16 @@ Available cursor operations:
- FETCH BACKWARD ALL - `cursor.fetch_backward_all()`

## Extra Types

Sometimes it's impossible to identify which type user tries to pass as a argument. But Rust is a strongly typed programming language so we have to help.

| Extra Type in Python | Type in PostgreSQL | Type in Rust |
| ------------- | ------------- | -------------
| SmallInt | SmallInt | i16 |
| Integer | Integer | i32 |
| BigInt | BigInt | i64 |
| PyUUID | UUID | Uuid |
| PyJSON | JSON, JSONB | Value |
| Extra Type in Python | Type in PostgreSQL | Type in Rust |
| -------------------- | ------------------ | ------------ |
| SmallInt | SmallInt | i16 |
| Integer | Integer | i32 |
| BigInt | BigInt | i64 |
| PyUUID | UUID | Uuid |
| PyJSON | JSON, JSONB | Value |

```python
from typing import Any
Expand Down Expand Up @@ -425,15 +453,17 @@ async def main() -> None:
```

## Benchmarks

We have made some benchmark to compare `PSQLPy`, `AsyncPG`, `Psycopg3`.
Main idea is do not compare clear drivers because there are a few situations in which you need to use only driver without any other dependencies.

**So infrastructure consists of:**
1) AioHTTP
2) PostgreSQL driver (`PSQLPy`, `AsyncPG`, `Psycopg3`)
3) PostgreSQL v15. Server is located in other part of the world, because we want to simulate network problems.
4) Grafana (dashboards)
5) InfluxDB
6) JMeter (for load testing)

The results are very promising! `PSQLPy` is faster than `AsyncPG` at best by 2 times, at worst by 45%. `PsycoPG` is 3.5 times slower than `PSQLPy` in the worst case, 60% in the best case.

1. AioHTTP
2. PostgreSQL driver (`PSQLPy`, `AsyncPG`, `Psycopg3`)
3. PostgreSQL v15. Server is located in other part of the world, because we want to simulate network problems.
4. Grafana (dashboards)
5. InfluxDB
6. JMeter (for load testing)

The results are very promising! `PSQLPy` is faster than `AsyncPG` at best by 2 times, at worst by 45%. `PsycoPG` is 3.5 times slower than `PSQLPy` in the worst case, 60% in the best case.
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,9 @@ line-length = 79
[tool.ruff.pydocstyle]
convention = "pep257"
ignore-decorators = ["typing.overload"]


[tool.ruff.isort]
lines-after-imports = 2
no-lines-before = ["standard-library", "local-folder"]
known-first-party = ["psqlpy"]
1 change: 1 addition & 0 deletions python/psqlpy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
Transaction,
)


__all__ = [
"PSQLPool",
"QueryResult",
Expand Down
13 changes: 7 additions & 6 deletions python/psqlpy/_internal/__init__.pyi
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import types
from enum import Enum
from typing import Any, Dict, List, Optional
from typing import Any, Optional

from typing_extensions import Self

class QueryResult:
"""Result."""

def result(self: Self) -> List[Dict[Any, Any]]:
def result(self: Self) -> list[dict[Any, Any]]:
"""Return result from database as a list of dicts."""

class IsolationLevel(Enum):
Expand Down Expand Up @@ -221,7 +221,7 @@ class Transaction:
async def execute(
self: Self,
querystring: str,
parameters: List[Any] | None = None,
parameters: list[Any] | None = None,
) -> QueryResult:
"""Execute the query.
Expand Down Expand Up @@ -377,7 +377,7 @@ class Transaction:
async def cursor(
self: Self,
querystring: str,
parameters: List[Any] | None = None,
parameters: list[Any] | None = None,
fetch_number: int | None = None,
scroll: bool | None = None,
) -> Cursor:
Expand Down Expand Up @@ -429,7 +429,7 @@ class Connection:
async def execute(
self: Self,
querystring: str,
parameters: List[Any] | None = None,
parameters: list[Any] | None = None,
) -> QueryResult:
"""Execute the query.
Expand Down Expand Up @@ -466,6 +466,7 @@ class Connection:
self,
isolation_level: IsolationLevel | None = None,
read_variant: ReadVariant | None = None,
deferable: bool | None = None,
) -> Transaction:
"""Create new transaction.
Expand Down Expand Up @@ -522,7 +523,7 @@ class PSQLPool:
async def execute(
self: Self,
querystring: str,
parameters: List[Any] | None = None,
parameters: list[Any] | None = None,
) -> QueryResult:
"""Execute the query.
Expand Down
4 changes: 2 additions & 2 deletions python/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ async def psql_pool(
postgres_password: str,
postgres_port: int,
postgres_dbname: str,
) -> AsyncGenerator[PSQLPool, None]:
) -> PSQLPool:
pg_pool = PSQLPool(
username=postgres_user,
password=postgres_password,
Expand All @@ -73,7 +73,7 @@ async def psql_pool(
db_name=postgres_dbname,
)
await pg_pool.startup()
yield pg_pool
return pg_pool


@pytest.fixture(autouse=True)
Expand Down
4 changes: 2 additions & 2 deletions python/tests/test_connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from psqlpy import PSQLPool, QueryResult, Transaction


@pytest.mark.anyio
@pytest.mark.anyio()
async def test_connection_execute(
psql_pool: PSQLPool,
table_name: str,
Expand All @@ -19,7 +19,7 @@ async def test_connection_execute(
assert len(conn_result.result()) == number_database_records


@pytest.mark.anyio
@pytest.mark.anyio()
async def test_connection_transaction(
psql_pool: PSQLPool,
) -> None:
Expand Down
Loading

0 comments on commit 0f53dbe

Please sign in to comment.