Is there a recipe for generating pre signed URLs to upload to object storage e.g. S3?

[aaronsgithub]

I thought I'd ask in case there is a quick answer to this before I go down the rabbit hole.

If we are storing user uploaded files e.g. in S3, is there a way to provide the frontend client a pre-signed url so that it can handle the upload, and have sqlalchemy-file store the details for later retrieval?

[jowilf]

The get_cdn_url() method can be called on a stored file object to retrieve a pre-signed URL for the file that expires in 24 hours ( The expiration time can be customized by setting the LIBCLOUD_S3_STORAGE_CDN_URL_EXPIRY_HOURS environment variable.)

[aaronsgithub]

Thanks for the speedy reply.

I did see the get_object_cdn_url method, but that returns a pre-signed URL which is read only according to the docstring.

I was looking for a way to generate a pre-signed URL to allow a frontend client to upload directly to the S3 bucket:
Uploading objects with presigned URLs

The motivation is to avoid using unnecessary bandwidth to upload files via backend server when those files don't need to be validated or processed by the backend. The frontend would send the file metadata for the backend to store in the DB and then it would return a pre-signed url for the frontend to upload.

But from what I can gather, writeable pre-signed urls are not implemented in libcloud, at least for S3.

[hammad-rb]

Hello,
So I am using sqlalchemy-file with starlette admin files presigned URL get expired in 24 hours. What's best way to solve this should we run cron job to update expired presigned-urls? Can you also please share code sample how to increase expiry time for stored object?

[jowilf]

The expiration time can be customized by setting the LIBCLOUD_S3_STORAGE_CDN_URL_EXPIRY_HOURS environment variable. This needs to be added to the documentation.
A better approach would be to generate the link when needed, for example:

sqlalchemy-file/examples/fastapi/app.py

Line 127 in bbdc68f

async def serve_files(storage: str = Path(...), file_id: str = Path(...)):

[hammad-rb]

@jowilf in this example it first fetches file and then from that File object it fetches presigned URL how can I just create presigned URL without any network call to S3?

[hammad-rb]

The solution that works for me

def create_presigned_url(
    s3_client: Any,
    bucket_name: str,
    object_name: str,
    method: str = "get_object",
    expiration: int = 3600,
    metadata: Dict[str, str] = {},
) -> str | None:
    """Generate a presigned URL for the S3 object

    Args:
        s3_client (Any): boto3 client for S3 service
        bucket_name (str): name of bucket
        object_name (str): path to an object
        expiration (int, optional): expiry time for url. Defaults to 3600.

    Returns:
        str: pre-signed url
    """
    params: Dict = {"Bucket": bucket_name, "Key": object_name}
    if metadata:
        params["Metadata"] = metadata
    logger.bind(payload=params).debug("Parameters.")
    response = s3_client.generate_presigned_url(
        method,
        Params=params,
        ExpiresIn=expiration,
    )
    logger.bind(payload=response).debug("Presigned URL generated.")
    return response

presigned_url = s3_util.create_presigned_url(
                                s3_client=s3_client,
                                bucket_name=server_config.AWS_S3_BUCKET,
                                object_name=images_temp["file_id"], # file_id is from stored file object in database
                            )