archive storage

[toxic0berliner]

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

Describe the solution you'd like
A way to configure an archive storage (either a different path that I mount using NFS or smb or even directly the connection parameters for frigate to use SMB client himself)

• A way to configure the retention individually for main and archive storage (7 days in main storage and 90 days in archive for example)

Describe alternatives you've considered
No real idea, except running frigate twice and keeping duplicate of the "main" storage.

Additional context
Having 2 level of storage is the key here, moving data from main to archive storage can be complicated within frigate, so a first version could be to add a "cache" storage that fully duplicate the main storage but allows to setup shorter retention.
This way I could set retention to 90 days on main storage and point it to a NFS mount and add a "cache" retention of 7 days in the local SSD under /tmp.
This way it could be less complicated to implement: any recording in the db is available in main storage (NFS) if it was not found in the "cache" (SSD or /tmp)

[rhatguy]

Just thinking...could something like this be created with UnionFS and a separate script that moved all files over X days old to the "archive" filesystem. I believe UnionFS would allow frigate to still see both drives as a continuous filesystem regardless of which disk the files actually live on.

https://en.wikipedia.org/wiki/UnionFS

[NickM-27]

yeah, I don't think this is something that makes sense for frigate to integrate itself. For example, my Unraid NAS already does this as the default behavior. There are shares which consist of one or many harddrives, and all writes to a share are done to a cache drive first and those files are moved to the harddrive when the cache gets full. Frigate just sees this as one big drive.

We don't want frigate managing the native filesystems, we want one volume mount for recordings without caring what the actual file system is behind that.

[toxic0berliner]

Seems a bit of a hack but could be possible. Unionfs seems quite old though or at least esoteric...
Given frigate has to manage massive volume of data I would expect archiving or caching strategies to fit many people's requirements but I might be mistaken.

[toxic0berliner]

I get that it's tempting to rely on the intelligent FS we have nowadays. But let's be honest too it will probably not be fine grained with a retention duration set in frigate, even people using zfs and cache layers might want to remove the recordings before the cache is actually full to keep some for other uses...

Managing storage seems like a must for me for a recorder of my IP cameras. If this is not in the future of frigate I'll have to look someplace else.

It's not directly linked to this feature but consider that most NVR implement a rotation of recordings when the drive is full as one of the first feature, so managing the storage does make sense for an NVR

[NickM-27]

This can also be done with zfs which is a modern file system: https://unix.stackexchange.com/questions/316422/ssd-as-cache-for-filesystem-e-g-on-hdd-or-any-other

Other options as well: https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storage-tiering/

[NickM-27]

I agree it would be useful, as I use it myself, but I don't think it's something that makes sense for frigate to re-implement as many solutions already exist and we have many users already doing this.

[NickM-27]

It's not directly linked to this feature but consider that most NVR implement a rotation of recordings when the drive is full as one of the first feature, so managing the storage does make sense for an NVR

This is already an existing feature request that is going to be implemented in #994

Touching multiple drives / volumes and managing all of that is a whole other level that I don't think it makes sense for frigate to manage itself.

That being said, I just speak for myself and it's ultimately not my decision so it could still be decided to be done at some point.

[rhatguy]

@toxic0berliner I'm not disagreeing with your feature request, but I think where this discussion really begins to be helpful is when you talk about tiering off to a cloud provider (probably via object). I suspect many people would be interested in keeping 7 days onsite, then tiering off to S3 or backblaze. Technically this could be achieved with a similar UnionFS type approach. I feel like this really boils down to whether the frigate developers think the app needs to be intelligent enough to tier data, or whether they are going to leave that up to lower layers. My personal opinion would be that there are more beneficial features to implement than tiering storage when there are other ways to accomplish tiering.

[toxic0berliner]

Not disagreeing with you either 😀
Storage tiering makes sense I believe but I could fully understand other priorities might take precedence. As well as what kind of tier to add if object or block can be debated.
I have little doubt that it's too complicated for my level of dev anyway so I'll be patient 😜

[javydekoning]

I disagree that this should be left to the Filesystem because that limits you to a single FS (type). This also brings un-needed complexity.

It's very common for container orchestrators to have different storage classes that are easy for the user to leverage.

For example:

"Hot" mounted at: /media/frigate/recordings/hot
"Cold" mounted at: /media/frigate/recordings/cold

This allows recent recordings to be local (block/nvme/ramdisk/whatever) while the rest is moved to remote NFS for example. The only thing frigate would have to do is mv the object using a tiering policy (e.g. 7d -> mv cold, 1y -> delete).

Implementing s3 compatible storage at a later point in time would be a nice to have as well. That would allow users to keep recordings at the cloud provider of their choice (durable and cost effective).

[sstratoti]

Just sharing my experience/use case. I've taken the plunge with Frigate off of Blue Iris, and so far the experience has been excellent. Especially with the .12 release! But in my previous setup, Blue Iris allowed me to set a size limit or an age limit for a directory of storage.

It also allowed me the options to either delete or move older files to another directory as it reached that limit.

It would run a maintenance job every few minutes (let's say 10, not actually sure) to move/rotate the files out or delete them.

The same settings could be set for the "other directory". Either age or max size.

This allows you to manage the maximum space that the app should use so you can leave other space on your main drive for other applications, docker containers, files, swap, etc.

It would also allow me to keep the most recent snapshots and videos on my SSD, and move older files that I'd like to keep, but probably not access, in a slower USB 3.0 SATA external drive that is much larger.

I could move everything over to the SATA drive, but then I'll have like 990 GB of SSD drive that'll just be sitting there unused. I could copy/archive things myself using rsync or something, but then I'd lose access to them in the timeline once the main files get removed by the new .12 file rotating feature.

Unlesssss...if I move the files myself, would it be possible to update the SQLite DB to point at the new location of the files so that they aren't lost in the UI timeline? Are those entries cleaned up somehow?

[rhatguy]

Not arguing whether this approach is the "best" way to achieve what you want, BUT...if you're willing to move the files yourself you can easily use unionfs today to combine /ssd_mountpoint and /hdd_mountpoint into a new mountpoint /frigate_storage. Then you can move files in the background manually and still have everything show up to frigate. Frigate doesn't know which underlying filesystem the files live on. No need to update the database. Something like the below should accomplish what you want. Then you just need to move the files manually from /ssd_mountpoint to /hdd_mountpoint at whatever criteria you decide.

mount -t unionfs -o dirs=/ssd_montpoint:/hdd_mountpoint=ro none /frigate_storage

[NickM-27]

I run in unraid which does this automatically, it uses fuse.fs

[kravemir]

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

This is already a solution, which is to "move" recordings, and not a problem statement, of what do you want to solve in real life.

Real life problems statement might be:

• have recent recordings available as proof of a burglary or an incident:
  • what's point of recording if not to have a proof?
  • in case of physical breach, the proof of burglary should not be stored only on just one local device, which increases probability of proof material being destroyed or stolen during burglary,
• have recent recordings available on SSD for fast browsing,
• have old recordings available on HDD on NAS for archivation:
  • cover very low probability of a need to dig into those in future.

However, when this is combined with other environmental constraints:

• solution is deployed in house/SOHO/facility with metered LTE or 5G connection, and it's not possible to constantly stream out camera recordings.

Depending on environmental constraints,

• one might stream out camera footage to remote frigate instance (via VPN),
• one might record simultaneously to multiple storage devices via single frigate instance:
  • keep duration is low for SSD device,
  • keep duration is high for HDD device,
• one might record simultaneously via multiple frigate instances:
  • primary instance has got all cool features (detection, browsing, events,..) enabled,
  • secondary (archival) instance has is only storing camera streams to HDD, which has low requirements on CPU/RAM.

In combination with above, when on doesn't do constant remote streaming/storing:

• primary instance could be always sending only events out (JPEG, parts of stream,...) to remote storage,
• primary instance could be sending only events in case it's turned on (i.e. away from house mode) to save data on metered connection - when people are present at site/home:
  • storage devices aren't unattended,
  • people generate much more events, and that increases amount of data sent to remote storage,
  • also these extra events aren't important to be saved on remote.

I wouldn't focus on how to solve things, as long as it solves real life problems in given environmental constrains, then I would just go with the easiest and/or cheapest available solution.

The constraint is something, that is not in power of owner to change, i.e. not that I want to do it this way. But, that in my location, I don't have any unmetered connection available.

[sstratoti]

My issue isn't that the files need to be moved. It's that once they're moved you can't access them within the UI to look for/at past events.

When using unionfs or mergefs, my understanding is that it fills up one disk and then moves onto the next while combining it to look like a single directory.

Doing it this way makes sense if you have multiple HDs that you want to combine into one, but if I want to utilize my SSD/NVMe drive for writes and then spinners for storage/archive, is there a way to do this unless Frigate controls which disk/folder it writes to and reads from?

If it sees it as a single directory, and FIFOs the files, would it maximize the benefit of the SSD?

I'm totally up for trying this method if my understanding of this is incorrect. Thanks!

[NickM-27]

but if I want to utilize my SSD/NVMe drive for writes and then spinners for storage/archive, is there a way to do this unless Frigate controls which disk/folder it writes to and reads from?

Yes, this is exactly what unraid does via fuse.fs and it works perfectly. It appears as one volume to frigate, and the OS decides when to move files from SSD to the HDD archival storage

[sstratoti]

And this works if I'm not using unraid?

[NickM-27]

It's nothing proprietary to unraid, though unraid makes it really easy

[sstratoti]

Alright, I'll give it a try. Thank you!

[toxic0berliner]

Regardless, this is a feature request and the proposed "real life problem statement" doesn't match neither my needs nor that of others. Storage tiering is implemented in several NVR solutions and for good reasons, it can then be used in many ways to achieve whatever the user can think of building with it.

On my side I got it working with unions with a /hot that is being written to by default and a custom script+cron that moves files to /cold.

Nonetheless I feel this is not a proper solution and I still hope tiering will someday come to frigate. It has the DB and moving files shouldn't be too tricky, options for retention duration on each tear would benefit from being integrated as well as storage statistics.
On my little ssd it took me a while to find how old files should be before I move them, doing the logic for moving files until drive is 80% full starting from the oldest files was too tricky for me to di in bash and not a long term solution I believe.

[javydekoning]

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

This is already a solution, which is to "move" recordings, and not a problem statement, of what do you want to solve in real life.

Real life problems statement might be:

• have recent recordings available as proof of a burglary or an incident:

  • what's point of recording if not to have a proof?

  • in case of physical breach, the proof of burglary should not be stored only on just one local device, which increases probability of proof material being destroyed or stolen during burglary,

• have recent recordings available on SSD for fast browsing,

• have old recordings available on HDD on NAS for archivation:

  • cover very low probability of a need to dig into those in future.

However, when this is combined with other environmental constraints:

• solution is deployed in house/SOHO/facility with metered LTE or 5G connection, and it's not possible to constantly stream out camera recordings.

Depending on environmental constraints,

• one might stream out camera footage to remote frigate instance (via VPN),

• one might record simultaneously to multiple storage devices via single frigate instance:

  • keep duration is low for SSD device,

  • keep duration is high for HDD device,

• one might record simultaneously via multiple frigate instances:

  • primary instance has got all cool features (detection, browsing, events,..) enabled,

  • secondary (archival) instance has is only storing camera streams to HDD, which has low requirements on CPU/RAM.

In combination with above, when on doesn't do constant remote streaming/storing:

• primary instance could be always sending only events out (JPEG, parts of stream,...) to remote storage,

• primary instance could be sending only events in case it's turned on (i.e. away from house mode) to save data on metered connection - when people are present at site/home:

  • storage devices aren't unattended,

  • people generate much more events, and that increases amount of data sent to remote storage,

  • also these extra events aren't important to be saved on remote.

I wouldn't focus on how to solve things, as long as it solves real life problems in given environmental constrains, then I would just go with the easiest and/or cheapest available solution.

The constraint is something, that is not in power of owner to change, i.e. not that I want to do it this way. But, that in my location, I don't have any unmetered connection available.

It's manual work. It's very typical to want to store your recordings on remote, durable storage.

Yes I can move it manually but then the Frigate interface won't be able to show them.

The s3 api is widely used and there are many providers offering affordable durable storage. It would be a good option to store recordingsAND metadata.

[kravemir]

It's manual work. It's very typical to want to store your recordings on remote, durable storage.

It's typical to store recordings on remote store simultaneously while storing it locally.

This redundancy is comes useful especially when recording happens to be actually needed, such as burglary event, when the primary NVR might have been stolen in action, and therefore retain the ability to see recordings.

... that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available ...

For me, it makes no sense to implement such "move" functionality.

The most recent recording is often the most precious one. Storing it only on single non-redundant storage is increasing the risk of making security system useless - recordings stolen in action.

So, IMO, storing recordings simultaneously to multiple storages, where each storage has different retention period, is a better alternative, because it provides redundancy.

Yes I can move it manually but then the Frigate interface won't be able to show them.

The s3 api is widely used and there are many providers offering affordable durable storage. It would be a good option to store recordingsAND metadata.

S3 based storage sounds as a viable option. The frigate would see it as one "logical" storage, while the physical storage would be redundant, where SSD and HDD is used for recent recordings, and also tiered, where old recordings deleted from SSD and kept on HDD. This would be transparent to frigate, and flexible - based on S3 storage configuration and capabilities.

The minio can be self-hosted, and there's something about tiering.

Though, not sure whether it would be possible to combine locally hosted S3 with some cloud provider, transparently, where local S3 storage, which is using local low-latency SSD, would work as LRU+write cache for provided remote S3 storage. So, maybe multi-storage support would be needed on frigate level.

[fpytloun]

I just migrated to Frigate from Zoneminder and really miss this feature. My use-case is that I have local storage of limited capacity and then Wasabi S3 mounted as POSIX FS via JuiceFS (k8s PVC). I would like to have option to move records older than x days into this cold storage with longer retention. With ZM it was done simply by having filter actions.

[fpytloun]

Looking at sqlite and it stores path so implementing move script would be simple. Basically just iterate over directories which are by date, move those matching threshold into another location and execute UPDATE statement to update path in sqlite, eg.:

echo "UPDATE recordings SET path = REPLACE(path, '/media/frigate', '/media/archive') WHERE path LIKE '%/2023-11-23/%';" | sqlite3 frigate.db

UPDATE: played a little bit, this should work, just run as a cronjob:

#!/bin/bash -e

FRIGATE_BASEPATH="/media/frigate"
ARCHIVE_BASEPATH="/media/archive"

# Number of previous days to keep
KEEP_DAYS=1

for day in "${FRIGATE_BASEPATH}"/recordings/*; do
    day=$(basename "${day}")
    SKIP=0
    for range in $(seq 0 $KEEP_DAYS); do
        match=$(date -d "-${range} day" '+%Y-%m-%d')
        if [[ $day == "$match" ]]; then
           SKIP=1
           break
        fi
    done

    [[ $SKIP -ne 1 ]] || continue

    echo "[INFO] day=${day} Copying recordings into archive storage"
    cp -ra "${FRIGATE_BASEPATH}/recordings/${day}" "${ARCHIVE_BASEPATH}/recordings/"
    echo "[INFO] day=${day} Updating paths in sqlite"
    echo "UPDATE recordings SET path = REPLACE(path, '${FRIGATE_BASEPATH}', '${ARCHIVE_BASEPATH}') WHERE path LIKE '%/${day}/%';" | sqlite3 "${FRIGATE_BASEPATH}/frigate.db"
    echo "[INFO] day=${day} Deleting original files on hot storage"
    rm -rf "${FRIGATE_BASEPATH}/recordings/${day}"
done

Enjoy 😄

[figadore]

this should work

This looks great, have you tested it? This will still work with all of Frigate's "retain" logic, right? I'd love to see something similar as a built-in feature, but it sounds like this will do the trick until then.

[WilldabeastHA]

@ckglobalroaming I just wanna make sure I understand what you’re saying. You have 2 Frigate’s running. you copy/move footage at some point from frigate #1 (NVMe drive) to frigate #2 (spinning disk). Are you just copying the files from one to another and frigate will import/ recognize the file change and in UI it reflects this?

thanks!

[alsobrsp]

@fpytloun This looks perfect for my use case. I am currently writing everything to NFS. Most of the other suggested solutions seem to be targeting local storage. I want the last day or so to be local with the rest remote.

[fpytloun]

Here is new version of my archivation script.

Changes:

• upload is now performed per file separately so process can be restarted at any time and also it can be run more frequently
• it supports KEEP_DAYS=-1 which will migrate everything
• added retry function to bypass database locking. Would not be needed if frigate has option to use WAL journal type
• cleanup for empty leftover directories

    #!/bin/bash -e

    FRIGATE_BASEPATH="/media/frigate"
    ARCHIVE_BASEPATH="/media/archive"

    # Number of previous days to keep (-1 means copy everything)
    KEEP_DAYS=-1

    function retry {
      local retries=$1
      shift

      local count=0
      until "$@"; do
        exit=$?
        wait=$((2 ** $count))
        count=$(($count + 1))
        if [ $count -lt $retries ]; then
          echo "Retry $count/$retries exited $exit, retrying in $wait seconds..."
          sleep $wait
        else
          echo "Retry $count/$retries exited $exit, no more retries left."
          return $exit
        fi
      done
      return 0
    }

    for day in "${FRIGATE_BASEPATH}"/recordings/*; do
        day=$(basename "${day}")
        SKIP=0
        for range in $(seq 0 $KEEP_DAYS); do
            match=$(date -d "-${range} day" '+%Y-%m-%d')
            if [[ $day == "$match" ]]; then
                SKIP=1
                break
            fi
        done

        [[ $SKIP -ne 1 ]] || continue

        echo "[INFO] day=${day} Copying recordings into archive storage"
        find "${FRIGATE_BASEPATH}/recordings/${day}" -type f | while read recording; do
          target_file=${recording/$FRIGATE_BASEPATH/$ARCHIVE_BASEPATH}
          target_dir=$(dirname ${target_file})
          [ -d "${target_dir}" ] || mkdir -p "${target_dir}"

          echo "[INFO] day=${day}, file=${recording} Copying into ${target_file}"
          cp -a "${recording}" "${target_file}"

          echo "[INFO] day=${day}, file=${recording} Updating path in sqlite"
          echo "UPDATE recordings SET path = '${target_file}' WHERE path = '${recording}'" | retry 5 sqlite3 "${FRIGATE_BASEPATH}/frigate.db"

          echo "[INFO] day=${day}, file=${recording} Deleting original file"
          rm -f "${recording}"
        done
    done

    echo "[INFO] Cleaning up empty directories on archive storage"
    find "${ARCHIVE_BASEPATH}" -type d -empty -delete

    echo "[INFO] Cleaning up empty directories on hot storage"
    find "${ARCHIVE_BASEPATH}" -type d -empty -delete

[NickM-27]

added retry function to bypass database locking. Would not be needed if frigate has option to use WAL journal type

Frigate does use the WAL journal type

[fpytloun]

added retry function to bypass database locking. Would not be needed if frigate has option to use WAL journal type

Frigate does use the WAL journal type

Interesting. Even when I used PRAGMA journal_mode=WAL; prior to UPDATE statement I sometimes hit database locked error.

[NickM-27]

Right, because there is still a write timeout that results in a locked error. If something is writing and something else waits for longer than that timeout it will have a locked error regardless of if it's using wal or not.

[pgiansante]

I tried the latest script above and I lose all access to the events via frigate app.
Does this work with the 0.13.2?

[fpytloun]

I tried the latest script above and I lose all access to the events via frigate app. Does this work with the 0.13.2?

Yes, I am using it and it works fine.

[jakubsuchybio]

Hi guys, just want to report my findings with trying to use unionfs, then mergefs.
I managed to get it to work.
So basically this setup:

• local mount to 1TB SSD
• NFS mount to 12TB NAS
• union/merge fs mount which merged two above
  this way the final storage said it has 13TB and writes from frigate would go defaultly to the first storage.
  So the frigate thinks it saves to one storage, but they are actually 2 tiers.
  Then I had a cron which would sync from local to NFS daily to clear up 300GB on the local storage. (My cameras takes like 150GB/day) so this would basically create a buffer for 2 days if something would go wrong at one day.

And go wrong it went constantly :(.
I don't have stable connection between my 2 locations (the NFS mount is not located on local storage, but across internet via VPN)
My network would sometime stall and disconnect.
That the unionfs and mergefs can't survive. They have some politics for disconnects, but I couldn't get them to survive disconnects.
Only way to restore the union/mergefs would be by restarting whole frigate vm :(

Having only archiving without being able to view the archive from frigate seems kinda pointless to me, that is why I was trying this solution.

One last solution that comes to my mind is to write custom script/program, which would basically do something similar to union/mergefs but with symlinks.
So when the local storage gets full, it will move oldest to the NFS archive and replace it locally with a symlink.
But that creates new problem. When the frigate tries to remove that by the retention, it will just remove the symlink and not the target of the link on the archive storage.
So to get it to work I would have to keep track of the symlinks on some db (file, sql, or something) and when going through, if some symlink would get deleted by frigate, but it was still tracked in the db, the script/program would delete it also from the NFS archive..

As I'm thinking about it, it might actually work. I will try to do that sometime soon and report back the findings.

[toxic0berliner]

I think this feature request is quite enough justified, I believe it's only waiting for some dev to start tackling it.
My opinion is archive will be slow and potentially subject to issues during transfer, but frigate could still let the user mount whatever archive storage as a separate path. Seems to me a more reasonable starting point than aiming at handling smb,NFS,S3,....
For me frigate should have 2 new settings: a keep time after which files are moved to the archive, and a path to said archive.
It should handle the copy and delete files from the main storage only once the archive has been successfully written. It should log any failure but also simply retry the transfer at the next occasion.
It should also handle actual deletion from the archive instead of the main storage when the usual keep policy expires.
Storage page could also benefit from being modified to display main and archive storage space.
I don't think it should be more complicated than that. (That said I also believe it's too much for me to try implementing so thanks in advance to anyone willing to tackle it)

[johntdyer]

One last solution that comes to my mind is to write custom script/program, which would basically do something similar to union/mergefs but with symlinks.
So when the local storage gets full, it will move oldest to the NFS archive and replace it locally with a symlink.
But that creates new problem. When the frigate tries to remove that by the retention, it will just remove the symlink and not the target of the link on the archive storage.
So to get it to work I would have to keep track of the symlinks on some db (file, sql, or something) and when going through, if some symlink would get deleted by frigate, but it was still tracked in the db, the script/program would delete it also from the NFS archive..

@jakubsuchybio - Would rsync w/ the delete flag work ? Just thinking when Frigate deletes the linked file the rsync might recognize this ( untested ) and delete the linked file on the other storage... Just an untested idea.... pre-coffee :)

[fpytloun]

@jakubsuchybio Frigate recordings has path in database so you can have multiple paths instead of using mergefs which seems to be very fragile. So why don't you use my script? I am using this so I have 1 day on local storage and then move recordings to JuiceFS volume (using Wasabi S3 backend). I can view both local and remote recordings just fine and remote storage instability (eg Internet outage) does not cause any harm.
#3673 (comment)

[jakubsuchybio]

@fpytloun

Juicefs, didn't hear of it, will look at it, thx for hint

yeah i wouldn't dare to modify db paths 😁

[zzachattack2]

+1 for the feature request for tiered storage. I think the flexibility that Frigate already has on what to record, and how long to retain it, are great, but the missing part of where is a big shortcoming IMO -- especially considering the volume of data that can be involved here.

Just throwing out some ideas, but what I think would be ideal is if we could at least identify a "hot-storage" recording path and a "cold-storage" recording path, and each recording 'option' that currently allows us to specify a retention period, could also specify an archiving period before moving to cold storage.

Example use case:
I want to keep a week's worth 24/7 recordings (~4TB), which means I pretty much need to store that on spinning platters. For recent footage of particular importance (e.g events with a person identified), which are substantially more likely to actually be viewed, it would be better to have it available on a faster storage medium. I also want to keep those events for much longer (say 30 days), but after a few days, the need for them on fast storage largely goes away (along with available space on the SSD), and they should be moved over to cold storage.

So a hypothetical configuration could look like:

record:
  enabled: True
  hot_path: /data/hot
  cold_path: /data/cold
  retain:
    mode: all
    days: 7
    archive: 0 #don't use hot-storage
  events:
    retain:
      mode: active_objects
      default: 
        days: 14
        archive: 1 #move to cold-storage after 1 day
      objects:
        person:
          days: 30
          archive: 3 #move to cold-storage after 3 days

[javydekoning]

+1 for the feature request for tiered storage. I think the flexibility that Frigate already has on what to record, and how long to retain it, are great, but the missing part of where is a big shortcoming IMO -- especially considering the volume of data that can be involved here.

Just throwing out some ideas, but what I think would be ideal is if we could at least identify a "hot-storage" recording path and a "cold-storage" recording path, and each recording 'option' that currently allows us to specify a retention period, could also specify an archiving period before moving to cold storage.

Example use case: I want to keep a week's worth 24/7 recordings (~4TB), which means I pretty much need to store that on spinning platters. For recent footage of particular importance (e.g events with a person identified), which are substantially more likely to actually be viewed, it would be better to have it available on a faster storage medium. I also want to keep those events for much longer (say 30 days), but after a few days, the need for them on fast storage largely goes away (along with available space on the SSD), and they should be moved over to cold storage.

So a hypothetical configuration could look like:

record:
  enabled: True
  hot_path: /data/hot
  cold_path: /data/cold
  retain:
    mode: all
    days: 7
    archive: 0 #don't use hot-storage
  events:
    retain:
      mode: active_objects
      default: 
        days: 14
        archive: 1 #move to cold-storage after 1 day
      objects:
        person:
          days: 30
          archive: 3 #move to cold-storage after 3 days

I like this and suggest splitting the datastore to it's own object given the (future) possibility for remote/cloud storage:

  datastore:
  - name: my-hot-volume
    type: hostPath
    config:
      path: /data/hot
  - name: my-cold-bucket
    type: s3
    config:
      bucket: s3://my-bucket
      prefix: /frigate
      storage_class: 'STANDARD'

[luckylinux]

Not sure what is wrong with rsync.

I agree that the --delete (and similar --delete-after. --delete-before, ...) Options are MISLEADING: they will REMOVE from the DESTINATION Directory if the File doesn't exist anymore in the SOURCE Directory ! Which it will not since we just moved it (Duh).

What should work (I just tried it now for moving from SSD to HDD for most of the Storage) is to use the --remove-source-files Flag.
Example (I did this under Fedora and I disabled the -X Flag since it was triggering many Errors due to SELinux apparently not working on a NFS mounted Share):

rsync -aAUvH --remove-source-files /home/podman/containers/data/frigate-server/ /mnt/frigate

Maybe also the --prune-empty-dirs could be useful.

Of course we can also agree that this is not perfect because it will just move EVERYTHING.

[jakubsuchybio]

Not sure what is wrong with rsync.

I agree that the --delete (and similar --delete-after. --delete-before, ...) Options are MISLEADING: they will REMOVE from the DESTINATION Directory if the File doesn't exist anymore in the SOURCE Directory ! Which it will not since we just moved it (Duh).

What should work (I just tried it now for moving from SSD to HDD for most of the Storage) is to use the --remove-source-files Flag. Example (I did this under Fedora and I disabled the -X Flag since it was triggering many Errors due to SELinux apparently not working on a NFS mounted Share):

rsync -aAUvH --remove-source-files /home/podman/containers/data/frigate-server/ /mnt/frigate

Maybe also the --prune-empty-dirs could be useful.

Of course we can also agree that this is not perfect because it will just move EVERYTHING.

Ofc this would work for archival purposes.
But that is not what we wanted.
We wanted some solution with an ability to look at the archived videos through frigate UI. Moving it will basically remove it from UI. => Not useful.

[luckylinux]

Not sure what is wrong with rsync.
I agree that the --delete (and similar --delete-after. --delete-before, ...) Options are MISLEADING: they will REMOVE from the DESTINATION Directory if the File doesn't exist anymore in the SOURCE Directory ! Which it will not since we just moved it (Duh).
What should work (I just tried it now for moving from SSD to HDD for most of the Storage) is to use the --remove-source-files Flag. Example (I did this under Fedora and I disabled the -X Flag since it was triggering many Errors due to SELinux apparently not working on a NFS mounted Share):

rsync -aAUvH --remove-source-files /home/podman/containers/data/frigate-server/ /mnt/frigate

Maybe also the --prune-empty-dirs could be useful.
Of course we can also agree that this is not perfect because it will just move EVERYTHING.

Ofc this would work for archival purposes. But that is not what we wanted. We wanted some solution with an ability to look at the archived videos through frigate UI. Moving it will basically remove it from UI. => Not useful.

Sure, but I thought @fpytloun Script had the same Issue (move files between 2 Storages) and only UnionFS/FUSE (for those of us without UnRAID) were the only Options to "merge" those Storages together.

Or has @javydekoning Configuration been implemented in the latest Release already ?

[fvdpol]

if you want to keep access to the archived videos from the frigate UI I suggest to have a look at script written by @fpytloun. I find it works quite well and definitely an acceptable workaround until archiving / tiered storage becomes part of the frigate feature set.

(see #3673 (comment))

The script moves files (depending on the age) to another location, and then updates the frigate database to reference the new location. The resulting "move" is therefore transparent to frigate/ui. This also makes it robust in case the migration gets stalled/stopped mid-way.

[luckylinux]

Aaah alright. Yeah I remembered a few Days I saw that SQL Query (don't know why I didn't see it again today, I must be too tired 🤣).

[jakubsuchybio]

if you want to keep access to the archived videos from the frigate UI I suggest to have a look at script written by @fpytloun. I find it works quite well and definitely an acceptable workaround until archiving / tiered storage becomes part of the frigate feature set.

(see #3673 (comment))

The script moves files (depending on the age) to another location, and then updates the frigate database to reference the new location. The resulting "move" is therefore transparent to frigate/ui. This also makes it robust in case the migration gets stalled/stopped mid-way.

Uff thanks for reminding. I totally missed that script and was waiting for that tiered/archival process from frigate. Will give it a try.

[iscofield]

Another person migrating from BI and was looking for a way to have the tiered storage. Another +1 from me for this to eventually come.

[cammurray]

Throwing my support against this one. +1

The issue for me is where the storage is located, my heavy disks are in the garage (they're noisy, butthey could be stolen) where my fast disks are in the house (because they're SSD and silent). I'd like to record to the fast disks and then after x days move them to the slow disk.

The scripted solution is fine, but it's a bit of a hack. Anything that is updating the database in parallel to how it's expected, is it at risk of being nuked in some kind of future update.

[magma1447]

The issue for me is where the storage is located, my heavy disks are in the garage (they're noisy, butthey could be stolen) where my fast disks are in the house (because they're SSD and silent). I'd like to record to the fast disks and then after x days move them to the slow disk.

The scripted solution is fine, but it's a bit of a hack. Anything that is updating the database in parallel to how it's expected, is it at risk of being nuked in some kind of future update.

More or less exactly my case as well. I have just been following the thread until now since I respect that it's not (currently?) in the scope of the project. But for bigger installations it's a really nice feature.

[toxic0berliner]

I've been using unionfs and my custom archive script to periodically move files from SSD to HDD.
It works reliably and fine but I figured running unionfs in my CT (proxmox) is the reason I'm stuck doing cold backups and can't do snapshot backups...

I would love for someone to take that up. Meanwhile I will give a try to manipulation of the db and the file location...

[cammurray]

I've been using unionfs and my custom archive script to periodically move files from SSD to HDD.
It works reliably and fine but I figured running unionfs in my CT (proxmox) is the reason I'm stuck doing cold backups and can't do snapshot backups...

When using unionfs, are you having to modify the database? or does the path remain the same (just the underlying filesystem changes)?

[toxic0berliner]

with unionfs I have

• /hot mounted on an ssd
• /cold mounted on an hdd
• /merged is the union of the 2
  so I pass /merged to frigate, and I have a script in crontab that check on /hot for files older than 1 day and simply mv them from /hot to /cold

they remain all available on /merged and therefore no fussing around in the frigate DB or whatever, all remains available... but... this uses fuse and makes my snapshot backups fail :-(

so I believe someone found we can edit the DB to change the path of some files in the DB, I assume if we ensure to move the files and make them available on the new path it should all be fine too... A bit worried editing what seems to me an sqlite db that frigate is suceptible to open/edit in // whenever it pleases, but hey...