Skip to content

Commit

Permalink
Refactor recordings retention to be based on review items (#13355)
Browse files Browse the repository at this point in the history 
* Refactor recordings config to be based off of review items

* Update object processing logic for when an event is created

* Migrate to deciding recording retention based on review items

* Refactor recording expiration to be based off of review items

* Remove remainder of recording events access

* Handle migration automatically

* Update version and cleanup

* Update docs

* Clarify docs

* Cleanup

* Target camera config

* Safely access all fields
  • Loading branch information
@NickM-27
NickM-27 authored Sep 2, 2024
1 parent e3da5ef commit 0acbd3d
Show file tree
Hide file tree
Showing 11 changed files with 364 additions and 279 deletions.
73 changes: 32 additions & 41 deletions docs/docs/configuration/record.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,21 @@ H265 recordings can be viewed in Chrome 108+, Edge and Safari only. All other br

### Most conservative: Ensure all video is saved

For users deploying Frigate in environments where it is important to have contiguous video stored even if there was no detectable motion, the following config will store all video for 3 days. After 3 days, only video containing motion and overlapping with events will be retained until 30 days have passed.
For users deploying Frigate in environments where it is important to have contiguous video stored even if there was no detectable motion, the following config will store all video for 3 days. After 3 days, only video containing motion and overlapping with alerts or detections will be retained until 30 days have passed.

```yaml
record:
enabled: True
retain:
days: 3
mode: all
events:
alerts:
retain:
default: 30
days: 30
mode: motion
detections:
retain:
days: 30
mode: motion
```
Expand All @@ -37,25 +41,28 @@ record:
retain:
days: 3
mode: motion
events:
alerts:
retain:
days: 30
mode: motion
detections:
retain:
default: 30
days: 30
mode: motion
```
### Minimum: Events only
### Minimum: Alerts only
If you only want to retain video that occurs during an event, this config will discard video unless an event is ongoing.
If you only want to retain video that occurs during an event, this config will discard video unless an alert is ongoing.
```yaml
record:
enabled: True
retain:
days: 0
mode: all
events:
alerts:
retain:
default: 30
days: 30
mode: motion
```
Expand Down Expand Up @@ -86,19 +93,22 @@ record:

Continuous recording supports different retention modes [which are described below](#what-do-the-different-retain-modes-mean)

### Event Recording
### Object Recording

If you only used clips in previous versions with recordings disabled, you can use the following config to get the same behavior. This is also the default behavior when recordings are enabled.
The number of days to record review items can be specified for review items classified as alerts as well as events.

```yaml
record:
enabled: True
events:
alerts:
retain:
days: 10 # <- number of days to keep alert recordings
detections:
retain:
default: 10 # <- number of days to keep event recordings
days: 10 # <- number of days to keep detections recordings
```

This configuration will retain recording segments that overlap with events and have active tracked objects for 10 days. Because multiple events can reference the same recording segments, this avoids storing duplicate footage for overlapping events and reduces overall storage needs.
This configuration will retain recording segments that overlap with alerts and detections for 10 days. Because multiple events can reference the same recording segments, this avoids storing duplicate footage for overlapping events and reduces overall storage needs.

**WARNING**: Recordings still must be enabled in the config. If a camera has recordings disabled in the config, enabling via the methods listed above will have no effect.

Expand All @@ -112,11 +122,7 @@ Let's say you have Frigate configured so that your doorbell camera would retain
- With the `motion` option the only parts of those 48 hours would be segments that Frigate detected motion. This is the middle ground option that won't keep all 48 hours, but will likely keep all segments of interest along with the potential for some extra segments.
- With the `active_objects` option the only segments that would be kept are those where there was a true positive object that was not considered stationary.

The same options are available with events. Let's consider a scenario where you drive up and park in your driveway, go inside, then come back out 4 hours later.

- With the `all` option all segments for the duration of the event would be saved for the event. This event would have 4 hours of footage.
- With the `motion` option all segments for the duration of the event with motion would be saved. This means any segment where a car drove by in the street, person walked by, lighting changed, etc. would be saved.
- With the `active_objects` it would only keep segments where the object was active. In this case the only segments that would be saved would be the ones where the car was driving up, you going inside, you coming outside, and the car driving away. Essentially reducing the 4 hours to a minute or two of event footage.
The same options are available with alerts and detections, except it will only save the recordings when it overlaps with a review item of that type.

A configuration example of the above retain modes where all `motion` segments are stored for 7 days and `active objects` are stored for 14 days would be as follows:

Expand All @@ -126,33 +132,18 @@ record:
retain:
days: 7
mode: motion
events:
alerts:
retain:
default: 14
days: 14
mode: active_objects
```

The above configuration example can be added globally or on a per camera basis.

### Object Specific Retention

You can also set specific retention length for an object type. The below configuration example builds on from above but also specifies that recordings of dogs only need to be kept for 2 days and recordings of cars should be kept for 7 days.

```yaml
record:
enabled: True
retain:
days: 7
mode: motion
events:
detections:
retain:
default: 14
days: 14
mode: active_objects
objects:
dog: 2
car: 7
```

The above configuration example can be added globally or on a per camera basis.

## Can I have "continuous" recordings, but only at certain times?

Using Frigate UI, HomeAssistant, or MQTT, cameras can be automated to only record in certain situations or at certain times.
Expand Down
46 changes: 30 additions & 16 deletions docs/docs/configuration/reference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -419,32 +419,46 @@ record:
# Optional: Quality of recording preview (default: shown below).
# Options are: very_low, low, medium, high, very_high
quality: medium
# Optional: Event recording settings
events:
# Optional: Number of seconds before the event to include (default: shown below)
# Optional: alert recording settings
alerts:
# Optional: Number of seconds before the alert to include (default: shown below)
pre_capture: 5
# Optional: Number of seconds after the event to include (default: shown below)
# Optional: Number of seconds after the alert to include (default: shown below)
post_capture: 5
# Optional: Objects to save recordings for. (default: all tracked objects)
objects:
- person
# Optional: Retention settings for recordings of events
# Optional: Retention settings for recordings of alerts
retain:
# Required: Default retention days (default: shown below)
default: 10
# Required: Retention days (default: shown below)
days: 14
# Optional: Mode for retention. (default: shown below)
# all - save all recording segments for events regardless of activity
# motion - save all recordings segments for events with any detected motion
# active_objects - save all recording segments for event with active/moving objects
# all - save all recording segments for alerts regardless of activity
# motion - save all recordings segments for alerts with any detected motion
# active_objects - save all recording segments for alerts with active/moving objects
#
# NOTE: If the retain mode for the camera is more restrictive than the mode configured
# here, the segments will already be gone by the time this mode is applied.
# For example, if the camera retain mode is "motion", the segments without motion are
# never stored, so setting the mode to "all" here won't bring them back.
mode: motion
# Optional: detection recording settings
detections:
# Optional: Number of seconds before the detection to include (default: shown below)
pre_capture: 5
# Optional: Number of seconds after the detection to include (default: shown below)
post_capture: 5
# Optional: Retention settings for recordings of detections
retain:
# Required: Retention days (default: shown below)
days: 14
# Optional: Mode for retention. (default: shown below)
# all - save all recording segments for detections regardless of activity
# motion - save all recordings segments for detections with any detected motion
# active_objects - save all recording segments for detections with active/moving objects
#
# NOTE: If the retain mode for the camera is more restrictive than the mode configured
# here, the segments will already be gone by the time this mode is applied.
# For example, if the camera retain mode is "motion", the segments without motion are
# never stored, so setting the mode to "all" here won't bring them back.
mode: motion
# Optional: Per object retention days
objects:
person: 15

# Optional: Configuration for the jpg snapshots written to the clips directory for each event
# NOTE: Can be overridden at the camera level
Expand Down
52 changes: 30 additions & 22 deletions frigate/config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -296,33 +296,21 @@ class RetainModeEnum(str, Enum):
active_objects = "active_objects"


class RetainConfig(FrigateBaseModel):
default: float = Field(default=10, title="Default retention period.")
mode: RetainModeEnum = Field(default=RetainModeEnum.motion, title="Retain mode.")
objects: Dict[str, float] = Field(
default_factory=dict, title="Object retention period."
)
class RecordRetainConfig(FrigateBaseModel):
days: float = Field(default=0, title="Default retention period.")
mode: RetainModeEnum = Field(default=RetainModeEnum.all, title="Retain mode.")


class EventsConfig(FrigateBaseModel):
pre_capture: int = Field(
default=5, title="Seconds to retain before event starts.", le=MAX_PRE_CAPTURE
)
post_capture: int = Field(default=5, title="Seconds to retain after event ends.")
objects: Optional[List[str]] = Field(
None,
title="List of objects to be detected in order to save the event.",
)
retain: RetainConfig = Field(
default_factory=RetainConfig, title="Event retention settings."
retain: RecordRetainConfig = Field(
default_factory=RecordRetainConfig, title="Event retention settings."
)


class RecordRetainConfig(FrigateBaseModel):
days: float = Field(default=0, title="Default retention period.")
mode: RetainModeEnum = Field(default=RetainModeEnum.all, title="Retain mode.")


class RecordExportConfig(FrigateBaseModel):
timelapse_args: str = Field(
default=DEFAULT_TIME_LAPSE_FFMPEG_ARGS, title="Timelapse Args"
Expand Down Expand Up @@ -355,8 +343,11 @@ class RecordConfig(FrigateBaseModel):
retain: RecordRetainConfig = Field(
default_factory=RecordRetainConfig, title="Record retention settings."
)
events: EventsConfig = Field(
default_factory=EventsConfig, title="Event specific settings."
detections: EventsConfig = Field(
default_factory=EventsConfig, title="Detection specific retention settings."
)
alerts: EventsConfig = Field(
default_factory=EventsConfig, title="Alert specific retention settings."
)
export: RecordExportConfig = Field(
default_factory=RecordExportConfig, title="Recording Export Config"
Expand Down Expand Up @@ -924,6 +915,14 @@ def validate_roles(cls, v):
return v


class RetainConfig(FrigateBaseModel):
default: float = Field(default=10, title="Default retention period.")
mode: RetainModeEnum = Field(default=RetainModeEnum.motion, title="Retain mode.")
objects: Dict[str, float] = Field(
default_factory=dict, title="Object retention period."
)


class SnapshotsConfig(FrigateBaseModel):
enabled: bool = Field(default=False, title="Snapshots enabled.")
clean_copy: bool = Field(
Expand Down Expand Up @@ -1278,10 +1277,19 @@ def verify_recording_retention(camera_config: CameraConfig) -> None:
if (
camera_config.record.retain.days != 0
and rank_map[camera_config.record.retain.mode]
> rank_map[camera_config.record.events.retain.mode]
> rank_map[camera_config.record.alerts.retain.mode]
):
logger.warning(
f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and alert retention is configured for {camera_config.record.alerts.retain.mode}. The more restrictive retention policy will be applied."
)

if (
camera_config.record.retain.days != 0
and rank_map[camera_config.record.retain.mode]
> rank_map[camera_config.record.detections.retain.mode]
):
logger.warning(
f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and event retention is configured for {camera_config.record.events.retain.mode}. The more restrictive retention policy will be applied."
f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and detection retention is configured for {camera_config.record.detections.retain.mode}. The more restrictive retention policy will be applied."
)


Expand Down Expand Up @@ -1429,7 +1437,7 @@ class FrigateConfig(FrigateBaseModel):
default_factory=TimestampStyleConfig,
title="Global timestamp style configuration.",
)
version: Optional[float] = Field(default=None, title="Current config version.")
version: Optional[str] = Field(default=None, title="Current config version.")

def runtime_config(self, plus_api: PlusApi = None) -> FrigateConfig:
"""Merge camera config with globals."""
Expand Down
24 changes: 18 additions & 6 deletions frigate/events/cleanup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,10 @@ def get_camera_labels(self, camera: str) -> list[Event]:
def expire(self, media_type: EventCleanupType) -> list[str]:
## Expire events from unlisted cameras based on the global config
if media_type == EventCleanupType.clips:
retain_config = self.config.record.events.retain
expire_days = max(
self.config.record.alerts.retain.days,
self.config.record.detections.retain.days,
)
file_extension = None # mp4 clips are no longer stored in /clips
update_params = {"has_clip": False}
else:
Expand All @@ -82,7 +85,11 @@ def expire(self, media_type: EventCleanupType) -> list[str]:
# loop over object types in db
for event in distinct_labels:
# get expiration time for this label
expire_days = retain_config.objects.get(event.label, retain_config.default)
if media_type == EventCleanupType.snapshots:
expire_days = retain_config.objects.get(
event.label, retain_config.default
)

expire_after = (
datetime.datetime.now() - datetime.timedelta(days=expire_days)
).timestamp()
Expand Down Expand Up @@ -132,7 +139,10 @@ def expire(self, media_type: EventCleanupType) -> list[str]:
## Expire events from cameras based on the camera config
for name, camera in self.config.cameras.items():
if media_type == EventCleanupType.clips:
retain_config = camera.record.events.retain
expire_days = max(
camera.record.alerts.retain.days,
camera.record.detections.retain.days,
)
else:
retain_config = camera.snapshots.retain

Expand All @@ -142,9 +152,11 @@ def expire(self, media_type: EventCleanupType) -> list[str]:
# loop over object types in db
for event in distinct_labels:
# get expiration time for this label
expire_days = retain_config.objects.get(
event.label, retain_config.default
)
if media_type == EventCleanupType.snapshots:
expire_days = retain_config.objects.get(
event.label, retain_config.default
)

expire_after = (
datetime.datetime.now() - datetime.timedelta(days=expire_days)
).timestamp()
Expand Down
9 changes: 3 additions & 6 deletions frigate/events/maintainer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from typing import Dict

from frigate.comms.events_updater import EventEndPublisher, EventUpdateSubscriber
from frigate.config import EventsConfig, FrigateConfig
from frigate.config import FrigateConfig
from frigate.events.types import EventStateEnum, EventTypeEnum
from frigate.models import Event
from frigate.util.builtin import to_relative_box
Expand Down Expand Up @@ -128,16 +128,13 @@ def handle_object_detection(
if should_update_db(self.events_in_process[event_data["id"]], event_data):
updated_db = True
camera_config = self.config.cameras[camera]
event_config: EventsConfig = camera_config.record.events
width = camera_config.detect.width
height = camera_config.detect.height
first_detector = list(self.config.detectors.values())[0]

start_time = event_data["start_time"] - event_config.pre_capture
start_time = event_data["start_time"]
end_time = (
None
if event_data["end_time"] is None
else event_data["end_time"] + event_config.post_capture
None if event_data["end_time"] is None else event_data["end_time"]
)
# score of the snapshot
score = (
Expand Down
Loading

0 comments on commit 0acbd3d

Please sign in to comment.