diff --git a/docs/docs/configuration/record.md b/docs/docs/configuration/record.md index 09ae420f97..8ba12a30d9 100644 --- a/docs/docs/configuration/record.md +++ b/docs/docs/configuration/record.md @@ -13,7 +13,7 @@ H265 recordings can be viewed in Chrome 108+, Edge and Safari only. All other br ### Most conservative: Ensure all video is saved -For users deploying Frigate in environments where it is important to have contiguous video stored even if there was no detectable motion, the following config will store all video for 3 days. After 3 days, only video containing motion and overlapping with events will be retained until 30 days have passed. +For users deploying Frigate in environments where it is important to have contiguous video stored even if there was no detectable motion, the following config will store all video for 3 days. After 3 days, only video containing motion and overlapping with alerts or detections will be retained until 30 days have passed. ```yaml record: @@ -21,9 +21,13 @@ record: retain: days: 3 mode: all - events: + alerts: retain: - default: 30 + days: 30 + mode: motion + detections: + retain: + days: 30 mode: motion ``` @@ -37,25 +41,28 @@ record: retain: days: 3 mode: motion - events: + alerts: + retain: + days: 30 + mode: motion + detections: retain: - default: 30 + days: 30 mode: motion ``` -### Minimum: Events only +### Minimum: Alerts only -If you only want to retain video that occurs during an event, this config will discard video unless an event is ongoing. +If you only want to retain video that occurs during an event, this config will discard video unless an alert is ongoing. ```yaml record: enabled: True retain: days: 0 - mode: all - events: + alerts: retain: - default: 30 + days: 30 mode: motion ``` @@ -86,19 +93,22 @@ record: Continuous recording supports different retention modes [which are described below](#what-do-the-different-retain-modes-mean) -### Event Recording +### Object Recording -If you only used clips in previous versions with recordings disabled, you can use the following config to get the same behavior. This is also the default behavior when recordings are enabled. +The number of days to record review items can be specified for review items classified as alerts as well as events. ```yaml record: enabled: True - events: + alerts: + retain: + days: 10 # <- number of days to keep alert recordings + detections: retain: - default: 10 # <- number of days to keep event recordings + days: 10 # <- number of days to keep detections recordings ``` -This configuration will retain recording segments that overlap with events and have active tracked objects for 10 days. Because multiple events can reference the same recording segments, this avoids storing duplicate footage for overlapping events and reduces overall storage needs. +This configuration will retain recording segments that overlap with alerts and detections for 10 days. Because multiple events can reference the same recording segments, this avoids storing duplicate footage for overlapping events and reduces overall storage needs. **WARNING**: Recordings still must be enabled in the config. If a camera has recordings disabled in the config, enabling via the methods listed above will have no effect. @@ -112,11 +122,7 @@ Let's say you have Frigate configured so that your doorbell camera would retain - With the `motion` option the only parts of those 48 hours would be segments that Frigate detected motion. This is the middle ground option that won't keep all 48 hours, but will likely keep all segments of interest along with the potential for some extra segments. - With the `active_objects` option the only segments that would be kept are those where there was a true positive object that was not considered stationary. -The same options are available with events. Let's consider a scenario where you drive up and park in your driveway, go inside, then come back out 4 hours later. - -- With the `all` option all segments for the duration of the event would be saved for the event. This event would have 4 hours of footage. -- With the `motion` option all segments for the duration of the event with motion would be saved. This means any segment where a car drove by in the street, person walked by, lighting changed, etc. would be saved. -- With the `active_objects` it would only keep segments where the object was active. In this case the only segments that would be saved would be the ones where the car was driving up, you going inside, you coming outside, and the car driving away. Essentially reducing the 4 hours to a minute or two of event footage. +The same options are available with alerts and detections, except it will only save the recordings when it overlaps with a review item of that type. A configuration example of the above retain modes where all `motion` segments are stored for 7 days and `active objects` are stored for 14 days would be as follows: @@ -126,33 +132,18 @@ record: retain: days: 7 mode: motion - events: + alerts: retain: - default: 14 + days: 14 mode: active_objects -``` - -The above configuration example can be added globally or on a per camera basis. - -### Object Specific Retention - -You can also set specific retention length for an object type. The below configuration example builds on from above but also specifies that recordings of dogs only need to be kept for 2 days and recordings of cars should be kept for 7 days. - -```yaml -record: - enabled: True - retain: - days: 7 - mode: motion - events: + detections: retain: - default: 14 + days: 14 mode: active_objects - objects: - dog: 2 - car: 7 ``` +The above configuration example can be added globally or on a per camera basis. + ## Can I have "continuous" recordings, but only at certain times? Using Frigate UI, HomeAssistant, or MQTT, cameras can be automated to only record in certain situations or at certain times. diff --git a/docs/docs/configuration/reference.md b/docs/docs/configuration/reference.md index 9cf998d6bc..fa5471f976 100644 --- a/docs/docs/configuration/reference.md +++ b/docs/docs/configuration/reference.md @@ -419,32 +419,46 @@ record: # Optional: Quality of recording preview (default: shown below). # Options are: very_low, low, medium, high, very_high quality: medium - # Optional: Event recording settings - events: - # Optional: Number of seconds before the event to include (default: shown below) + # Optional: alert recording settings + alerts: + # Optional: Number of seconds before the alert to include (default: shown below) pre_capture: 5 - # Optional: Number of seconds after the event to include (default: shown below) + # Optional: Number of seconds after the alert to include (default: shown below) post_capture: 5 - # Optional: Objects to save recordings for. (default: all tracked objects) - objects: - - person - # Optional: Retention settings for recordings of events + # Optional: Retention settings for recordings of alerts retain: - # Required: Default retention days (default: shown below) - default: 10 + # Required: Retention days (default: shown below) + days: 14 # Optional: Mode for retention. (default: shown below) - # all - save all recording segments for events regardless of activity - # motion - save all recordings segments for events with any detected motion - # active_objects - save all recording segments for event with active/moving objects + # all - save all recording segments for alerts regardless of activity + # motion - save all recordings segments for alerts with any detected motion + # active_objects - save all recording segments for alerts with active/moving objects + # + # NOTE: If the retain mode for the camera is more restrictive than the mode configured + # here, the segments will already be gone by the time this mode is applied. + # For example, if the camera retain mode is "motion", the segments without motion are + # never stored, so setting the mode to "all" here won't bring them back. + mode: motion + # Optional: detection recording settings + detections: + # Optional: Number of seconds before the detection to include (default: shown below) + pre_capture: 5 + # Optional: Number of seconds after the detection to include (default: shown below) + post_capture: 5 + # Optional: Retention settings for recordings of detections + retain: + # Required: Retention days (default: shown below) + days: 14 + # Optional: Mode for retention. (default: shown below) + # all - save all recording segments for detections regardless of activity + # motion - save all recordings segments for detections with any detected motion + # active_objects - save all recording segments for detections with active/moving objects # # NOTE: If the retain mode for the camera is more restrictive than the mode configured # here, the segments will already be gone by the time this mode is applied. # For example, if the camera retain mode is "motion", the segments without motion are # never stored, so setting the mode to "all" here won't bring them back. mode: motion - # Optional: Per object retention days - objects: - person: 15 # Optional: Configuration for the jpg snapshots written to the clips directory for each event # NOTE: Can be overridden at the camera level diff --git a/frigate/config.py b/frigate/config.py index a430b3d6c5..20433e6443 100644 --- a/frigate/config.py +++ b/frigate/config.py @@ -296,12 +296,9 @@ class RetainModeEnum(str, Enum): active_objects = "active_objects" -class RetainConfig(FrigateBaseModel): - default: float = Field(default=10, title="Default retention period.") - mode: RetainModeEnum = Field(default=RetainModeEnum.motion, title="Retain mode.") - objects: Dict[str, float] = Field( - default_factory=dict, title="Object retention period." - ) +class RecordRetainConfig(FrigateBaseModel): + days: float = Field(default=0, title="Default retention period.") + mode: RetainModeEnum = Field(default=RetainModeEnum.all, title="Retain mode.") class EventsConfig(FrigateBaseModel): @@ -309,20 +306,11 @@ class EventsConfig(FrigateBaseModel): default=5, title="Seconds to retain before event starts.", le=MAX_PRE_CAPTURE ) post_capture: int = Field(default=5, title="Seconds to retain after event ends.") - objects: Optional[List[str]] = Field( - None, - title="List of objects to be detected in order to save the event.", - ) - retain: RetainConfig = Field( - default_factory=RetainConfig, title="Event retention settings." + retain: RecordRetainConfig = Field( + default_factory=RecordRetainConfig, title="Event retention settings." ) -class RecordRetainConfig(FrigateBaseModel): - days: float = Field(default=0, title="Default retention period.") - mode: RetainModeEnum = Field(default=RetainModeEnum.all, title="Retain mode.") - - class RecordExportConfig(FrigateBaseModel): timelapse_args: str = Field( default=DEFAULT_TIME_LAPSE_FFMPEG_ARGS, title="Timelapse Args" @@ -355,8 +343,11 @@ class RecordConfig(FrigateBaseModel): retain: RecordRetainConfig = Field( default_factory=RecordRetainConfig, title="Record retention settings." ) - events: EventsConfig = Field( - default_factory=EventsConfig, title="Event specific settings." + detections: EventsConfig = Field( + default_factory=EventsConfig, title="Detection specific retention settings." + ) + alerts: EventsConfig = Field( + default_factory=EventsConfig, title="Alert specific retention settings." ) export: RecordExportConfig = Field( default_factory=RecordExportConfig, title="Recording Export Config" @@ -924,6 +915,14 @@ def validate_roles(cls, v): return v +class RetainConfig(FrigateBaseModel): + default: float = Field(default=10, title="Default retention period.") + mode: RetainModeEnum = Field(default=RetainModeEnum.motion, title="Retain mode.") + objects: Dict[str, float] = Field( + default_factory=dict, title="Object retention period." + ) + + class SnapshotsConfig(FrigateBaseModel): enabled: bool = Field(default=False, title="Snapshots enabled.") clean_copy: bool = Field( @@ -1278,10 +1277,19 @@ def verify_recording_retention(camera_config: CameraConfig) -> None: if ( camera_config.record.retain.days != 0 and rank_map[camera_config.record.retain.mode] - > rank_map[camera_config.record.events.retain.mode] + > rank_map[camera_config.record.alerts.retain.mode] + ): + logger.warning( + f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and alert retention is configured for {camera_config.record.alerts.retain.mode}. The more restrictive retention policy will be applied." + ) + + if ( + camera_config.record.retain.days != 0 + and rank_map[camera_config.record.retain.mode] + > rank_map[camera_config.record.detections.retain.mode] ): logger.warning( - f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and event retention is configured for {camera_config.record.events.retain.mode}. The more restrictive retention policy will be applied." + f"{camera_config.name}: Recording retention is configured for {camera_config.record.retain.mode} and detection retention is configured for {camera_config.record.detections.retain.mode}. The more restrictive retention policy will be applied." ) @@ -1429,7 +1437,7 @@ class FrigateConfig(FrigateBaseModel): default_factory=TimestampStyleConfig, title="Global timestamp style configuration.", ) - version: Optional[float] = Field(default=None, title="Current config version.") + version: Optional[str] = Field(default=None, title="Current config version.") def runtime_config(self, plus_api: PlusApi = None) -> FrigateConfig: """Merge camera config with globals.""" diff --git a/frigate/events/cleanup.py b/frigate/events/cleanup.py index 393aeea0b6..e4c571be7a 100644 --- a/frigate/events/cleanup.py +++ b/frigate/events/cleanup.py @@ -68,7 +68,10 @@ def get_camera_labels(self, camera: str) -> list[Event]: def expire(self, media_type: EventCleanupType) -> list[str]: ## Expire events from unlisted cameras based on the global config if media_type == EventCleanupType.clips: - retain_config = self.config.record.events.retain + expire_days = max( + self.config.record.alerts.retain.days, + self.config.record.detections.retain.days, + ) file_extension = None # mp4 clips are no longer stored in /clips update_params = {"has_clip": False} else: @@ -82,7 +85,11 @@ def expire(self, media_type: EventCleanupType) -> list[str]: # loop over object types in db for event in distinct_labels: # get expiration time for this label - expire_days = retain_config.objects.get(event.label, retain_config.default) + if media_type == EventCleanupType.snapshots: + expire_days = retain_config.objects.get( + event.label, retain_config.default + ) + expire_after = ( datetime.datetime.now() - datetime.timedelta(days=expire_days) ).timestamp() @@ -132,7 +139,10 @@ def expire(self, media_type: EventCleanupType) -> list[str]: ## Expire events from cameras based on the camera config for name, camera in self.config.cameras.items(): if media_type == EventCleanupType.clips: - retain_config = camera.record.events.retain + expire_days = max( + camera.record.alerts.retain.days, + camera.record.detections.retain.days, + ) else: retain_config = camera.snapshots.retain @@ -142,9 +152,11 @@ def expire(self, media_type: EventCleanupType) -> list[str]: # loop over object types in db for event in distinct_labels: # get expiration time for this label - expire_days = retain_config.objects.get( - event.label, retain_config.default - ) + if media_type == EventCleanupType.snapshots: + expire_days = retain_config.objects.get( + event.label, retain_config.default + ) + expire_after = ( datetime.datetime.now() - datetime.timedelta(days=expire_days) ).timestamp() diff --git a/frigate/events/maintainer.py b/frigate/events/maintainer.py index e83194ede3..6707bfb41a 100644 --- a/frigate/events/maintainer.py +++ b/frigate/events/maintainer.py @@ -5,7 +5,7 @@ from typing import Dict from frigate.comms.events_updater import EventEndPublisher, EventUpdateSubscriber -from frigate.config import EventsConfig, FrigateConfig +from frigate.config import FrigateConfig from frigate.events.types import EventStateEnum, EventTypeEnum from frigate.models import Event from frigate.util.builtin import to_relative_box @@ -128,16 +128,13 @@ def handle_object_detection( if should_update_db(self.events_in_process[event_data["id"]], event_data): updated_db = True camera_config = self.config.cameras[camera] - event_config: EventsConfig = camera_config.record.events width = camera_config.detect.width height = camera_config.detect.height first_detector = list(self.config.detectors.values())[0] - start_time = event_data["start_time"] - event_config.pre_capture + start_time = event_data["start_time"] end_time = ( - None - if event_data["end_time"] is None - else event_data["end_time"] + event_config.post_capture + None if event_data["end_time"] is None else event_data["end_time"] ) # score of the snapshot score = ( diff --git a/frigate/object_processing.py b/frigate/object_processing.py index dcf6014fcc..d4e86889e1 100644 --- a/frigate/object_processing.py +++ b/frigate/object_processing.py @@ -1070,25 +1070,27 @@ def should_retain_recording(self, camera: str, obj: TrackedObject): if obj.obj_data["position_changes"] == 0: return False - # If there are required zones and there is no overlap + # If the object is not considered an alert or detection review_config = self.config.cameras[camera].review - required_zones = ( - review_config.alerts.required_zones - + review_config.detections.required_zones - ) - if len(required_zones) > 0 and not set(obj.entered_zones) & set(required_zones): - logger.debug( - f"Not creating clip for {obj.obj_data['id']} because it did not enter required zones" + if not ( + ( + obj.obj_data["label"] in review_config.alerts.labels + and ( + not review_config.alerts.required_zones + or set(obj.entered_zones) & set(review_config.alerts.required_zones) + ) + ) + or ( + not review_config.detections.labels + or obj.obj_data["label"] in review_config.detections.labels + ) + and ( + not review_config.detections.required_zones + or set(obj.entered_zones) & set(review_config.alerts.required_zones) ) - return False - - # If the required objects are not present - if ( - record_config.events.objects is not None - and obj.obj_data["label"] not in record_config.events.objects ): logger.debug( - f"Not creating clip for {obj.obj_data['id']} because it did not contain required objects" + f"Not creating clip for {obj.obj_data['id']} because it did not qualify as an alert or detection" ) return False diff --git a/frigate/record/cleanup.py b/frigate/record/cleanup.py index 6f4e3c29fc..9f00fb50e4 100644 --- a/frigate/record/cleanup.py +++ b/frigate/record/cleanup.py @@ -12,7 +12,7 @@ from frigate.config import CameraConfig, FrigateConfig, RetainModeEnum from frigate.const import CACHE_DIR, CLIPS_DIR, MAX_WAL_SIZE, RECORD_DIR -from frigate.models import Event, Previews, Recordings, ReviewSegment +from frigate.models import Previews, Recordings, ReviewSegment from frigate.record.util import remove_empty_directories, sync_recordings from frigate.util.builtin import clear_and_unlink, get_tomorrow_at_time @@ -61,8 +61,42 @@ def truncate_wal(self) -> None: db.execute_sql("PRAGMA wal_checkpoint(TRUNCATE);") db.close() + def expire_review_segments(self, config: CameraConfig, now: datetime) -> None: + """Delete review segments that are expired""" + alert_expire_date = ( + now - datetime.timedelta(days=config.record.alerts.retain.days) + ).timestamp() + detection_expire_date = ( + now - datetime.timedelta(days=config.record.detections.retain.days) + ).timestamp() + expired_reviews: ReviewSegment = ( + ReviewSegment.select(ReviewSegment.id) + .where( + ReviewSegment.camera == config.name + and ( + ( + ReviewSegment.severity == "alert" + and ReviewSegment.end_time < alert_expire_date + ) + or ( + ReviewSegment.severity == "detection" + and ReviewSegment.end_time < detection_expire_date + ) + ) + ) + .namedtuples() + ) + + max_deletes = 100000 + deleted_reviews_list = list(map(lambda x: x[0], expired_reviews)) + logger.info(f"the list is {deleted_reviews_list}") + for i in range(0, len(deleted_reviews_list), max_deletes): + ReviewSegment.delete().where( + ReviewSegment.id << deleted_reviews_list[i : i + max_deletes] + ).execute() + def expire_existing_camera_recordings( - self, expire_date: float, config: CameraConfig, events: Event + self, expire_date: float, config: CameraConfig, reviews: ReviewSegment ) -> None: """Delete recordings for existing camera based on retention config.""" # Get the timestamp for cutoff of retained days @@ -86,47 +120,47 @@ def expire_existing_camera_recordings( .iterator() ) - # loop over recordings and see if they overlap with any non-expired events + # loop over recordings and see if they overlap with any non-expired reviews # TODO: expire segments based on segment stats according to config - event_start = 0 + review_start = 0 deleted_recordings = set() kept_recordings: list[tuple[float, float]] = [] for recording in recordings: keep = False + mode = None # Now look for a reason to keep this recording segment - for idx in range(event_start, len(events)): - event: Event = events[idx] + for idx in range(review_start, len(reviews)): + review: ReviewSegment = reviews[idx] - # if the event starts in the future, stop checking events + # if the review starts in the future, stop checking reviews # and let this recording segment expire - if event.start_time > recording.end_time: + if review.start_time > recording.end_time: keep = False break - # if the event is in progress or ends after the recording starts, keep it - # and stop looking at events - if event.end_time is None or event.end_time >= recording.start_time: + # if the review is in progress or ends after the recording starts, keep it + # and stop looking at reviews + if review.end_time is None or review.end_time >= recording.start_time: keep = True + mode = ( + config.record.alerts.retain.mode + if review.severity == "alert" + else config.record.detections.retain.mode + ) break - # if the event ends before this recording segment starts, skip - # this event and check the next event for an overlap. - # since the events and recordings are sorted, we can skip events + # if the review ends before this recording segment starts, skip + # this review and check the next review for an overlap. + # since the review and recordings are sorted, we can skip review # that end before the previous recording segment started on future segments - if event.end_time < recording.start_time: - event_start = idx + if review.end_time < recording.start_time: + review_start = idx # Delete recordings outside of the retention window or based on the retention mode if ( not keep - or ( - config.record.events.retain.mode == RetainModeEnum.motion - and recording.motion == 0 - ) - or ( - config.record.events.retain.mode == RetainModeEnum.active_objects - and recording.objects == 0 - ) + or (mode == RetainModeEnum.motion and recording.motion == 0) + or (mode == RetainModeEnum.active_objects and recording.objects == 0) ): Path(recording.path).unlink(missing_ok=True) deleted_recordings.add(recording.id) @@ -202,65 +236,6 @@ def expire_existing_camera_recordings( Previews.id << deleted_previews_list[i : i + max_deletes] ).execute() - review_segments: list[ReviewSegment] = ( - ReviewSegment.select( - ReviewSegment.id, - ReviewSegment.start_time, - ReviewSegment.end_time, - ReviewSegment.thumb_path, - ) - .where( - ReviewSegment.camera == config.name, - ReviewSegment.end_time < expire_date, - ) - .order_by(ReviewSegment.start_time) - .namedtuples() - .iterator() - ) - - # expire review segments - recording_start = 0 - deleted_segments = set() - for segment in review_segments: - keep = False - # look for a reason to keep this segment - for idx in range(recording_start, len(kept_recordings)): - start_time, end_time = kept_recordings[idx] - - # if the recording starts in the future, stop checking recordings - # and let this segment expire - if start_time > segment.end_time: - keep = False - break - - # if the recording ends after the segment starts, keep it - # and stop looking at recordings - if end_time >= segment.start_time: - keep = True - break - - # if the recording ends before this segment starts, skip - # this recording and check the next recording for an overlap. - # since the kept recordings and segments are sorted, we can skip recordings - # that end before the current segment started - if end_time < segment.start_time: - recording_start = idx - - # Delete segments without any relevant recordings - if not keep: - Path(segment.thumb_path).unlink(missing_ok=True) - deleted_segments.add(segment.id) - - # expire segments - logger.debug(f"Expiring {len(deleted_segments)} segments") - # delete up to 100,000 at a time - max_deletes = 100000 - deleted_segments_list = list(deleted_segments) - for i in range(0, len(deleted_segments_list), max_deletes): - ReviewSegment.delete().where( - ReviewSegment.id << deleted_segments_list[i : i + max_deletes] - ).execute() - def expire_recordings(self) -> None: """Delete recordings based on retention config.""" logger.debug("Start expire recordings.") @@ -302,30 +277,31 @@ def expire_recordings(self) -> None: logger.debug("Start all cameras.") for camera, config in self.config.cameras.items(): logger.debug(f"Start camera: {camera}.") + now = datetime.datetime.now() + + self.expire_review_segments(config, now) expire_days = config.record.retain.days - expire_date = ( - datetime.datetime.now() - datetime.timedelta(days=expire_days) - ).timestamp() - - # Get all the events to check against - events: Event = ( - Event.select( - Event.start_time, - Event.end_time, + expire_date = (now - datetime.timedelta(days=expire_days)).timestamp() + + # Get all the reviews to check against + reviews: ReviewSegment = ( + ReviewSegment.select( + ReviewSegment.start_time, + ReviewSegment.end_time, + ReviewSegment.severity, ) .where( - Event.camera == camera, - # need to ensure segments for all events starting + ReviewSegment.camera == camera, + # need to ensure segments for all reviews starting # before the expire date are included - Event.start_time < expire_date, - Event.has_clip, + ReviewSegment.start_time < expire_date, ) - .order_by(Event.start_time) + .order_by(ReviewSegment.start_time) .namedtuples() ) - self.expire_existing_camera_recordings(expire_date, config, events) + self.expire_existing_camera_recordings(expire_date, config, reviews) logger.debug(f"End camera: {camera}.") logger.debug("End all cameras.") diff --git a/frigate/record/maintainer.py b/frigate/record/maintainer.py index 2d12e2c320..e066602236 100644 --- a/frigate/record/maintainer.py +++ b/frigate/record/maintainer.py @@ -28,7 +28,7 @@ MAX_SEGMENTS_IN_CACHE, RECORD_DIR, ) -from frigate.models import Event, Recordings +from frigate.models import Recordings, ReviewSegment from frigate.util.services import get_video_properties logger = logging.getLogger(__name__) @@ -159,25 +159,27 @@ async def move_files(self) -> None: ): self.audio_recordings_info[camera].pop(0) - # get all events with the end time after the start of the oldest cache file + # get all reviews with the end time after the start of the oldest cache file # or with end_time None - events: Event = ( - Event.select( - Event.start_time, - Event.end_time, - Event.data, + reviews: ReviewSegment = ( + ReviewSegment.select( + ReviewSegment.start_time, + ReviewSegment.end_time, + ReviewSegment.data, ) .where( - Event.camera == camera, - (Event.end_time == None) - | (Event.end_time >= recordings[0]["start_time"].timestamp()), - Event.has_clip, + ReviewSegment.camera == camera, + (ReviewSegment.end_time == None) + | ( + ReviewSegment.end_time + >= recordings[0]["start_time"].timestamp() + ), ) - .order_by(Event.start_time) + .order_by(ReviewSegment.start_time) ) tasks.extend( - [self.validate_and_move_segment(camera, events, r) for r in recordings] + [self.validate_and_move_segment(camera, reviews, r) for r in recordings] ) recordings_to_insert: list[Optional[Recordings]] = await asyncio.gather(*tasks) @@ -189,10 +191,11 @@ async def move_files(self) -> None: ) async def validate_and_move_segment( - self, camera: str, events: list[Event], recording: dict[str, any] + self, camera: str, reviews: list[ReviewSegment], recording: dict[str, any] ) -> None: cache_path = recording["cache_path"] start_time = recording["start_time"] + record_config = self.config.cameras[camera].record # Just delete files if recordings are turned off if ( @@ -232,10 +235,10 @@ async def validate_and_move_segment( ): # if the cached segment overlaps with the events: overlaps = False - for event in events: + for review in reviews: # if the event starts in the future, stop checking events # and remove this segment - if event.start_time > end_time.timestamp(): + if review.start_time > end_time.timestamp(): overlaps = False Path(cache_path).unlink(missing_ok=True) self.end_time_cache.pop(cache_path, None) @@ -243,12 +246,16 @@ async def validate_and_move_segment( # if the event is in progress or ends after the recording starts, keep it # and stop looking at events - if event.end_time is None or event.end_time >= start_time.timestamp(): + if review.end_time is None or review.end_time >= start_time.timestamp(): overlaps = True break if overlaps: - record_mode = self.config.cameras[camera].record.events.retain.mode + record_mode = ( + record_config.alerts.retain.mode + if review.severity == "alert" + else record_config.detections.retain.mode + ) # move from cache to recordings immediately return await self.move_segment( camera, @@ -257,12 +264,14 @@ async def validate_and_move_segment( duration, cache_path, record_mode, - event.data["type"] == "api", ) # if it doesn't overlap with an event, go ahead and drop the segment # if it ends more than the configured pre_capture for the camera else: - pre_capture = self.config.cameras[camera].record.events.pre_capture + pre_capture = max( + record_config.alerts.pre_capture, + record_config.detections.pre_capture, + ) camera_info = self.object_recordings_info[camera] most_recently_processed_frame_time = ( camera_info[-1][0] if len(camera_info) > 0 else 0 @@ -349,12 +358,11 @@ async def move_segment( duration: float, cache_path: str, store_mode: RetainModeEnum, - manual_event: bool = False, # if this segment is being moved due to a manual event ) -> Optional[Recordings]: segment_info = self.segment_stats(camera, start_time, end_time) # check if the segment shouldn't be stored - if not manual_event and segment_info.should_discard_segment(store_mode): + if segment_info.should_discard_segment(store_mode): Path(cache_path).unlink(missing_ok=True) self.end_time_cache.pop(cache_path, None) return @@ -427,8 +435,7 @@ async def move_segment( Recordings.duration.name: duration, Recordings.motion.name: segment_info.motion_count, # TODO: update this to store list of active objects at some point - Recordings.objects.name: segment_info.active_object_count - + (1 if manual_event else 0), + Recordings.objects.name: segment_info.active_object_count, Recordings.regions.name: segment_info.region_count, Recordings.dBFS.name: segment_info.average_dBFS, Recordings.segment_size.name: segment_size, diff --git a/frigate/record/record.py b/frigate/record/record.py index 98faa390c4..00634f1578 100644 --- a/frigate/record/record.py +++ b/frigate/record/record.py @@ -11,7 +11,7 @@ from setproctitle import setproctitle from frigate.config import FrigateConfig -from frigate.models import Event, Recordings +from frigate.models import Recordings, ReviewSegment from frigate.record.maintainer import RecordingMaintainer from frigate.util.services import listen @@ -41,7 +41,7 @@ def receiveSignal(signalNumber: int, frame: Optional[FrameType]) -> None: }, timeout=max(60, 10 * len([c for c in config.cameras.values() if c.enabled])), ) - models = [Event, Recordings] + models = [ReviewSegment, Recordings] db.bind(models) maintainer = RecordingMaintainer( diff --git a/frigate/test/test_config.py b/frigate/test/test_config.py index f5592be191..c703de8930 100644 --- a/frigate/test/test_config.py +++ b/frigate/test/test_config.py @@ -381,9 +381,7 @@ def test_global_object_mask(self): def test_motion_mask_relative_matches_explicit(self): config = { "mqtt": {"host": "mqtt"}, - "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} - }, + "record": {"alerts": {"retain": {"days": 20}}}, "cameras": { "explicit": { "ffmpeg": { @@ -555,9 +553,7 @@ def test_ffmpeg_params_input(self): def test_inherit_clips_retention(self): config = { "mqtt": {"host": "mqtt"}, - "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} - }, + "record": {"alerts": {"retain": {"days": 20}}}, "cameras": { "back": { "ffmpeg": { @@ -577,15 +573,17 @@ def test_inherit_clips_retention(self): assert config == frigate_config.model_dump(exclude_unset=True) runtime_config = frigate_config.runtime_config() - assert ( - runtime_config.cameras["back"].record.events.retain.objects["person"] == 30 - ) + assert runtime_config.cameras["back"].record.alerts.retain.days == 20 def test_roles_listed_twice_throws_error(self): config = { "mqtt": {"host": "mqtt"}, "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} + "alerts": { + "retain": { + "days": 20, + } + } }, "cameras": { "back": { @@ -609,7 +607,11 @@ def test_zone_matching_camera_name_throws_error(self): config = { "mqtt": {"host": "mqtt"}, "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} + "alerts": { + "retain": { + "days": 20, + } + } }, "cameras": { "back": { @@ -633,7 +635,11 @@ def test_zone_assigns_color_and_contour(self): config = { "mqtt": {"host": "mqtt"}, "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} + "alerts": { + "retain": { + "days": 20, + } + } }, "cameras": { "back": { @@ -664,7 +670,11 @@ def test_zone_relative_matches_explicit(self): config = { "mqtt": {"host": "mqtt"}, "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} + "alerts": { + "retain": { + "days": 20, + } + } }, "cameras": { "back": { @@ -695,37 +705,6 @@ def test_zone_relative_matches_explicit(self): frigate_config.cameras["back"].zones["relative"].contour, ) - def test_clips_should_default_to_global_objects(self): - config = { - "mqtt": {"host": "mqtt"}, - "record": { - "events": {"retain": {"default": 20, "objects": {"person": 30}}} - }, - "objects": {"track": ["person", "dog"]}, - "cameras": { - "back": { - "ffmpeg": { - "inputs": [ - {"path": "rtsp://10.0.0.1:554/video", "roles": ["detect"]} - ] - }, - "detect": { - "height": 1080, - "width": 1920, - "fps": 5, - }, - "record": {"events": {}}, - } - }, - } - frigate_config = FrigateConfig(**config) - assert config == frigate_config.model_dump(exclude_unset=True) - - runtime_config = frigate_config.runtime_config() - back_camera = runtime_config.cameras["back"] - assert back_camera.record.events.objects is None - assert back_camera.record.events.retain.objects["person"] == 30 - def test_role_assigned_but_not_enabled(self): config = { "mqtt": {"host": "mqtt"}, diff --git a/frigate/util/config.py b/frigate/util/config.py index e7744e56d4..b7fbb7a614 100644 --- a/frigate/util/config.py +++ b/frigate/util/config.py @@ -13,7 +13,7 @@ logger = logging.getLogger(__name__) -CURRENT_CONFIG_VERSION = 0.14 +CURRENT_CONFIG_VERSION = "0.15-0" def migrate_frigate_config(config_file: str): @@ -29,7 +29,7 @@ def migrate_frigate_config(config_file: str): with open(config_file, "r") as f: config: dict[str, dict[str, any]] = yaml.load(f) - previous_version = config.get("version", 0.13) + previous_version = str(config.get("version", "0.13")) if previous_version == CURRENT_CONFIG_VERSION: logger.info("frigate config does not need migration...") @@ -38,12 +38,12 @@ def migrate_frigate_config(config_file: str): logger.info("copying config as backup...") shutil.copy(config_file, os.path.join(CONFIG_DIR, "backup_config.yaml")) - if previous_version < 0.14: + if previous_version < "0.14": logger.info(f"Migrating frigate config from {previous_version} to 0.14...") new_config = migrate_014(config) with open(config_file, "w") as f: yaml.dump(new_config, f) - previous_version = 0.14 + previous_version = "0.14" logger.info("Migrating export file names...") for file in os.listdir(EXPORT_DIR): @@ -55,6 +55,13 @@ def migrate_frigate_config(config_file: str): os.path.join(EXPORT_DIR, file), os.path.join(EXPORT_DIR, new_name) ) + if previous_version < "0.15-0": + logger.info(f"Migrating frigate config from {previous_version} to 0.15-0...") + new_config = migrate_015_0(config) + with open(config_file, "w") as f: + yaml.dump(new_config, f) + previous_version = "0.15-0" + logger.info("Finished frigate config migration...") @@ -141,7 +148,99 @@ def migrate_014(config: dict[str, dict[str, any]]) -> dict[str, dict[str, any]]: new_config["cameras"][name] = camera_config - new_config["version"] = 0.14 + new_config["version"] = "0.14" + return new_config + + +def migrate_015_0(config: dict[str, dict[str, any]]) -> dict[str, dict[str, any]]: + """Handle migrating frigate config to 0.15-0""" + new_config = config.copy() + + # migrate record.events to record.alerts and record.detections + global_record_events = config.get("record", {}).get("events") + if global_record_events: + alerts_retention = {"retain": {}} + detections_retention = {"retain": {}} + + if global_record_events.get("pre_capture"): + alerts_retention["pre_capture"] = global_record_events["pre_capture"] + + if global_record_events.get("post_capture"): + alerts_retention["post_capture"] = global_record_events["post_capture"] + + if global_record_events.get("retain", {}).get("default"): + alerts_retention["retain"]["days"] = global_record_events["retain"][ + "default" + ] + + # decide logical detections retention based on current detections config + if not config.get("review", {}).get("alerts", {}).get( + "required_zones" + ) or config.get("review", {}).get("detections"): + if global_record_events.get("pre_capture"): + detections_retention["pre_capture"] = global_record_events[ + "pre_capture" + ] + + if global_record_events.get("post_capture"): + detections_retention["post_capture"] = global_record_events[ + "post_capture" + ] + + if global_record_events.get("retain", {}).get("default"): + detections_retention["retain"]["days"] = global_record_events["retain"][ + "default" + ] + else: + detections_retention["retain"]["days"] = 0 + + new_config["record"]["alerts"] = alerts_retention + new_config["record"]["detections"] = detections_retention + + del new_config["record"]["events"] + + for name, camera in config.get("cameras", {}).items(): + camera_config: dict[str, dict[str, any]] = camera.copy() + + record_events: dict[str, any] = camera_config.get("record", {}).get("events") + + if record_events: + alerts_retention = {"retain": {}} + detections_retention = {"retain": {}} + + if record_events.get("pre_capture"): + alerts_retention["pre_capture"] = record_events["pre_capture"] + + if record_events.get("post_capture"): + alerts_retention["post_capture"] = record_events["post_capture"] + + if record_events.get("retain", {}).get("default"): + alerts_retention["retain"]["days"] = record_events["retain"]["default"] + + # decide logical detections retention based on current detections config + if not camera_config.get("review", {}).get("alerts", {}).get( + "required_zones" + ) or camera_config.get("review", {}).get("detections"): + if record_events.get("pre_capture"): + detections_retention["pre_capture"] = record_events["pre_capture"] + + if record_events.get("post_capture"): + detections_retention["post_capture"] = record_events["post_capture"] + + if record_events.get("retain", {}).get("default"): + detections_retention["retain"]["days"] = record_events["retain"][ + "default" + ] + else: + detections_retention["retain"]["days"] = 0 + + camera_config["record"]["alerts"] = alerts_retention + camera_config["record"]["detections"] = detections_retention + del camera_config["record"]["events"] + + new_config["cameras"][name] = camera_config + + new_config["version"] = "0.15-0" return new_config