[Feature] Ability to add non detected faces

[rosaLux161]

Ability to add non detected faces

It should be possible to specify manually at a defined point in the image that there is a face there.

I have already reduced the MIN DETECTION SCORE to 0.3, but I still can't detected all faces. More false positives now, of course.

I would like to set the MIN DETECTION SCORE higher, and the function to manually add faces to photos that were not detected by the model would be helpful.

Probably not so easy to implement, but also interesting: training the model that the currently selected image section is a face and that it can be detected better the next time.

Platform

• Server
• Web
• Mobile

[mertalev]

It's more likely that the face you're trying to see is just not being recognized as opposed to not being detected. If there are fewer than 3 faces of the person, those faces won't be grouped. This is controlled by the Min Recognized Faces setting.

[rosaLux161]

I already set the distance to 1.

[chodthewacko]

This would be very useful because:

1. Sometimes face recognition just skips pictures or skips certain faces in pictures.
   There's no way to do a force refresh of one picture, and redoing all pictures is not a valid option.
   Being able to just manually tag a photo is kludgy but a semi decent workaround

2. Sometimes you want to tag someone whose face isn't visible, but you recognize them due to their clothes in other pictures.

[LackesLab]

Just for background information: Facial recognition is not that trivial. A Face Detection Model finds a face and provides a specific BoundingBox and key points (e.g. like ears, nose, cheeks etcl). These keypoints are used to align the facial crop. This step is essential for a reliable facial recognition routine. It is not simple to manually provide these key points. And without key points, faces cannot be aligned.

[mertalev]

This is right. I tested giving the recognition model the same faces both aligned using landmarks and as a simple bounding box crop. The cosine distance between the two is in the range of 0.3 - 0.94: different enough that they're different people as far as the model is concerned. This was with the same face in the same image.

This means whatever bounding box is provided needs to go through landmark detection and alignment to be useful.

[chodthewacko]

Let's not overthink/overengineer this:
All I need is a way to tag a photo (that may have zero faces in it right now) and say, 'person X is in this picture'.

For a picture, you already have: 'replace person A with person B'
This is basically the same thing, but instead of:
Select Face A.
Edit face -> Select Face B.
Delete Face A from face list.
Add face B to face list with Face A's coordinates.
you just do do:
Add -> select Face B
Add face B to face list with coordinates of <0,0>

In both cases, No, face B isn't really at those coordinates.
But I don't care. All I care about is

1. When I view this picture later, I see that 'peter' is in this list.
2. when I later view 'pictures of peter' this picture is in that list.

I do NOT need this face cleverly fed back to facial recognition to make the facial recognition smarter.
Hell, it might not even be a face.
It might be the back of someone's head, and I recognize them by their clothes.
It might be some random blob of color that I'm tagging because my 2 year old said it's a picture of daddy.

[WorksOnMyBox]

This is it! I don't care about identifying the face. I have a lot of racing, scuba diving, and firefighting photos. No facial recognition is going to detect the person in full gear (and obviously not a car) but I still want to be able to tag the photo to those people. Or somethings like the ring hands in a wedding photo.

[alextran1502]

Implement in #16062

[RazielKanos]

From my perspective, this feature is basically useless, as long as I can't edit stuff by hand. There is no real need to add a landmark to the picture. If you really need those coordinates, I guess there are codes for that, since there are a ton of WordPress plugins out there that handle exactly that.

[leocb]

I second this. I imagine this being Like the tag functionality on some Social platforms, you click were you want to tag and specify the name, done! No need to run AI anything.
While at it, allow some way to remove a face tag from a photo, because false positives. I have way too many walls detected as faces in my library.

[RedCommander735]

This would be a wonderful approach for this feature, but might as well store two corner points so that users can say "+ add face/person" and then drag a bounding box around it, (like e.g. the rectangle tool in paint or something like that) so that the visualization can at least match the AI detected faces (maybe do a dashed or dotted line to somewhat separate them).

[stephen304]

I like this approach, it's most flexible for tagging things that aren't faces or can't be recognized by machine learning. This way you have 2 types of people tags, a basic bounding box (manual tag) or an AI tag that is recognized by immich and has facial features & is incorporated into matching additional faces - which I think covers every use-case between the 2.

Another benefit is that there are some additional cool things that could be added in the future:

• If the ML detects a face and landmarks but doesn't consider it a match to any face and you go to manually tag, it could show you any detected but not matched bounding boxes to allow you to tag more quickly (IIRC facebook does/did this). That way you can create a higher quality tag instead of a simple bounding box - and this should be equivalent to finding the unmatched face in the People tab and assigning it to a person (currently you can't assign a detected but unmatched face to a person through edit people / edit faces when viewing the specific picture so this would be one way of doing that, in addition to hopefully being able to assign the face to a person in the right panel of a photo's details).
• If ML is updated in the future and starts detecting faces that weren't detectable before, a maintenance tool could be added (similar to the duplicate checker) to check all manual tags to see if they intersect with AI tags, then you could click through and upgrade any manual tags you want to AI tags so the AI can use face landmarks to improve matching (I think that's how that works). This would replace the manual bounding box with the AI one and the detected face landmarks.

[Joly0]

What about a feature to rescan single images with different settings? I have a few images with faces not being detected although they are clearly visible, but i dont want to rescan all, as i would loose all the faces again with their settings

[Fran314]

+1 for this feature. While machine learning is doing a great job in recognizing faces in my collection, there are some cases where it's just impossible to recognize properly (such as photos where someone is intentionally turned the other way around or wearing a mask or something). I think an option to manually add photos to people (either from the image preview with an "add people" button, or from the person preview with an "add photos" button) would be great

[maglub]

Simplified, there are two steps in the process:

• Face detection
• Face recognition

What I am missing is some sort of debug mode, where you would see that a face has actually been detected on an image. Example:

On this picture there are clearly two faces, me and my brother. My brother is not recognized, which might be ok since the picture is old and he doesn't really look that way anymore. But I just don't know how to figure out if the face at all was detected in the image, and which boundaries it has.

So, to extend this feature request it would be nice to see which parts of the image has been detected as faces.

[ajpanton]

One way to manually add a face would be to manually draw a box - it doesn't have to be perfect, as long as you're happy about it. If you want to be "perfect", make it look like a box that the AI would've drawn. Now, Immich should treat this as a manual face and not even try to identify it. In order to have it identified with a person, the user has to do it manually.

Another way would be to have the user draw a box around a face, or possibly faces. It should be big enough to completely cover any faces you want recognized. Then, Immich would do a new AI search only in that box with highly sensitive settings, that normally would result in lots of false positives, but obviously the user really wants to find a face here, so let's make sure we find a face here. (edit: of course show the results first and ask the user if they are happy, before proceeding with saving the results in the database)

[AdamT20054]

Bumping this thread. Some models - no matter the settings - refuse to even aknowledge certain faces, would love a way to manually tag a face and then assign it a person (possibly feed back into the ml loop?)

[duncaninnes]

I have this issue too. One of my daughters is recognised WAY less often than her sisters. And there's no great difference in the quality of the pictures they are in. Manually tagging her is nearly my only options. But would also be great if this could feed back into the ML loop to say "I have positively identified this area of the picture as the face of person X, please take note" 😀

But also important to be able to tag without having any ML feedback. If I tag the back of my head because I'm in the photo, I don't want ML to start recognising the back of my head 😬

[dedelastuce]

Bumping also this functionnality
I have many pictures were my children are not facing me and are not recognized as person. I would love the functionnality where I can tag them as Persons inside Immich

[bo0tzz]

Bumping a feature request doesn't do anything to make it happen faster. If you don't have something substantial to add, please just upvote the original post.

That said, #6455 just got merged which lays some groundwork for this.

[petersuba]

I was just thinking about that PR (see my comment below) and would think that there are now multiple "pipelines" with regards to associating people with assets. Manual "tagging" would fit into this overall architecture of another source for people and/or face data

[petersuba]

Just to add another usecase - videos! At the moment I think ML only looks a the initial scene but there may well be people in the video not in the first frame. I would like to be able to see all assets related to a person, including videos that that person is appearing, and at the moment this is far from reliable when it comes to videos. The simplest way to make sure they do appear in the person's assets is to be able to manually associate the person with an asset when it is not detected.

[dfyx]

This is literally the one feature that keeps immich from being the perfect fit for me. I have a lot of photos where people look away from the camera but I still want to have them tagged.

I don't need landmark detection or automatic clustering of manually added faces, just a way to draw a bounding box and manually assign a person.

[TeChn4K]

Simply tagging an image and linking that tag with a recognized person, without involving facial recognition in the process, would cover these issues/gaps: no face detection (person from behind, pets, draws), recognition failure (children, disguises, make-up)

And a final suggestion: manually select an image to be analyzed with more permissive recognition parameters, to give the algorithm a better chance of identifying a person.

[spark108]

This request has been hanging since February 24, why hasn't it been thought through yet?

I read a few comments and came to the conclusion that you are trying to tie yourself to a training model, but why? What's the use of it? Let it continue to determine faces as it did before, why change it?

Just make it possible to specify faces manually and it is not necessary for such faces to participate in the automatic face detection model. Why should they participate? Such manually selected faces will only interfere with the model, because faces can be completely closed, but the person who selected this face as a person knows what kind of person it is.

Manual addition will not bring additional bonuses to the analysis of faces, but will simply help to more or less structure the persons.

[spark108]

I absolutely agree with you, I thought about it too, which is what I wrote in the post.

But for example, there are faces that are simply difficult to recognize.

I would simply do it this way: add the ability to manually add faces, and the model would already take these manual additions into account as material that simply needs to be checked more thoroughly, and if even a thorough check did not help, then the model would simply ignore these faces, and if a thorough analysis did reveal something, then with some degree of probability it would take them into account for future analyses.

But still, even so, in places it seems that the model can behave in an unconventional way. In this case, such an approach can simply be introduced as an experimental one with the ability to turn it on/off.

P.S. However, even now the model sometimes makes mistakes, but still somehow identifies, for example, blurred faces, or faces whose quality is catastrophically equal to zero.

[c4sti3l]

I believe what we are looking for is simply the ability to add tags. This way, when searching/filtering for a person, additional images can be manually assigned to that person and appear in the results. The model shouldn't change or learn anything. Just a straightforward solution where, for instance, searching for "John Doe" also shows an image that was manually linked to that person.

[petersuba]

Kinda like tags, but the given the functionality of tags at the moment, the behaviour would not be the same. So let's say I have person A with face detection in 200 photos, and I have an additional 50 that I can "tag" that person in. If I use the current tag functionality, those photos would not show up under the same person (i.e. I would still only have 200 photos showing up, not 250 which is the goal). So in terms of how I manually can add/remove the association of the photo with the person, it should behave similar to how I can add/remove tags to any photo, BUT it should instruct the system to display/filter etc. these photos as belonging to the same person as the ones where the association is made through face detection.

[spark108]

That's right, that's what I want to start with. And what was said earlier is just a bonus offer.

[duncaninnes]

I feel an 'optimal' solution would be to allow tagging of an area of the image to associate with a person. But the tag itself would need some metadata to indicate that it is a manual add rather than a ML tag. This allows for the Person database to grow with how the user wants it to - and for it to be clear which tags are manual and which are ML. Perhaps even a small icon in the corner of the recognised match in the People section of the photo? Something to make it clear what type of tag was present in the image.

There should be no automatic loopback through the ML to say "please try and recognise this face in future" because, as has been noted previously, masks, snoods, makeup, disguises, people turning away, etc. But there should (perhaps as a future request) be the ability for the user to then take a manual tag and ask the ML to reanalyse based on the area provided. Only as a user option though.

This then allows users to tag people in all their pictures without bothering the ML. But if they feel the ML could do with a bit of a nudge for a particular face, they can select specific manual tags to be reanalysed. If ML is then happy about the match, the manual tag could be replaced with the ML tag. Or the user might get the option of replacing the manual tag with the ML tag.

Next part is somewhat offtopic, but I also like the idea of giving more weight to a facial match which has been approved by the user. In my mind there is a difference between the ML indicating that area X is my face, and me agreeing with the ML and saying "yes, that is my face". I don't know enough about the ML algorithms for facial matching, but is there any way to give such information as feedback to the ML?

[daltimus11]

I would like to have the ability to manually add faces too. And also; add people, remove faces, & remove people (not hide them). It doesn't need to then be implemented by the machine learning, I just want to be able to customize my library to be what I want it to be instead of being restricted to what the machine learning comes up with.

This feature would also help resolve the lack of pet detection #7151 . Being able to manually add them and tag them would be better than not being able to have them listed as "people" at all.

[mv-l]

+1 for this feature

[alextran1502]

Implemented in #16062, will be available in the release 1.127.0

[tobse91]

Are manually tagged faces included into the self learning process for people/face recognition?

[dfyx]

The implementation in 1.127.0 is a good start but it could be improved:

• Have it respect dark theme
• Add a filter function to the popup. I have hundreds of people in my library. Finding someone specific, especially in a list that isn't sorted by name is almost impossible.

[dfyx]

Maybe for consistency, we could copy (edit: or reuse) the UI from the "edit faces" panel which already has a search function?

(For the screenshot, I've scrolled to some barely recognizable background faces. If I had picked people with names assigned, those would be displayed below the face)

[dfyx]

Another idea: add an "apply to whole stack" checkbox. I often have raw and JPEG versions of the same photos stacked together and it would be cool if I only had to add a face once.

[mertalev]

Assets in a stack don't necessarily have the same content, resolution, orientation, etc. This feature could be helpful in certain cases, but it could also give a poor impression if it just blindly adds bounding boxes to images that shouldn't have them.

[dfyx]

That's why I proposed a checkbox. It can be unchecked by default, maybe have a tooltip that explains when to check it and when not to check it. That way, all use cases can be handled.

Right now, I'm working on adding a command to my ImmichTools that syncs faces between photos as a workaround.

[lukaslindnermusic]

Awesome! Thanks for adding this! <3

Currently, it is not (yet) possible to also manually tag a person in a video.
Is this planned and possible?
(If not, maybe we could hide the "+" in this case, signaling to the user that this feature is not available for videos.)