FileSystemWatcher does not raise events when target directory is symlink (on linux)

[JustArchi]

Steps to reproduce:

mkdir /tmp/realdir
ln -s /tmp/realdir /tmp/symdir

Launch a program similar to below:

var fsw = new FileSystemWatcher("/tmp/symdir") { NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite };
fsw.Created += (o, i) => Console.WriteLine(i.Name);
fsw.EnableRaisingEvents = true;

Create empty file (or any other one):

touch /tmp/symdir/test.txt # Could be also /tmp/realdir/test.txt, since it's one and the same

No event being raised. Event is raised properly if we listen in /tmp/realdir instead.

Since expected behaviour of linux programs is to handle symlinks transparently, I expected that FileSystemWatcher would properly raise events that happened in real directory. That didn't happen though (no events being raised, I also tested with other event types). I couldn't find similar issue so I consider it a bug, but if this is expected then it should probably be noted somewhere.

Thank you in advance for looking into this.

.NET Command Line Tools (2.1.300-preview2-008210)

Product Information:
 Version:            2.1.300-preview2-008210
 Commit SHA-1 hash:  f6065dcc62

Runtime Environment:
 OS Name:     debian
 OS Version:
 OS Platform: Linux
 RID:         debian-x64
 Base Path:   /opt/dotnet/sdk/2.1.300-preview2-008210/

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview2-26131-06
  Build    : b13a0d5c331f374afd35ded57b9a4b4ab128864c

[stefanforsberg]

A kubernetes ConfigMap is using symlinked files and I'm thinking that this issue might be what's causing a reloadOnChange to not trigger when the ConfigMap is changed.

[JustArchi]

If the file/folder being listened is a symlink, you have 100% chance to run into this bug, at least on Linux, very likely also OS X. I didn't test symlinks mechanism on Windows, since it's vastly different.

[stefanforsberg]

We're running our dotnet core application on an alpine docker image so I'd say with high certainty that we run into the bug you describe above. By some magic chance you haven't figured out any workarounds? Long shot I know, but I had to ask =)

[JustArchi]

Sorry for late response @stefanforsberg, I completely missed your message. I don't have a workaround other than using readlink first and targetting the real directory in FileSystemWatcher. This can be done during the runtime, but you'll need /bin/readlink or at least DllImport of libc with ssize_t readlink(const char *pathname, char *buf, size_t bufsiz).

Possible, and the workaround with DllImport isn't even that bad, but it's definitely a lot of code to fix this (and depending on your scenario you might also need to readlink recursively to find the real directory, i.e. what readlink -f does on linux, but for that realpath is probably better).

[wfurt]

It seems like this is intentional:
/~https://github.com/dotnet/runtime/blob/master/src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs#L352-L356

// Add a watch for the full path.  If the path is already being watched, this will return
// the existing descriptor.  This works even in the case of a rename. We also add the DONT_FOLLOW
// and EXCL_UNLINK flags to keep parity with Windows where we don't pickup symlinks or unlinked
// files (which don't exist in Windows)
 int wd = Interop.Sys.INotifyAddWatch(_inotifyHandle, fullPath, (uint)(this._watchFilters | Interop.Sys.NotifyEvents.IN_DONT_FOLLOW | Interop.Sys.NotifyEvents.IN_EXCL_UNLINK));

however maybe the intention was for links within directory. If passed in explicitly we can perhaps check if given name is link and follow linked path instead.

cc: @JeremyKuhne

[carlossanlop]

I started poking into this, and I'm wondering if we could add a new notify event to enable listening to the contents of symlinked directories:

public enum NotifyFilters
{
    FollowSymlinks
}

And to avoid disrupting the world, the code @wfurt pasted above could add the IN_DONT_FOLLOW flag by default, and would remove it if the user passed FollowSymlinks. Something like:

// User did not pass the flag, so we keep adding the IN_DONT_FOLLOW flag as before
if ((filters & NotifyFilters.FollowSymlinks) == 0)
{
    _watchFilters |= Interop.Sys.NotifyEvents.IN_DONT_FOLLOW;
}

int wd = Interop.Sys.INotifyAddWatch(_inotifyHandle, fullPath, (uint)(this._watchFilters | Interop.Sys.NotifyEvents.IN_EXCL_UNLINK));

Thoughts, @wfurt, @JeremyKuhne ?

Edit: I suspect this won't work on Windows. I'll investigate more.

[wfurt]

I think there are two separate issues @carlossanlop. First one is if the directory we are asked to watch is symlink and then what do we do if there are symlinks within watched directory.

For first one I think we can:
1.a close this with "work as expected" and perhaps improve documentation
1.b Do fstat/readlink as @JustArchi suggested. That would add one time cost when creating watcher. If we do this we should do macOS as well.

For second I think it is more tricky. With following links the search may be huge if you ever have something like ../.. in the tree as well as there may be loop. You can try

 Directory.GetFiles("/sys/devices", "foo", SearchOption.AllDirectories));

There is really no way how the caller may know or have influence over.

If we want to proceed, I would open new issue for API proposal. (2a)
We should also validate that that can work on macOS (2b) and Windows (2c - not sore if that is even possible) If this gets through, we would have option to express the desire to treat symlinks transparently and that can be 1.c option to do it only when explicitly asked.

The current behavior seems to mimic old Windows behavior. However I do agree that on Unix symbolic links are generally handled transparently. My personal recommendation would be to try 1b (and perhaps mark it as breaking change) but I would be worried about recursively following symbolic links because of possible DoS and other pitfalls.

[ericstj]

Do you know if clearing IN_DONT_FOLLOW would still follow file symbolic links when watching a directory? The docs indicate that IN_DONT_FOLLOW controls following pathname which could be a directory. It wasn't clear to me if it would just follow the symlink on the directory or it would also follow symlinks on files within the directory. If it didn't do the latter we'd probably need to enumerate the files in the directory to find those that were symlinks and add them as well.

I don't think Windows has this type of functionality, so we might need to enumerate all paths to find the symlinks and add them. We'd have to watch multiple handles for each directory symlink. For file symlinks we'd have to watch the containing directory and make sure to only raise events for the single file.

[wfurt]

following nested links can be dangerous as that can create loops. (something windows do not need to deal with)
My suggestion would be to do it only for the name passed in via

runtime/src/libraries/Common/src/Interop/Unix/System.Native/Interop.ReadLink.cs

Line 32 in a9c5ead

public static string? ReadLink(string path)

Something similar to #48151

[ericstj]

I believe loops can be detected and short-circuited: would have to be if we followed them. Loops can exist on windows too. I agree it makes this more complicated.

If all we do is follow the passed in link, why do that here? Why not instead just allow the user to do that with #24271.

[wfurt]

We could but #24271 sits there for very long time. And pushes the complexity to the caller. We could make small improvement to FileSystemWatcher instead without new API dependency.

[ericstj]

At least in the case I'm looking it I don't believe the root watch is a symlink directory. The symlinks are files in the directory. So following symlinks on the root directory won't help. See #36091 (comment). I'm not sure how many cases will really be helped by only following the root.

IMHO a set of brand new API is easier since it is additive and well defined. I don't see any major blockers to #24271 (WinRT was mentioned but that's no longer a constraint), just some work to prototype more and drive it through API review.

[wfurt]

#36091 seems different. This issue was written for the linked root as far as I can tell.