Reduce scanning of local filesystem needlessly every sync in monitor mode

[abraunegg]

Reduce scanning the entire local system in monitor mode for local changes

After performing all the DB sync fixes, there is still excessive CPU in monitor mode when performing this scan:

...
Uploading new items of .
...

This is due to scanning the entire sync_dir path for any changes.

This scan is excessing and needless, as the sync_dir path, in monitor mode, is already being monitored by inotify for any local changes, thus we should not need to scan the entire path / tree for changes, as they would have already been captured.

Example: a 35 sec 100% CPU load when 'scanning' for local changes with 50K files / folders

With the proposed patch - zero extra CPU load:

Thoughts?

[norbusan]

Hmmm, but if an upload triggered due to inotify doesn't work, eg because offline, when will the files be uploaded?

[abraunegg]

Good point ...

What about some how scanning the local FS once at startup and then if 'detected been offline' ?

[abraunegg]

Or even a full scan every X hours ? scanning EVERY (by default) 45 seconds a full path walk to me is rather excessive ...

[norbusan]

I would go for both: Setting a dirty bit in the inotify handler and doing a full scan if that is set, plus every N (60? 100? Option) sync runs a full sync.

We have already a counter variable for the number of syncs (log frequency) that could be reused.

WDYT?

[abraunegg]

@norbusan
with suggested changes:

(dmd-2.083.0)[alex@centos7full onedrive-pr433]$ ./onedrive --confdir '~/.config/onedrive-personal' --monitor --verbose
Loading config ...
Using Config Dir: /home/alex/.config/onedrive-personal
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/alex/OneDrivePersonal
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 66d53be8a5056eca
Default Root ID: 66D53BE8A5056ECA!101
Remaining Free Space: 4498492245
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing monitor ...
OneDrive monitor interval (seconds): 10
Monitor directory: .
Applying changes of Path ID: 66D53BE8A5056ECA!101
Uploading differences of .
Processing root
The directory has not changed
Processing zero-file-size
The file has not changed
Processing 2nd-zero-file-size
The file has not changed
Uploading new items of .
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Syncing changes from OneDrive ...
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Syncing changes from OneDrive ...
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Uploading differences of .
Processing root
The directory has not changed
Processing zero-file-size
The file has not changed
Processing 2nd-zero-file-size
The file has not changed
Uploading new items of .
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101
Applying changes of Path ID: 66D53BE8A5056ECA!101

• runs every 10 'sync' operations
• defaults to true, so if service is restarted, sync will run

[abraunegg]

@norbusan
Should be good to approve / go if you are OK with the changes

[norbusan]

Fine with me. Strange enough GH says that it cannot rebase the conflicts, while master is already merged in ... it should not rebase, it should simply merge which is actually a fast-forward merge.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.