Key not found: path when syncing from shared folders #209

DerArchivar · 2018-10-24T13:54:30Z

First of all, I'd like to thank you so much for taking over this project and fixing and improving it. I really do appreciate your efforts!

I was doing the initial synchronization with your latest version of onedrive when I got this error. It already downloaded most of the files. It could be, that it even already downloaded all the files, I can't tell for sure. It does look complete, though.

Anyway, whenever I do the onedrive --synchronize --verbose, the following error log returns as a result:

(dmd-2.082.1)user@Chaobuntu:~$ onedrive --synchronize --verbose
Loading config ...
Using Config Dir: /home/user/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/username/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 779df50f98b35005
Default Root ID: 779DF50F98B35005!125
Remaining Free Space: 1095793992083
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
std.json.JSONException@std/json.d(508): Key not found: path
----------------
/home/user/dlang/dmd-2.082.1/linux/bin64/../../src/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x917e69d7]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0x9188ab3a]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0x9183e755]
src/sync.d:462 void sync.SyncEngine.applyDifferences(immutable(char)[], const(char)[]) [0x91810e00]
src/sync.d:229 void sync.SyncEngine.applyDifferences() [0x91810133]
src/main.d:443 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0x917f549a]
src/main.d:312 _Dmain [0x917f4d30]

I can provide the following detailed information:

Ubuntu 18.10 x64
dmd-2.082.1
onedrive v2.1.4-4-g79d86e0
Private (Office 365) OneDrive Account
First installation and initialization via onedrive --synchronize --verbose
Deleting items.sqlite3 results in same error

I hope someone can help me find a solution for this issue, as I'm pretty heavily relying on OneDrive. Thank you in advance!

Update:
As it still downloads all files that I newly add to my OneDrive remotely, it appears that it successfully downloaded everything.

See following updated log:

Using Config Dir: /home/user/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/user/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 779df50f98b35005
Default Root ID: 779DF50F98B35005!125
Remaining Free Space: 1095793969794
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
The local item is unsynced, renaming
Downloading file Dokumente/Texte/folder1/folder2/file1.docx ... done.
Downloading file Dokumente/Texte/Übersetzungen/folder3/file2.docx ... done.
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
std.json.JSONException@std/json.d(508): Key not found: path
----------------
/home/user/dlang/dmd-2.082.1/linux/bin64/../../src/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0xd50f49d7]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0xd5198b3a]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0xd514c755]
src/sync.d:462 void sync.SyncEngine.applyDifferences(immutable(char)[], const(char)[]) [0xd511ee00]
src/sync.d:229 void sync.SyncEngine.applyDifferences() [0xd511e133]
src/main.d:443 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0xd510349a]
src/main.d:312 _Dmain [0xd5102d30]

The text was updated successfully, but these errors were encountered:

abraunegg · 2018-10-24T20:21:36Z

~~Did you upgrade from the 'skilion' version? Did you follow the upgrade advice in the readme?~~

If possible, can you re-run the failure using the following command:

onedrive --synchronize --verbose --debug-https

and email this output to support@mynas.com.au for analysis

abraunegg · 2018-10-24T21:42:52Z

It looks as though you are syncing a shared folder / shared path. Can you confirm?

I have attempted to replicate using various shared folders & adding them into my OneDrive path under the root. Can you help / assist with some understanding of your scenario?

abraunegg · 2018-10-24T22:27:33Z

Unable to replicate this using O365 Personal & Ubuntu 18.10

DerArchivar · 2018-10-25T07:51:27Z

Hello and thank you for your quick and informative reply! Sorry that I'm replying that "late", but your responses came in in the middle of the night for me.

So anyway, I am indeed syncing two folders that have been shared with me and that are owned by my wife. I didn't notice that, as I haven't accessed them for quite some time, but they are most probably the culprit, as they are the only ones that I can clearly state have no files in them whatsoever. So onedrive --synchronize downloaded (or created) the folders, but was unable to download the contents (which are only three small files, but that is irrelevant for the topic).

I'll send you the log output via mail right away.

abraunegg · 2018-10-25T08:08:37Z

Sorry that I'm replying that "late", but your responses came in in the middle of the night for me.

The joy's of being a world citizen :)

I'll send you the log output via mail right away.

Received the email - will go through the provided log and see what it provides.

abraunegg · 2018-10-25T09:09:18Z

@DerArchivar
Can you help validate this PR against your problem please:

git clone /~https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/211/head:pr211
git checkout pr211
make
make install

DerArchivar · 2018-10-25T11:58:45Z

Unfortunately, your change does not seem to fix the issue, as it still behaves the same.

I uninstalled the old version (sudo make uninstall).
I deleted the items.sqlite3.
I checked out your PR.
I installed the updated version.
Same error, see log.

Log of onedrive --synchronize --verbose:

Loading config ...
Using Config Dir: /home/user/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/user/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 779df50f98b35005
Default Root ID: 779DF50F98B35005!125
Remaining Free Space: 1095793995404
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Downloading file Dokumente/file.mp3 ... 
Downloading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:13                                                                                                                                       
done.
This item type (OneNote Notebook) is not supported
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Key not found: path
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
std.json.JSONException@std/json.d(508): Key not found: path
----------------
/home/user/dlang/dmd-2.082.1/linux/bin64/../../src/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0xae93d9d7]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0xae9e1b9a]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0xae9957b5]
src/sync.d:483 void sync.SyncEngine.applyDifferences(immutable(char)[], const(char)[]) [0xae967f23]
src/sync.d:234 void sync.SyncEngine.applyDifferences() [0xae96717f]
src/main.d:443 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0xae94c49a]
src/main.d:312 _Dmain [0xae94bd30]

If you again need the --debug-https output, just say so and you'll receive another mail.

abraunegg · 2018-10-25T19:44:20Z

@DerArchivar
Same error - but different location now. Updating the patch.

abraunegg · 2018-10-25T19:46:48Z

@DerArchivar
Can you rebuild your client with the latest changes to PR #211 and report back.

abraunegg · 2018-10-26T20:23:14Z

@DerArchivar
Any update ?

DerArchivar · 2018-10-29T05:47:54Z

@abraunegg I'm terribly sorry for the late reply! I was with my wife at her parents' house over the weekend and didn't have the laptop with me.

Well, it again synced all the changes from OneDrive to the client that I had, but in the end, it failed with an error. But it's a new error, so that's progress!

(dmd-2.082.1)user@Chaobuntu:~/Git/private/onedrive$ onedrive --synchronize --verbose
Loading config ...
Using Config Dir: /home/user/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/user/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 779df50f98b35005
Default Root ID: 779DF50F98B35005!125
Remaining Free Space: 1095320245244
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Applying changes of Path ID: 153367A4CD0D46D8!784
Uploading differences of .
Processing root
The directory has not changed
Processing Folder1
The directory has not changed
Processing Folder2
The directory has not changed
--- long list of unchanged files removed for brevity ---
Processing Folder3
The directory has not changed
core.exception.AssertError@src/sync.d(942): Assertion failure
----------------
??:? _d_assertp [0xcce0105d]
src/sync.d:947 void sync.SyncEngine.uploadRemoteDirDifferences(itemdb.Item, immutable(char)[]) [0xccdf0106]
src/sync.d:897 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0xccdefc8b]
src/sync.d:914 void sync.SyncEngine.uploadDirDifferences(itemdb.Item, immutable(char)[]) [0xccdefe15]
src/sync.d:891 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0xccdefc25]
src/sync.d:848 void sync.SyncEngine.scanForDifferences(immutable(char)[]) [0xccdefa0d]
src/main.d:447 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0xccdd24b3]
src/main.d:312 _Dmain [0xccdd1d30]

Update:
I of course deleted the items.sqlite3 file before trying again.

abraunegg · 2018-10-29T08:38:33Z

OK .. unsure why it is asserting where it is.

I have updated the code to output some more diagnosis details. Can you rebuild the client (again .. sorry) and see what the code outputs? It will still assert - trying to work out why it is doing so.

DerArchivar · 2018-10-29T09:27:58Z

No problem! I'm a developer myself, just haven't read a line of D in my life, so I cannot assist you in debugging, unfortunately. I feel ya. ;) And I'm really grateful for your efforts!

Did a clean rebuild once again and it resulted in (as far as I can see) almost the same result, even though apparently the line numbers of core.exception.AssertError@src/sync.d(946) and sync.SyncEngine.uploadRemoteDirDifferences(itemdb.Item, immutable(char)[]) [0x6f2f3781] calls changed (which is most likely your mentioned debug output):

Loading config ...
Using Config Dir: /home/user/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/user/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: 779df50f98b35005
Default Root ID: 779DF50F98B35005!125
Remaining Free Space: 1095320245244
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Syncing changes from OneDrive ...
Applying changes of Path ID: 779DF50F98B35005!125
This item type (Notebook of User) is not supported
Applying changes of Path ID: 153367A4CD0D46D8!105
Applying changes of Path ID: 153367A4CD0D46D8!784
Uploading differences of .
Processing root
The directory has not changed
Processing Folder1
The directory has not changed
Processing Folder2
The directory has not changed
--- long list of unchanged files removed for brevity ---
Processing Folder3
The directory has not changed
found: false
core.exception.AssertError@src/sync.d(946): Assertion failure
----------------
??:? _d_assertp [0x6f3046dd]
src/sync.d:951 void sync.SyncEngine.uploadRemoteDirDifferences(itemdb.Item, immutable(char)[]) [0x6f2f3781]
src/sync.d:897 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0x6f2f32cf]
src/sync.d:914 void sync.SyncEngine.uploadDirDifferences(itemdb.Item, immutable(char)[]) [0x6f2f3459]
src/sync.d:891 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0x6f2f3269]
src/sync.d:848 void sync.SyncEngine.scanForDifferences(immutable(char)[]) [0x6f2f3051]
src/main.d:447 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0x6f2d5af7]
src/main.d:312 _Dmain [0x6f2d5374]

abraunegg · 2018-10-29T09:34:32Z

found: false

Thanks - so essentially what is happening is the code is checking to see if the 'remote drive id' & 'remote id' exists in the local database - and if false, assert.

Given this branch has fixed up a lot of the database issues, this function should be reworked.

I have - for the moment - commented out the offending assert - can you rebuild and see how this goes? If this solves the issue, will do a complete function clean up.

DerArchivar · 2018-10-29T09:44:00Z

Ah, I see, overlooked that line.

I'm just gonna shorten the verbose output to the most useful part:

...
Processing Folder3
The directory has not changed
Processing 
core.exception.AssertError@src/itemdb.d(283): Assertion failure
----------------
??:? _d_assertp [0x4c4ee035]
src/itemdb.d:286 immutable(char)[] itemdb.ItemDatabase.computePath(const(char)[], const(char)[]) [0x4c4bae97]
src/sync.d:861 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0x4c4dcaea]
src/sync.d:943 void sync.SyncEngine.uploadRemoteDirDifferences(itemdb.Item, immutable(char)[]) [0x4c4dd044]
src/sync.d:897 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0x4c4dcc8b]
src/sync.d:914 void sync.SyncEngine.uploadDirDifferences(itemdb.Item, immutable(char)[]) [0x4c4dce15]
src/sync.d:891 void sync.SyncEngine.uploadDifferences(itemdb.Item) [0x4c4dcc25]
src/sync.d:848 void sync.SyncEngine.scanForDifferences(immutable(char)[]) [0x4c4dca0d]
src/main.d:447 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0x4c4bf4b3]
src/main.d:312 _Dmain [0x4c4bed30]

Seems like that or similar assertions are being done in several locations. But it indeed seems like that is the way to go for resolving this.

I don't know the full extent of the circumstances leading to this error, but just from my opinion:
Instead of cryptically throwing errors like that the hard way, failing that assertion should be "sort of expected" and be handled more smoothly. Don't wanna be criticizing you, it's just that the general existence of that assertion makes me believe that the case of it failing is actually expected. And if it is expected, the application shouldn't crash out like that.

But I'm a (mostly) Java, C# and JavaScript guy, so maybe I'm just not enough into the behavior of D like languages. ;)

abraunegg · 2018-10-29T09:49:21Z

Instead of cryptically throwing errors like that the hard way, failing that assertion should be "sort of expected" and be handled more smoothly.

Agree - the issue here is right now I am more 'maintaining' the original poor code rather than doing a wholesale re-write which is what it really needs - and that is a time thing at the moment.

I have updated the code - see how that goes

DerArchivar · 2018-10-29T09:58:33Z

Would you believe that?

Uploading new items of .
OneDrive Client requested to create remote path: ./test
The requested directory to create was not found on OneDrive - creating remote directory: ./test
Successfully created the remote directory ./test on OneDrive
Uploading file ./test/DOCX 文档.docx ... done.
Remaining free space: 1095320234212
Applying changes of Path ID: 779DF50F98B35005!125
Applying changes of Path ID: 153367A4CD0D46D8!105
Applying changes of Path ID: 153367A4CD0D46D8!784

It is actually working now. Smoothly.

You, good Sir, just made my life a lot easier and your support and incredibly fast responses and solutions are unbelievable.

With the greatest appreciation, I want to thank you and confirm the fix to be working.

Regarding the rewrite:
I see the size of this code and its complexity and can imagine the trouble you're having. I actually worked on a Google Drive (or actually generic) synchronization client in Java for cross platform use to fix this gaping whole in the Linux software world. But the dynamics of all those cloud storage providers made that a pretty damn hopeless effort, which is why I fell back to only trying to make it work for Google Drive and stopped at a most basic implementation that is just barely usable for myself. I quit it quite some time ago, as I switched to OneDrive, but that taught me a lessen about this tasks complexity.

I'm looking forward to this client (or your rewrite) becoming a stable application that can be distributed in binary form. Once that happens, I'll maybe take the time to build a cross platform GUI for it, just to give something back to this project.

Again, thank you so much. You do a great job and it was and is a pleasure to work and be in contact with you! :)

abraunegg · 2018-10-29T10:13:42Z

Glad it is all working now as it should have been.

I actually worked on a Google Drive (or actually generic) synchronization client in Java for cross platform use to fix this gaping whole in the Linux software world.

I looked at the same thing - ended up using the Go version of 'drive' and packaging it up for use under MyNAS Storage Appliance (www.mynas.com.au) which I developed from scratch - lots of learning there as well!

abraunegg · 2018-10-29T19:21:34Z

PR #211 merged

lock · 2019-01-05T21:35:57Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg added Investigating More Information Needed Waiting on Response labels Oct 24, 2018

abraunegg added Bug Something isn't working In Progress Currently being worked on PR Provided - Needs Testing and removed Investigating More Information Needed Waiting on Response labels Oct 25, 2018

abraunegg changed the title ~~Key not found: path~~ Key not found: path when syncing from shared folders Oct 25, 2018

abraunegg added the OneDrive Personal label Oct 25, 2018

abraunegg added the Fixed label Oct 29, 2018

abraunegg removed In Progress Currently being worked on PR Provided - Needs Testing labels Oct 29, 2018

abraunegg closed this as completed Oct 29, 2018

lock bot locked and limited conversation to collaborators Jan 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Key not found: path when syncing from shared folders #209

Key not found: path when syncing from shared folders #209

DerArchivar commented Oct 24, 2018 •

edited

Loading

abraunegg commented Oct 24, 2018 •

edited

Loading

abraunegg commented Oct 24, 2018

abraunegg commented Oct 24, 2018

DerArchivar commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 25, 2018

DerArchivar commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 26, 2018

DerArchivar commented Oct 29, 2018 •

edited

Loading

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018 •

edited

Loading

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018

abraunegg commented Oct 29, 2018

abraunegg commented Oct 29, 2018

lock bot commented Jan 5, 2019

Key not found: path when syncing from shared folders #209

Key not found: path when syncing from shared folders #209

Comments

DerArchivar commented Oct 24, 2018 • edited Loading

abraunegg commented Oct 24, 2018 • edited Loading

abraunegg commented Oct 24, 2018

abraunegg commented Oct 24, 2018

DerArchivar commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 25, 2018

DerArchivar commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 25, 2018

abraunegg commented Oct 26, 2018

DerArchivar commented Oct 29, 2018 • edited Loading

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018 • edited Loading

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018

abraunegg commented Oct 29, 2018

DerArchivar commented Oct 29, 2018

abraunegg commented Oct 29, 2018

abraunegg commented Oct 29, 2018

lock bot commented Jan 5, 2019

DerArchivar commented Oct 24, 2018 •

edited

Loading

abraunegg commented Oct 24, 2018 •

edited

Loading

DerArchivar commented Oct 29, 2018 •

edited

Loading

DerArchivar commented Oct 29, 2018 •

edited

Loading