Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ADR for sync with external databases #9220

Merged
merged 14 commits into from
Apr 15, 2023
Merged

Add ADR for sync with external databases #9220

merged 14 commits into from
Apr 15, 2023

Conversation

tobiasdiez
Copy link
Member

Based on earlier discussion in #7618.

Also closes #7618.

  • Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
  • Tests created for changes (if applicable)
  • Manually tested changed features in running JabRef (always required)
  • Screenshots added in PR description (for UI changes)
  • Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
  • Checked documentation: Is the information available and up to date? If not, I created an issue at /~https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

@tobiasdiez tobiasdiez added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Oct 7, 2022
@koppor
Copy link
Member

koppor commented Oct 7, 2022

Note that the current implementation is based on the Optimisitc Offline Lock, which does not appear in the ADR. (Was also part at the discussion - see #7618 (comment))

@koppor koppor self-requested a review October 7, 2022 18:55
@tobiasdiez
Copy link
Member Author

The proposed design is an optimistic currency control.

Optimistic Offline Lock solves this problem by validating that the changes about to be committed by one session don't conflict with the changes of another session.

That's also so in the proposal. But the main part of the proposal concerns the question "how do you recognize that things changed on the server and on the client"?

@koppor
Copy link
Member

koppor commented Oct 9, 2022

I started to review the ADR - and put it more in the ADR format.

It seems, you describe what we already implemented 5 years ago. We also introduced separate IDs. Will need to dig deeper in this. More input will come the next days. - Thereby, I will also add some results of our discussions of edge cases in the issue.

@tobiasdiez
Copy link
Member Author

Of course there are similarities with the current algorithm. The current version corresponds roughly to what is described in the paragraph "At this point, we could already sync the server and client by asking the server for all up-to-date entries and then using the Revision information to merge with the local data."
For example, at the start of the sync process we currently get the versions of all entries from the server:

public Map<Integer, Integer> getSharedIDVersionMapping() {
Map<Integer, Integer> sharedIDVersionMapping = new HashMap<>();
StringBuilder selectEntryQuery = new StringBuilder()
.append("SELECT * FROM ")
.append(escape("ENTRY"))
.append(" ORDER BY ")
.append(escape("SHARED_ID"));

This works of course but is not efficient.

@koppor
Copy link
Member

koppor commented Oct 18, 2022

IMHO this is not an ADR. The text should go into /~https://github.com/JabRef/jabref/blob/main/docs/code-howtos/remote-storage.md (if we want to have a huge file; otherwise, two files need to be created).

The difference between the current implementation and the described approach is that the user might have a local file. - This is not implemented in the current Postgres/MySQL-sync. The "Shared Database" approach assumes that a) JabRef has the data stored on the remote system, b) there is no connection loss during work, c) the session starts when JabRef is started and the session stops if JabRef is shut down.

@koppor koppor self-assigned this Nov 2, 2022
@koppor
Copy link
Member

koppor commented Mar 27, 2023

Compare with https://automerge.org/blog/automerge-2/

@tobiasdiez
Copy link
Member Author

CRDTs would be another option, the problem is however that you would need to locally store a lot more metadata (e.g. for operational CRDTs you essentially need to have the full history of all edits). So you would need another file next to the bib file to store these. Also CRDTs are mainly used when you need low latency and high frequency of edits (e.g. multi-user chat or text editing). Not really something we care about.

@Siedlerchr Siedlerchr merged commit 977aeec into main Apr 15, 2023
@Siedlerchr Siedlerchr deleted the adr_sync branch April 15, 2023 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: shared-database status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Shared data synchronization
4 participants