Import only events whose UID does not already exist

[brunt82]

Steps to reproduce

1. Import ics.
2. Import same ics again.

Expected behaviour

It will be only imported events, whose UID does not already exist in calender for the user.

Actual behaviour

Every event is imported regardless if it already exist or not. Therefore it exists events with the same UID in the same calender, which is not allowed by CalDAV specification: https://tools.ietf.org/html/rfc4791.

The UID property value of the calendar components contained in a
calendar object resource MUST be unique in the scope of the calendar
collection in which they are stored.

Calendar components in a calendar collection that have different UID
property values MUST be stored in separate calendar object resources.

Calendar components with the same UID property value, in a given
calendar collection, MUST be contained in the same calendar object
resource. This ensures that all components in a recurrence "set" are
contained in the same calendar object resource. It is possible for a
calendar object resource to just contain components that represent
"overridden" instances (ones that modify the behavior of a regular
instance, and thus include a RECURRENCE-ID property) without also
including the "master" recurring component (the one that defines the
recurrence "set" and does not contain any RECURRENCE-ID property).

I already found #53, which was closed earlier this year. However I confirm that it makes no sense to import already existing events. Therefore I would not ask the user about replacing / adding the event. Instead, the item to be imported should be ignored if another one already exists with the same UID in the calendar.

Independent of the CalDAV specification, here is an use case:

User deletes an event and needs to restore a backup. The backup does not contain all events of the current calendar but the event, which was deleted by accident. Therefore I cannot delete the whole calendar (because I would delete events, which are not in the backup), but I also cannot import the ics because of the duplicated events.

[georgehrke]

The UID property value of the calendar components contained in a
calendar object resource MUST be unique in the scope of the calendar
collection in which they are stored.

I would actually consider this a server-side bug in the CalDAV server, it if allows dup UIDs.
do you agree @tcitworld?

User deletes an event and needs to restore a backup. The backup does not contain all events of the current calendar but the event, which was deleted by accident. Therefore I cannot delete the whole calendar (because I would delete events, which are not in the backup), but I also cannot import the ics because of the duplicated events

There will be a calendar / address book trash bin, so users are able to restore them.

[brunt82]

There will be a calendar / address book trash bin, so users are able to restore them.

Ok, fine. I hope, it is a server feature?! So that the user is also able to restore the event, if it was deleted by Thunderbird or Davdroid?

[georgehrke]

Ok, fine. I hope, it is a server feature?! So that the user is also able to restore the event, if it was deleted by Thunderbird or Davdroid?

Yes, obviously :)

[georgehrke]

(see nextcloud/server#1662)

[tcitworld]

I would actually consider this a server-side bug in the CalDAV server, it if allows dup UIDs.

I would first check if such a check is done in other CalDAV servers.

[brunt82]

As I can see sabreDAV currently also allows to upload multiple vcfs with same UID in it. But there is also a similar issue with sabreDAV and updating the UID (/~https://github.com/fruux/sabre-dav/issues/993). Maybe there is a relation?

[georgehrke]

Maybe there is a relation?

Yes, Nextcloud uses Sabre/DAV ;)

I think we should treat this as 2 issues:

1. There is the upstream issue in sabre/dav that it allows to create multiple objects with the same UID

2. Before importing, the calendar should query a list of all UIDs in a calendar, compare UIDs to the ones the user wants to import and treat duplicates with a special routine.

The big question is:
What does this special routine do?

1. Simply not import existing UIDs again?
2. Compare Last-Modified and overwrite if UID is newer?

I would actually vote for 1 for simplicities sake.

[Arno500]

I'm voting for 2. Let me explain my situation :
I am in a french IUT (attached to an university). Unfortunately, the software is badly made in such a way that we can't sync our phones or have something that is refreshed once per day. ADE (the software that my university use) only provides an ICS file, that can be only imported once, because if we do it twice or more the events are duplicated each time. This can be really problematic as our schedule change almost every day, and courses are added like 4 days before it occur.
The second solution is then perfect my my situation where I need to update events a lot of times whitout re-adding them each time

[tcitworld]

the software is badly made

Can confirm this. :-)

ADE (the software that my university use) only provides an ICS file, that can be only imported once, because if we do it twice or more the events are duplicated each time.

Instead, you should add the ICS URL as a subscription. Note that username/password will be required, in the following form : http(s)://user:password@url

[georgehrke]

There is another issue with option 2:

Scenario:

1. You create an event (Last-Modified: 2017-09-01).
2. You export the ics and send it to someone
3. You make changes (Last-Modified: 2017-09-03).
4. The person you sent it to makes changes (Last-Modified: 2017-09-05).
5. The person you sent it to sends you their ics.

If you now import it, your changes will be lost.

[Arno500]

Trying subscriptions ;), I did not know this could be done since I didn't find any documentation... Anyway, tried it, and it solved my problem ;) ! I just put the direct link to the ics (not identification, are they serious ?) and voilà !
But for the project, I really support the second solution, which is far smarter than the first, or even better, add a windows to manage conflicts 👍! Of course no solution is perfect and conflict management is always a big problem in synchronisation...

[georgehrke]

add a windows to manage conflicts

Then we would have to add versioning of calendars, because otherwise we don't know what the first version looked like.

And generally saying "just implement a manage conflict dialog" is much easier than actually implementing it properly ;)

[Arno500]

I agree XD, but when doing things, the best is to do the best when possible ;)
Also for the case you exposed, the two guys should just have the same calendar and sync them, and if this is just an event, they should not have to add a HUUUUUGE quantity of informations to it (who does, and if they do, why do they use events ?)
Oh and I encountered another problem : subscriptions are not exposed to CalDAV, thus I can't sync the calendar to my Android phone :(

[tcitworld]

Oh and I encountered another problem : subscriptions are not exposed to CalDAV, thus I can't sync the calendar to my Android phone :(

You can use https://icsdroid.bitfire.at/ for CalDAV subscriptions.

[Arno500]

Unfortunately, ICSDroid isn't free, so I can't share it to anyone... legally.
In the end, waiting for a real Nextcloud server update, I created a calendar that I sync with Khal and vdirsyncer on Linux, and it support ICS importing without multiplying events. It looks like it's working and I should be able to manage it quite easily with a script of my own.
Thanks for the help anyway 👍 !
(I know it wasn't a tech support zone, sorry for that)

[georgehrke]

Unfortunately, ICSDroid isn't free, so I can't share it to anyone... legally.

Despite being paid on various android stores, its open source and available via fDroid.

[Arno500]

Oops, my bad ! They don't advertise that ;)
Anyway, I don't see asking people downloading APK on internet, for F-Droid or directly the app. Thanks for the advice though !

[brunt82]

Today I found another (important) use case for this. We want to migrate from another groupware solution to Nextcloud: The migration can be the easiest way for users to export and import their calendars themselves.

But as soon as two users are in the same meeting, one of the both users will get the appointment twice:
There is an appointment with user A and B: When user A imports his calendar, the appointment appears in user B's calendar. When user B imports his calendar, the appointment with user A will be imported a second time although it is the same UID.

[brunt82]

It also depends on the order if organizer or participant imports his calendar first. Reproduce above described behavior following:

1. User A invites user B
2. User B accepts invitation
3. User A exports current calendar and imports it to NC
   -> event appears in both calendars
4. User B exports current calendar and imports it to NC
   -> event appears a second time in user B's calendar

When user B imports his calendar before user A, it will not created a second event neither in user A nor in user B calendar).

[georgehrke]

Pull request in #667