ICS import fails - OPTION_FORGIVING should handle underscores

[mainmachine]

Steps to reproduce

1. Import (via file upload or thunderbird import) meeting invite via email client or *.ics file containing underscore(s)
2. Error message: "Imported 0 out of 1, one failure"

Expected behaviour

sabredav OPTION_FORGIVING should handle this, converting underscores into hyphens, etc.

Actual behaviour

Import fails, error message received

Server configuration detail

Operating system: Linux 4.15.0-52-generic nextcloud/calendar#56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64

Webserver: Apache/2.4.38 (Debian) (apache2handler)

Database: mysql 5.7.25

PHP version:

7.3.7
Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, session, posix, Reflection, standard, SimpleXML, pdo_sqlite, Phar, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, exif, gd, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, zip, Zend OPcache

Nextcloud version: 16.0.3 - 16.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: Official Docker image

Signing status

Array
(
)

List of activated apps

Enabled:
 - accessibility: 1.2.0
 - activity: 2.9.1
 - admin_audit: 1.6.0
 - apporder: 0.7.1
 - bruteforcesettings: 1.4.0
 - calendar: 1.7.0
 - cloud_federation_api: 0.2.0
 - comments: 1.6.0
 - contacts: 3.1.3
 - cospend: 0.1.0
 - dav: 1.9.2
 - federatedfilesharing: 1.6.0
 - federation: 1.6.0
 - files: 1.11.0
 - files_pdfviewer: 1.5.0
 - files_rightclick: 0.13.0
 - files_sharing: 1.8.0
 - files_texteditor: 2.8.0
 - files_trashbin: 1.6.0
 - files_versions: 1.9.0
 - files_videoplayer: 1.5.0
 - firstrunwizard: 2.5.0
 - gallery: 18.3.0
 - issuetemplate: 0.5.0
 - logreader: 2.1.0
 - lookup_server_connector: 1.4.0
 - nextcloud_announcements: 1.5.0
 - notifications: 2.4.1
 - oauth2: 1.4.2
 - passman: 2.3.2
 - password_policy: 1.6.0
 - phonetrack: 0.5.2
 - privacy: 1.0.0
 - provisioning_api: 1.6.0
 - recommendations: 0.4.0
 - richdocuments: 3.3.14
 - serverinfo: 1.6.0
 - sharebymail: 1.6.0
 - spreed: 6.0.4
 - support: 1.0.0
 - survey_client: 1.4.0
 - systemtags: 1.6.0
 - tasks: 0.11.0
 - theming: 1.7.0
 - theming_customcss: 1.3.0
 - twofactor_backupcodes: 1.5.0
 - updatenotification: 1.6.0
 - viewer: 1.0.0
 - workflowengine: 1.6.0
Disabled:
 - encryption
 - files_external
 - ocr
 - user_ldap

Configuration (config/config.php)

{
    "memcache.local": "\\OC\\Memcache\\APCu",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "134.209.32.208",
        "files.b-sharpmusic.com",
        "staging.b-sharpmusic.com",
        "172.17.0.4",
        "172.17.0.3",
        "172.17.0.0\/16"
    ],
    "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/files.b-sharpmusic.com",
    "overwritehost": "files.b-sharpmusic.com",
    "overwriteprotocol": "https",
    "dbtype": "mysql",
    "version": "16.0.3.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbtableprefix": "iaq",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "UTC",
    "installed": true,
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "appstore.experimental.enabled": true,
    "maintenance": false,
    "updatechecker": false,
    "theme": "nextcloud-breeze-dark",
    "loglevel": 2,
    "htaccess.RewriteBase": "\/",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "ssl",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mysql.utf8mb4": true,
    "apps_paths": [
        {
            "path": "\/var\/www\/html\/apps",
            "url": "\/apps",
            "writable": false
        },
        {
            "path": "\/var\/www\/html\/custom_apps",
            "url": "\/custom_apps",
            "writable": true
        }
    ],
    "app_install_overwrite": [
        "passman",
        "calendar",
        "spreed",
        "ocr"
    ]
}

Are you using external storage, if yes which one: no

Are you using encryption:

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36

Operating system: Ubuntu 18.04.3 LTS, Cinnamon desktop, thunderbird, firefox, chrome

Logs

Web server error log

Running in Docker - PITA to access and probably not relevant

Nextcloud log

[webdav] Fatal: Sabre\DAV\Exception\UnsupportedMediaType: This resource only supports valid iCalendar 2.0 data. Parse error: Invalid Mimedir file. Line starting at 11 did not follow iCalendar/vCard conventions at <<closure>>

 0. /var/www/html/3rdparty/sabre/dav/lib/CalDAV/Plugin.php line 801
    Sabre\CalDAV\Plugin->validateICalendar("BEGIN:VCALENDAR ... R", "calendars/daver ... s", false, Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {}, true)
 1. <<closure>>
    Sabre\CalDAV\Plugin->beforeCreateFile("calendars/daver ... s", "BEGIN:VCALENDAR ... R", OCA\DAV\CalDAV\Calendar {}, false)
 2. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    undefinedundefinedcall_user_func_array([OCA\DAV\CalDAV\ ... "], ["calendars/dave ... e])
 3. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1094
    Sabre\Event\EventEmitter->emit("beforeCreateFile", ["calendars/dave ... e])
 4. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
    Sabre\DAV\Server->createFile("calendars/daver ... s", "BEGIN:VCALENDAR ... R", null)
 5. <<closure>>
    Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
 6. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
 7. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
 8. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
 9. /var/www/html/apps/dav/lib/Server.php line 316
    Sabre\DAV\Server->exec()
10. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
11. /var/www/html/remote.php line 163
    undefinedundefinedrequire_once("/var/www/html/a ... p")

PUT /remote.php/dav/calendars/daver/martinka/Nextcloud-KVIVPADMS2DH5UI4XTEBJ.ics
from 50.252.194.70 by daver at 2019-08-06T19:49:46+00:00

Browser log

Not relevant, behavior is identical regardless of browser or email client used

Here is a sample ICS file that fails:

bad_data.ics.txt

If you change X-ENTOURAGE_UUID to X-ENTOURAGE-UUID in the file, then everything works as expected. Since OPTION_FORGIVING is supposed to handle this, logically either the option is being ignored or whatever is processing the ICS file is not configured to use this option.

[georgehrke]

@mainmachine Please make sure to report this to the software which generated the faulty ICS.

[georgehrke]

Moving to the server repo since it's related to server-side parsing.

[mainmachine]

@mainmachine Please make sure to report this to the software which generated the faulty ICS.

I would, but I have no idea if the source of the problem is in Outlook for MacOS, the MacOS calendar system, nor do I know how to even begin debugging that as I don't have direct access to a Mac myself. :/

If you're suggesting the root of this problem is invalid iCalendar syntax from Outlook on MacOS, then yes I agree. That said, I think we have to adapt to their bug as I don't expect either MS or Apple care enough to fix it.

[georgehrke]

The icalendar data that you posted contains the following line:

PRODID:Data::ICal 0.22

Seems to be generated by https://metacpan.org/pod/Data::ICal.
But well, their last commit was 4 years ago, so we probably can't expect a fix from their side.
(But I don't even think it's a big in their software, but someone who is using that library is using it wrong, anyway)

[mainmachine]

The only common factors I have is that I get this problem with invites from only 2 associates out of all of those who send me ics invites. These 2 folks are both using Outlook on Mac, so I believe that Outlook is sending those invites (they are attachments to an email, so this is logical).

Interesting, it appears that library was just bumped to 0.23 only two days ago...

Maybe they saw your comment? 😉

[mainmachine]

Ah wait - that file was a mock file, in other words not generated purely by that library. I used it in place of actual ics files I have recieved as those contain sensitive data.

The ics files that cause me trouble have this line:

PRODID:-//Microsoft Corporation//Outlook for Mac MIMEDIR//EN

Sorry for the confusion!

[mainmachine]

I've sanitized an actual problem ics file: bad-from-outlook-for-mac_meeting.ics.txt

Simply changing the underscore for a dash fixes the problem:

- X-ENTOURAGE_UUID:XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
+ X-ENTOURAGE-UUID:XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

[szaimen]

If a file isn't correct, I guess we cannot do much about that. So I would recommend to edit the file in this case with your favorite text editor to become valid -> closing.

[kvikshaug]

If a file isn't correct, I guess we cannot do much about that.

@szaimen, I think it is actually possible to add OPTION_FORGIVING to the Reader on this and this line.

For example change from:

                $vobj = VObject\Reader::readJson($data);

To this:

                $vobj = VObject\Reader::readJson($data, VObject\Reader::OPTION_FORGIVING);

I tried patching my own NextCloud instance manually like this, and the ics file with an invalid X-ENTOURAGE_UUID key was successfully imported. It would be amazing if this option could be exposed as a configuration setting somehow. Or maybe even be set as default, if being forgiving doesn't cause any unwanted behaviour?

This might also solve a subset of causes for #17915.

[szaimen]

@kvikshaug do you mind opening a PR with your patch for discussion? Thank you!

[derSascha]

Same here with files exported by E-Groupware that contains X-EGROUPWARE-NOTIFY_EXTERNALS lines with illegal underscores. After replacing them with X-EGROUPWARE-NOTIFY-EXTERNALS everything works. Would be great if nextcloud might be able to read such broken files more relaxed as a normal users would never understand what to do in this situation.

[szaimen]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

[section1]

Hi guys! This still happens in version 27.1.x, importing an ics with a underscores in property.
I'm researching if i can found the problem but im bad at php.

thanks.