[Bug]: references/resolved endoint runs into memory exhausted when targeted at big files

[blizzz]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When a link against, for example, a release tar ball is posted in talk, the references endpoint will run into a memory exhaustion.

{
  "reqId": "7C6OXv2OPTNhSRcF0KpJ",
  "level": 3,
  "time": "2022-10-21T07:40:06+00:00",
  "remoteAddr": "1.2.3.4",
  "user": "foobar",
  "app": "PHP",
  "method": "GET",
  "url": "/ocs/v2.php/references/resolve?reference=%20https%3A%2F%2Fdownload.nextcloud.com%2Fserver%2Freleases%2Flatest-25.zip",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 178257949 bytes) at /var/www/cloud.nextcloud.com/nextcloud/3rdparty/symfony/dom-crawler/Crawler.php#1189",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
  "version": "25.0.0.18",
  "data": {
    "app": "PHP"
  }
}

Steps to reproduce

1. Post a link to https://download.nextcloud.com/server/releases/latest.tar.bz2
2. Check the log file

Expected behavior

The endpoint will evaluate the content size first, if possible, but certainly only read up to a certain byte size.

Installation method

No response

Operating system

No response

PHP engine version

No response

Web server

No response

Database engine version

No response

Is this bug present after an update or on a fresh install?

No response

Are you using the Nextcloud Server Encryption module?

No response

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - announcementcenter: 6.4.0
  - bruteforcesettings: 2.5.0
  - calendar: 4.1.0-rc.1
  - calendar_resource_management: 0.3.1
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - collectives: 2.0.0-beta3
  - comments: 1.15.0
  - contacts: 5.0.1
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - deck: 1.8.0
  - droneci_fast_lane: 1.0.2
  - end_to_end_encryption: 1.11.0-beta.1
  - external: 5.0.0
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_lock: 24.0.1
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - forms: 3.0.0
  - groupfolders: 13.0.0
  - guests: 2.3.0
  - integration_discourse: 1.0.4
  - integration_github: 1.0.12
  - integration_mastodon: 1.0.3
  - integration_reddit: 1.0.5
  - integration_twitter: 1.0.3
  - integration_zammad: 2.0.1
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - mail: 2.1.0-rc.1
  - nextcloud_announcements: 1.14.0
  - notes: 4.6.0
  - notifications: 2.13.1
  - notify_push: 0.5.0
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.0
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - related_resources: 1.0.1
  - richdocuments: 7.0.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - spreed: 15.0.0
  - support: 1.8.0
  - survey_client: 1.13.0
  - suspicious_login: 4.3.0
  - systemtags: 1.15.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.0
  - twofactor_backupcodes: 1.14.0
  - twofactor_nextcloud_notification: 3.5.0-beta.1
  - twofactor_totp: 7.0.0
  - twofactor_webauthn: 1.0.0
  - updatenotification: 1.15.0
  - user_ldap: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
  - zammad_organisation_management: 1.1.3

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

[blizzz]

cc @juliushaertl

[nickvergessen]

Additionally the timeout at

server/lib/private/Collaboration/Reference/LinkReferenceProvider.php

Line 108 in ac7e3ef

$response = $client->get($reference->getId(), [ 'timeout' => 10 ]);

is critical. there is a second one more down, so a user could keep a request busy for ~20sec. if he makes a target server that responds in 9.9sec

[juliushaertl]

@nickvergessen Should we just go with a rather low timeout like 1s then?

[nickvergessen]

Yeah I'd say so

[joshtrichards]

Main issue (memory/content size) was fixed in #36016 & #47627.

Timeouts were left alone. 10s does seem high.