Optimize PROPFIND for non-shared/shared by reducing the number of que… #27284

mrow4a · 2017-03-01T12:32:45Z

Hello guys, after exams I am back at work with a lot of new energy, knowledge and ideas. Lets start the fun with this PR.

The idea here is behavior of SharedPlugin, which I found out reviewing our code with BlackFire e.g:
This is example query if you have 100 NON-SHARED files, it put my attention on quering sharing table while we dont have shares there, and why dont we just 1 query or so..

I have been interested in what causes that amount of share_table accesses and started optimizing the code:

If the same provider, query cumulatively instead of separate queries
Write Unit tests for opt. 1
Dont do query for all the nodes, rather use IN() clause Shares Plugin really slow when listing lots of files in folder with high php-fpm load #25266 (comment)
Write Unit tests for opt. 2
Give to support @felixboehm @cdamken to test with customers

mention-bot · 2017-03-01T12:32:46Z

@mrow4a, thanks for your PR! By analyzing the history of the files in this pull request, we identified @PVince81, @DeepDiver1975 and @phisch to be potential reviewers.

mrow4a · 2017-03-01T12:34:35Z

Results for the PROPFIND query with 1st optimization:

Case 1
-17 queries

Case 2
-300 queries
100 NON-SHARED files, just someone uploaded 100 files to the folder

DeepDiver1975 · 2017-03-01T12:37:29Z

lib/public/Share/IManager.php

+	 * @param Node $path
+	 * @param bool $reshares
+	 * @return IShare[]
+	 * @since 9.0.0


PVince81

Nice step forward, this is going in the right direction I think.

Next step is to take an array of nodes so you can do where file_source in (...) (with array_chunk of 1000 blocks)

PVince81 · 2017-03-01T12:53:58Z

apps/federatedfilesharing/lib/FederatedShareProvider.php

@@ -564,6 +564,16 @@ public function deleteFromSelf(IShare $share, $recipient) {
 	/**
 	 * @inheritdoc
 	 */
+	public function getAllSharesBy($userId, $shareTypes, $node, $reshares) {


let's directly add $limit and $offset to avoid having to change the interface again in the future

I think this function is not interested in any offsets, as name states, ALL :>

ALL meaning "the whole result set contains all the shares of that user" but it doesn't mean we shouldn't provide a way to paginate over said result set.

it's the same semantics as if we say "get all users" from an install that has 10000 users, we need a way to paginate as we don't want to return 10000 entries as it would be slow, especially for the UI parts

PVince81 · 2017-03-01T12:56:43Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+		$allShares = $this->shareManager->getAllSharesBy(
+			$this->userId,
+			$requestedShareTypes,
+			$node


next step is taking an array of nodes ?

DeepDiver1975 · 2017-03-01T13:21:56Z

💥

OCA\DAV\Tests\unit\Connector\Sabre\SharesPluginTest::testGetProperties with data set #0 (array())
Invalid argument supplied for foreach()

/var/lib/jenkins/workspace/owncloud-core_core_PR-27284-5KMQXE2IARTR6EZOXEANYYNTQ5MB25RWDD7UWVSPM56ZJNO46SIA/apps/dav/lib/Connector/Sabre/SharesPlugin.php:128
/var/lib/jenkins/workspace/owncloud-core_core_PR-27284-5KMQXE2IARTR6EZOXEANYYNTQ5MB25RWDD7UWVSPM56ZJNO46SIA/apps/dav/lib/Connector/Sabre/SharesPlugin.php:171
/var/lib/jenkins/workspace/owncloud-core_core_PR-27284-5KMQXE2IARTR6EZOXEANYYNTQ5MB25RWDD7UWVSPM56ZJNO46SIA/lib/composer/sabre/dav/lib/DAV/PropFind.php:98
/var/lib/jenkins/workspace/owncloud-core_core_PR-27284-5KMQXE2IARTR6EZOXEANYYNTQ5MB25RWDD7UWVSPM56ZJNO46SIA/apps/dav/lib/Connector/Sabre/SharesPlugin.php:175
/var/lib/jenkins/workspace/owncloud-core_core_PR-27284-5KMQXE2IARTR6EZOXEANYYNTQ5MB25RWDD7UWVSPM56ZJNO46SIA/apps/dav/tests/unit/Connector/Sabre/SharesPluginTest.php:125

mrow4a · 2017-03-01T13:42:39Z

Thomas, I just been doing prove of concept, as discussed I will now adjust unit tests :> Wanted to see how much we can get

mrow4a · 2017-03-02T10:12:05Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+			foreach ($allShares as $share) {
+				$shareType = $share->getShareType();
+				if (in_array($shareType, $requestedShareTypes)) {
+					$shareTypes[] = $shareType;


@PVince81 we need uniqueness here

Ok, we did with @PVince81 using hashmap and array_keys to do it efficiently will push together with unit tests

DeepDiver1975 · 2017-03-03T16:09:58Z

@SergioBertolinSG @PVince81 do we have behat tests for this? if not we need to add them BEFORE this change - THX

mrow4a · 2017-03-03T16:14:43Z

For already obtained nodes through $folderNode->getDirectoryListing (do not replicate queries which already had to be done for other reasons), query all share-types using IN() clause, injecting nodeID predicates directly into the query. Results for the PROPFIND query with 2nd optimization:

Case 1
10 files, having all possible cases for sharing e.g. (folder share, file shares link, link+.., link+group+2xuser+feder)

reduced by 104 -> 59 queries and -15% shorter execution time

Case 2
100 NON-SHARED files, just someone uploaded 100 files to the folder
reduced from 745 -> 244 queries and -48% shorter execution time

If all checks pass, please switch to review @DeepDiver1975 @PVince81. Please also check with the customer to "tick" the last check from 1st post

mrow4a · 2017-03-03T16:45:49Z

@DeepDiver1975 Yes, please test it, later I will write more unit tests to check more cases e.g IN() chunking .

SergioBertolinSG · 2017-03-06T07:52:22Z

@SergioBertolinSG @PVince81 do we have behat tests for this? if not we need to add them BEFORE this change - THX

What are the approximate steps here?

PVince81 · 2017-03-06T08:48:53Z

Steps:

Login as user user1
Share with user/group/link
PROPFIND with the "oc:share-types" property, verify that the value matches the share types

The share types are unique, so if the user had two outgoing user shares, the type "user" only appears once. This field is used for the web UI to display the correct share status icon in each file row.

PVince81

Great progress!

PVince81 · 2017-03-06T09:57:04Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

@@ -104,34 +104,31 @@ public function initialize(\Sabre\DAV\Server $server) {
 	}

 	/**
-	 * Return a list of share types for outgoing shares
+	 * Update cachedShareTypes for specific nodeIDs


the method is called get not update.

I suggest you keep it to be a get method and transparently handle the cache. The caller doesn't need to know about the caching.

Basically a construct like:

function getSomething($key) { if (!isset($this->cacheSomething[$key])) { $this->cacheSomething[$key] = $this->getTheExpensiveThing($key); } return $this->cacheSomething[$key]; }

This is what we're doing in a lot of the OC code so let's keep this pattern.

PVince81 · 2017-03-06T09:57:37Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+		foreach ($allShares as $share) {
+			$currentNodeID = $share->getNodeId();
+			$currentShareType = $share->getShareType();
+			$this->cachedShareTypes[$currentNodeID][$currentShareType] = true;


Do we really need caching here ? Is this function called repeatedly ?

PVince81 · 2017-03-06T10:09:01Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+
+				// Put node ID into an array and initialize cache for it
+				$nodeId = intval($childNode->getId());
+				array_push($nodeIdsArray, $nodeId);


we usually use $nodeIdsArray[] = $nodeId;. not sure if this makes a difference performance-wise

Should I change it? isnt it equivalent?

PVince81 · 2017-03-06T10:09:01Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+
+				// Put node ID into an array and initialize cache for it
+				$nodeId = intval($childNode->getId());
+				array_push($nodeIdsArray, $nodeId);


we usually use $nodeIdsArray[] = $nodeId;. not sure if this makes a difference performance-wise

Should I change it? isnt it equivalent?

PVince81 · 2017-03-06T10:09:34Z

lib/public/Share/IManager.php

@@ -90,6 +90,18 @@ public function moveShare(IShare $share, $recipientId);
 	 * Get shares shared by (initiated) by the provided user.


Adjust to say Get ALL shares

done https://github.com/owncloud/core/pull/27284/files#diff-89d407e632cb8a1bfe08c6018b8e3455R90

PVince81 · 2017-03-06T10:11:35Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

 			}
+
+			// Cache share-types obtaining them from DB
+			$this->getNodesShareTypes($nodeIdsArray);


A get function that doesn't return anything. this will likely confuse other developers. Please make that function return something.

PVince81 · 2017-03-06T10:12:05Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

+				$nodeId = $this->userFolder->get($sabreNode->getPath())->getId();
+				$this->cachedShareTypes[$nodeId] = [];
+				$this->getNodesShareTypes([$nodeId]);
+				$shareTypesHash = $this->cachedShareTypes[$nodeId];


Don't use the cache directly here and rely on the function doing that for you.

$shareTypesHash = $this->getNodesShareTypes([$nodeId]);

PVince81 · 2017-03-06T10:16:30Z

apps/federatedfilesharing/lib/FederatedShareProvider.php

+			);
+		}
+
+		$qb->andWhere($qb->expr()->in('file_source', $qb->createParameter('file_source_ids')));


We will need chunking here with array_chunk (because some databases have a max limit for their IN operator...). See how it's done here: https://github.com/owncloud/core/blob/v9.1.4/lib/private/Share20/DefaultShareProvider.php#L930

I did chunking in different part of the code, in Manager, should I change it?

What if another part of code written by a future dev calls getAllSharesBy ? It is better if chunking is done inside this to avoid every API consumer to have to think of doing chunking (which is likely to be duplicate code, if not forgotten)

mrow4a · 2017-03-06T10:55:32Z

@SergioBertolinSG I will today give you a full script to test it, initializing users/shares and profiling this etc.

SergioBertolinSG · 2017-03-06T12:33:46Z

@PVince81 there are already tests about this:

https://github.com/owncloud/core/blob/master/tests/integration/features/webdav-related.feature#L301-L372

mrow4a · 2017-03-06T12:40:22Z

Ok Vincent I will add the code style you require and let's change label to review so they can test it

PVince81 · 2017-03-06T13:07:21Z

@SergioBertolinSG thanks for the info, that's perfect.

mrow4a · 2017-03-06T16:14:21Z

@SergioBertolinSG https://github.com/owncloud/automated-tests/pull/47

mrow4a · 2017-03-06T17:29:42Z

@PVince81 please review fixes and decide about changing to Review and get it tested on real sized DB? @SergioBertolinSG @SamuAlfageme

@DeepDiver1975 is everything ok with Jenkins?

PVince81 · 2017-03-06T18:02:11Z

rebase onto master to get rid of that JS failure

PVince81 · 2017-03-07T07:54:52Z

now the test failures are related to your changes

mrow4a · 2017-03-07T08:52:59Z

We should probably also smashbox this PR against all tests, especially share related.

mrow4a · 2017-03-07T10:20:41Z

@DeepDiver1975 @PVince81 Please change flag to review to indicate the PR is ready

mrow4a · 2017-03-07T10:23:28Z

@DeepDiver1975 @PVince81 Ok, flag changed, I just need to fullfil last tick:

integration tests (also smashbox)
benchmark it on bigger instance @felixboehm

DeepDiver1975

I'm not yet sure if all changes to the unit tests are valid .... I need to have a closer look

DeepDiver1975 · 2017-03-07T12:04:21Z

apps/federatedfilesharing/tests/FederatedShareProviderTest.php

@@ -529,9 +529,20 @@ public function testGetAllSharedByWithReshares() {
 			->setNode($node);
 		$this->provider->create($share2);

+		for($i = 0; $i < 200; $i++) {
+			$receiver = strval($i)."[email protected]";


"$[email protected]"

Is it relevant? It is just test.

well - as a student you are allowed to learn even from tests 😉

DeepDiver1975 · 2017-03-07T12:04:39Z

apps/federatedfilesharing/tests/FederatedShareProviderTest.php

@@ -529,9 +529,20 @@ public function testGetAllSharedByWithReshares() {
 			->setNode($node);
 		$this->provider->create($share2);

+		for($i = 0; $i < 200; $i++) {


why do you create 200 shared here?

To test chunking of Node IDs.

DeepDiver1975 · 2017-03-07T12:05:09Z

lib/private/Share20/DefaultShareProvider.php


-		$orX = $qb->expr()->orX();
+		$nodeIdsChunks = array_chunk($nodeIDs, 100);


why 100? should be 1000

I think in other parts of the code we do 100

PVince81 · 2017-03-07T13:20:34Z

lib/private/Share20/DefaultShareProvider.php

+
+		$nodeIdsChunks = array_chunk($nodeIDs, 100);
+		foreach ($nodeIdsChunks as $nodeIdsChunk) {
+			$qb->select('*')


In theory you don't need to rebuild the whole query in this loop, you only need to re-set the value of the IN operator.

You can leave this as is now.

nope, the andWhere calls will add duplicate where conditions to the query builder.

PVince81 · 2017-03-07T13:23:30Z

Code looks good to me now 👍

According to @SergioBertolinSG we already have integration tests for the "oc:share-types" property so since the tests passed I think this is ready to merge.

It seems obvious to me that the performance will be significantly better and we anyway want this in 10.0. Perf testing can be done with tomorrow's daily after this is merged.

To be discussed: backporting.

@DeepDiver1975 any objections ?

mrow4a · 2017-03-07T13:29:16Z

General conclusion here is: if you will have questions from customers -> we have function $folderNode->getDirectoryListing() which basicaly does query to filecache and returns the nodes. For these nodes, we are quering sharing table. I considered Join with filecache here as well, because IN() clause is also kind of it, but JOIN makes no sense here because we would redo twice the work (doing "joining" twice), and $folderNode->getDirectoryListing() is necessary from what I can see. The only disadvantage here is that we have to do it "in chunks", so in case of >100 files in the folder, we need more indexed queries.

PVince81 · 2017-03-07T13:31:10Z

@mrow4a I think getDirectoryListing here is out of scope for this PR. Please look into it in a separate ticket/PR.

DeepDiver1975 · 2017-03-09T10:38:17Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

 	 *
-	 * @param \OCP\Files\Node $node file node
+	 * @param  IShare[] array containing shares


please fix phpdoc

DeepDiver1975 · 2017-03-09T10:38:37Z

apps/dav/lib/Connector/Sabre/SharesPlugin.php

 	 *
-	 * @return int[] array of share types
+	 * @param int[] array of folder/file nodeIDs


DeepDiver1975 · 2017-03-09T10:42:34Z

lib/private/Share20/DefaultShareProvider.php

@@ -462,6 +462,59 @@ public function move(\OCP\Share\IShare $share, $recipient) {
 	/**
 	 * @inheritdoc
 	 */
+	public function getAllSharesBy($userId, $shareTypes, $nodeIDs, $reshares) {


the interface method looks like this

Node $node vs nodeIds - please correct

/** * Get all shares by the given user for specified shareTypes array * * @param string $userId * @param int[] $shareTypes * @param Node|null $node * @param bool $reshares Also get the shares where $user is the owner instead of just the shares where $user is the initiator * @return \OCP\Share\IShare[] * @since 10.0.0 */ public function getAllSharesBy($userId, $shareTypes, $node, $reshares);

PVince81 · 2017-03-10T07:15:42Z

Rebased.

Now Jenkins, I know it's Friday, but still please do your job!

…ries For already obtained nodes through `$folderNode->getDirectoryListing()`, query all share-types using IN() clause, injecting nodeID predicates directly into the query Full coverage with unit tests and fixes

PVince81 · 2017-03-31T09:05:51Z

Backporting would have been nice but there are interface additions which could break.
The only way to backport is to not add the new methods to the interfaces and add a "method_exists" check in the class that would call it and log an error if it doesn't, or fallback to to the old behavior.

@DeepDiver1975 @mrow4a thoughts ?

mrow4a · 2017-03-31T09:52:07Z

I consider backporting dark magic, but if it is possible why not :>

phil-davis · 2019-03-28T01:12:38Z

Note: this code is in stable10 - there is no backport PR. I guess that the "backport" discussion above from 2017 is about backporting to v9.*, and that looks like it did not happen (which is OK).

It looks like there is no "backport to stable10" because the code was already in master at the point where the stable10 branch was created.

mrow4a requested review from PVince81 and DeepDiver1975 March 1, 2017 12:32

DeepDiver1975 added this to the 10.0 milestone Mar 1, 2017

DeepDiver1975 added the 3 - To Review label Mar 1, 2017

DeepDiver1975 requested changes Mar 1, 2017

View reviewed changes

PVince81 reviewed Mar 1, 2017

View reviewed changes

DeepDiver1975 added 2 - Developing and removed 3 - To Review labels Mar 1, 2017

mrow4a force-pushed the optimize_propfind branch from cf89ea6 to c34c554 Compare March 2, 2017 09:46

mrow4a commented Mar 2, 2017

View reviewed changes

mrow4a force-pushed the optimize_propfind branch from c34c554 to 7d100cd Compare March 2, 2017 11:24

PVince81 reviewed Mar 6, 2017

View reviewed changes

mrow4a force-pushed the optimize_propfind branch from 538f78b to 4ce4db2 Compare March 6, 2017 17:28

mrow4a force-pushed the optimize_propfind branch from 4ce4db2 to b0a63a6 Compare March 6, 2017 18:14

mrow4a force-pushed the optimize_propfind branch from 6150e57 to dd620b6 Compare March 7, 2017 07:27

mrow4a force-pushed the optimize_propfind branch from dd620b6 to 331b82e Compare March 7, 2017 08:21

mrow4a added 3 - To Review and removed 2 - Developing labels Mar 7, 2017

DeepDiver1975 requested changes Mar 7, 2017

View reviewed changes

PVince81 reviewed Mar 7, 2017

View reviewed changes

DeepDiver1975 requested changes Mar 9, 2017

View reviewed changes

mrow4a force-pushed the optimize_propfind branch 3 times, most recently from c29cbfc to 274fe9b Compare March 9, 2017 22:20

PVince81 force-pushed the optimize_propfind branch from 274fe9b to ca8e114 Compare March 10, 2017 07:15

Optimize PROPFIND for non-shared/shared by reducing the number of que…

28ff6fe

…ries For already obtained nodes through `$folderNode->getDirectoryListing()`, query all share-types using IN() clause, injecting nodeID predicates directly into the query Full coverage with unit tests and fixes

mrow4a force-pushed the optimize_propfind branch from ca8e114 to 28ff6fe Compare March 10, 2017 08:25

PVince81 merged commit a651f4f into master Mar 10, 2017

PVince81 deleted the optimize_propfind branch March 10, 2017 10:27

PVince81 mentioned this pull request Mar 15, 2017

Shares Plugin really slow when listing lots of files in folder with high php-fpm load #25266

Closed

mrow4a mentioned this pull request Mar 15, 2017

Depth infinity: Make recursive PROPFIND calls cheaper #14531

Closed

ogoffart mentioned this pull request Mar 31, 2017

[Overlay] Update overlay icon in case of files/folders you are sharing owncloud/client#4788

Closed

lock bot locked as resolved and limited conversation to collaborators Mar 27, 2020

		@@ -90,6 +90,18 @@ public function moveShare(IShare $share, $recipientId);
		* Get shares shared by (initiated) by the provided user.


		$orX = $qb->expr()->orX();
		$nodeIdsChunks = array_chunk($nodeIDs, 100);

Optimize PROPFIND for non-shared/shared by reducing the number of que… #27284

Optimize PROPFIND for non-shared/shared by reducing the number of que… #27284

Conversation

mrow4a commented Mar 1, 2017 • edited Loading

mention-bot commented Mar 1, 2017

mrow4a commented Mar 1, 2017 • edited Loading

Choose a reason for hiding this comment

PVince81 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DeepDiver1975 commented Mar 1, 2017

mrow4a commented Mar 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a Mar 2, 2017 • edited Loading

Choose a reason for hiding this comment

DeepDiver1975 commented Mar 3, 2017

mrow4a commented Mar 3, 2017 • edited Loading

mrow4a commented Mar 3, 2017

SergioBertolinSG commented Mar 6, 2017

PVince81 commented Mar 6, 2017

PVince81 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a Mar 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a Mar 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a commented Mar 6, 2017

SergioBertolinSG commented Mar 6, 2017

mrow4a commented Mar 6, 2017

PVince81 commented Mar 6, 2017

mrow4a commented Mar 6, 2017

mrow4a commented Mar 6, 2017 • edited Loading

PVince81 commented Mar 6, 2017

PVince81 commented Mar 7, 2017

mrow4a commented Mar 7, 2017

mrow4a commented Mar 7, 2017

mrow4a commented Mar 7, 2017 • edited Loading

DeepDiver1975 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PVince81 commented Mar 7, 2017

mrow4a commented Mar 7, 2017 • edited Loading

PVince81 commented Mar 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PVince81 commented Mar 10, 2017

PVince81 commented Mar 31, 2017

mrow4a commented Mar 31, 2017 • edited Loading

phil-davis commented Mar 28, 2019

mrow4a commented Mar 1, 2017 •

edited

Loading

mrow4a commented Mar 1, 2017 •

edited

Loading

mrow4a Mar 2, 2017 •

edited

Loading

mrow4a commented Mar 3, 2017 •

edited

Loading

mrow4a Mar 7, 2017 •

edited

Loading

mrow4a Mar 6, 2017 •

edited

Loading

mrow4a commented Mar 6, 2017 •

edited

Loading

mrow4a commented Mar 7, 2017 •

edited

Loading

mrow4a commented Mar 7, 2017 •

edited

Loading

mrow4a commented Mar 31, 2017 •

edited

Loading