Customize scanner for root storage to disable scanning.

[icewind1991]

The scanner for the root folder be customized to not put entries in the cache for the root folder

[icewind1991]

cc @PVince81

[felixboehm]

@MTRichards Please provide a milestone.

[MorrisJobke]

Once a solution for this is available a backport to stable branches is highly appreciated. cc @karlitschek

[karlitschek]

sure. i hope we can come up with a fix that is backportable without a big risk

[icewind1991]

I dont think we can have a nice backportable sollution here, for older versions the blacklist should work for the original issue

[MTRichards]

I get this is a blue ticket, but ... why are we disabling file scan? Is it that it is scanning a lot of folders that should not be scanned in root? How does not scanning root help with 27M rows in the database?

Sorry, don't have any context.

[icewind1991]

One of the problems is non-oc files in the data directory being scanned but the idea behind this issue is broader in scope in how we handle the root storage.

I don't think this issue should be linked to the original blue ticket since it was more raised as a long term improvement on handling the root storage.

[MTRichards]

Ok, we should be limiting our scan to ownCloud files...how do we know if they are ownCloud files?

Don't we assume the ownCloud root data directory is owned by ownCloud? Is there a case where this is an unreasonable assumption? Perhaps in windows home directories and other external storage?

Seems there is if it is the root cause of 27M file cache entries, but why are there non ownCloud files in there?

The long term answer is then something for 9.1 or later. But without a reference ticket or any context I can't judge the importance.

[MorrisJobke]

Don't we assume the ownCloud root data directory is owned by ownCloud? Is there a case where this is an unreasonable assumption? Perhaps in windows home directories and other external storage?

The original case was caused by a folder called .snapshots which was created by the underlying filesystem and can't be removed or have more restrictive permissions. This folder shouldn't be scanned and also shouldn't be a problem for ownCloud. As it is a backup access folder it contains the very same folder structure many times (i.e. once per day in the last week, once per week in the last 8 weeks, once per month in the last year, etc) which causes owncloud to create a ton of file cache entries in the root storage with no real sense.

[karlitschek]

hmmm. there was aseperate issue/pr somewhere to make it possible to ignore specific files and folders that come from the filesystem and don't containand owncloud payload. this sounds like the right approach. but can't find the issue at the moment.

[MorrisJobke]

hmmm. there was aseperate issue/pr somewhere to make it possible to ignore specific files and folders that come from the filesystem and don't containand owncloud payload. this sounds like the right approach. but can't find the issue at the moment.

#19385 that one?

[karlitschek]

yes. and the linked PR

[MTRichards]

Excellent, THAT makes sense and now I understand. Thanks @MorrisJobke @karlitschek .

What would it take to get that in?

[icewind1991]

That's what the blacklist should be used for, it might need to be extended to handle some extra use cases but that seems the correct place to handle that.

Since this issue is about a specific "would be nice to have" improvement to our handling of the root storage and was never intended as direct fix for the blue tick we should remove the blue ticket label and move the discussion about ignoring files to a better place

[MTRichards]

@icewind1991 Is this a filescan blacklist that you are talking about?
Is that in the PR liked above, or something else new and different?

[MTRichards]

@felixboehm can you open a github for the customer issue and link it to this one, get a support case opened, and tag the new github with the blue ticket?

[icewind1991]

The blacklist is currently a config option in config.php which will completely block files matching a specific name.

It's currently used to block .htaccess files from being created for security reasons but can also be used to prevent the scanner from going into folders on the filesystem

[MTRichards]

Ah, ok - I understand and that keeps client side hidden files off the server, in addition to the sync hidden file checkbox on the client. How does that work for a directory like .snapshots that is created on the server in the data directory and filescan?

We would extend that to not scan a blacklist directory?

Ok.

Any sense for the level of effort here?

@karlitschek @DeepDiver1975

[icewind1991]

The blacklist is a hard block on the server side, if you add .snapshots to the blacklist the server will ignore any .snapshots folder it finds on the server and block any attempts by the client/webui to create such a folder

[MorrisJobke]

@felixboehm can you open a github for the customer issue and link it to this one, get a support case opened, and tag the new github with the blue ticket?

Already present: owncloud/enterprise#901

[PVince81]

An alternative fix from @MorrisJobke was provided here recently: #22565

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.