Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed file IDs in Groupfolders #1468

Open
gzoritchak opened this issue Apr 27, 2021 · 13 comments
Open

Changed file IDs in Groupfolders #1468

gzoritchak opened this issue Apr 27, 2021 · 13 comments
Labels
1. to develop Issues that are ready for development bug high high priority

Comments

@gzoritchak
Copy link

We experienced the same issue mentioned here:

https://help.nextcloud.com/t/changed-file-ids-in-groupfolders/91916

It's very annoying since we used a root spreadsheet to referenced a lot of internal documents. All links are now broken.

@gzoritchak gzoritchak added 0. Needs triage Issues that need to be triaged bug labels Apr 27, 2021
@fschrempf
Copy link
Contributor

@juliushaertl @icewind1991 Are the file ids meant to stay unchanged? If yes, this is a bug. If no there should be something in the docs that informs users about this. Personally I think it is a bug? What's your point of view?

@juliushaertl
Copy link
Member

Yes, file ids should remain unchanged and I haven't encountered such a behavior until now. Please fill out the issue template and it would be especially interesting if there is any groupfolder related errors in the logs. Might also be worth to check if there was any suspicious action like a delete or move during the time this occured in the access logs of the webserver.

@fschrempf fschrempf added the high high priority label May 18, 2021
@solracsf
Copy link
Member

solracsf commented May 25, 2021

Hi @juliushaertl there was nothing in the logs either about group folders or anything else. The problem faced by @gzoritchak has arrived at night, with little or no activity at all.

After investigation, at the SQL dump from 3h26am (nightly backup) & production at 13h43 the same day, we could obseve an ID 2281 and an ID 6208 for the same file, exposing this ID increment, respectivelly (this is an example, all files were affected).

There was nothing unusual in the webserver, database, php or nextcloud logs. Only things related to #1381 and #1472 essentially.

@SimJoSt
Copy link
Contributor

SimJoSt commented Jun 22, 2021

@juliushaertl when we encountered this, there was a bulk file ownership change on the server. The file ownerships were adjusted to match the ones used by the Nextcloud Docker image. Some files were previously owned by the root user and group.
The change was made with chown.

@konfiot
Copy link

konfiot commented Jul 8, 2021

Also experienced the same issue, we did a files:scan for manually imported files, and some file ids have changed in other group folders, maybe this has something to do with the ids changing

@freshdomi
Copy link

freshdomi commented Aug 19, 2021

we have the same problem after moving the data directory....
I have checked that in the table oc_filecache, entries are duplicated. Example below..

Have anyone a solution for this problem?
Our current idea is that we replace the fileids of the new database with the old one... but if that will work...
would be great if someone has another idea that does not attack the DB...

example: (same path but different fileid)

MariaDB [nextcloud]> SELECT * FROM oc_filecache WHERE fileid=3485643;
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
| fileid  | storage | path                                           | path_hash                        | parent  | name      | mimetype | mimepart | size     | mtime      | storage_mtime | encrypted | unen          crypted_size | etag          | permissions | checksum |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
| 3485643 |     280 | __groupfolders/83/2021-09-10_ISAB-02/Upload AM | e9779aced0ca83c359098cc637dfebf1 | 3485639 | Upload AM |        2 |        1 | 61154122 | 1629368109 |    1629367052 |         0 |                          0 | 611e2f2d32386 |          31 |          |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
1 row in set (0.00 sec)

MariaDB [nextcloud]> SELECT * FROM oc_filecache WHERE fileid=1689517;
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
| fileid  | storage | path                                           | path_hash                        | parent  | name      | mimetype | mimepart | size     | mtime      | storage_mtime | encrypted | unen          crypted_size | etag          | permissions | checksum |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
| 1689517 |       1 | __groupfolders/83/2021-09-10_ISAB-02/Upload AM | e9779aced0ca83c359098cc637dfebf1 | 1689457 | Upload AM |        2 |        1 | 48815043 | 1629124416 |    1629124416 |         0 |                          0 | 611a7740865db |          31 |          |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+-----          -------------+---------------+-------------+----------+
1 row in set (0.00 sec)

@juliushaertl
Copy link
Member

When moving the data directory to a different path this is expected as the existing filecache entries are not longer valid for the new location. For a possible workaround for that see https://help.nextcloud.com/t/howto-change-move-data-directory-after-installation/17170 however this is not a supported approach.

Note that execution of the scan command could cause creation of new filecache entries if Nextcloud is not able to map the existing entries to the path.

@freshdomi
Copy link

after this instruction I have actually also proceeded... I have just not updated the oc_storage table... should be no longer necessary from version 13.1 I thought...
but now our problem is that I have moved the directory 3 days ago and the operation continued normally... today we have seen that the shares are no longer there...
Is this now a problem if I adjust the oc_storage afterwards?

@fschrempf fschrempf added 1. to develop Issues that are ready for development and removed 0. Needs triage Issues that need to be triaged labels Nov 4, 2021
@individual-it
Copy link
Contributor

Has anyone ever found the reason for this issue (besides moving the data folder)?

@OlJohnny
Copy link

Still happening with our nextcloud installation on the currently latest version.

@juliushaertl
Copy link
Member

juliushaertl commented Nov 15, 2023

When reading related code I might have a guess where this could potentially go wrong. I prepared a patch that could provide some logging in this case for the Nextcloud server itself.

index 3d5a2f098b2..2dadfe66d50 100644
--- a/lib/private/Files/Storage/Common.php
+++ b/lib/private/Files/Storage/Common.php
@@ -894,6 +894,9 @@ abstract class Common implements Storage, ILockingStorage, IWriteStreamStorage {

        public function getDirectoryContent($directory): \Traversable {
                $dh = $this->opendir($directory);
+               if ($dh === false) {
+                       throw new \Exception('opendir failed, not yielding any results');
+               }
                if (is_resource($dh)) {
                        $basePath = rtrim($directory, '/');
                        while (($file = readdir($dh)) !== false) {

If anyone who encounters this regularly could apply this patch to their instance and share the log results to see if this part is logged with a trace, this would be much appreciated.


More context for further debugging:

When a scan runs for some reason this could lead to no results which then causes the scanner to diff and remove those entries, the next scan probably readds it again with new file ids:

https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L427
https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L494

The same might happen with scanFile where getMetaData/stat call could return false which is not considered an error case, but can lead to a remove from cache.

https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L121

https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L248

@juliushaertl
Copy link
Member

Ok, opendir should still log an error already unless E_WARNING is suppressed via PHP config.

@max-nextcloud
Copy link

I managed to reproduce the new fileids with some fairly simple steps:

  • create a groupfolder, added some files, inspected the database
  • move the __groupfolders folder to __groupfolder
  • run occ groupfolders:scan for the folder in question
  • move the folder back to __groupfolders
  • run the scan again
  • see new file ids, lost tags and shares and so on.

If this behavior persists I suspect that scans with a missing groupfolder are causing this bug.

We can prevent this by checking if the __groupfolders folder is present before running any scan. Otherwise the scan should err out.

We can also check if the folder for the groupfolder ( __groupfolders/123 ) in question is still present before performing a scan. If it's not we should inform the admin rather than dropping all the file cache entries.

So far I reproduced this on Nextcloud 23. I will try current master next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1. to develop Issues that are ready for development bug high high priority
Projects
None yet
Development

No branches or pull requests

10 participants