Move files_external backend + config logic to core

[PVince81]

@DeepDiver1975 @Xenopathic @karlitschek @jmaciasportela

Currently external apps that provide storage backends would require files_external to be enabled too.
Goal is to move that part to core so that the files_external app disappears completely.
The other backends currently provided by files_external should be moved to one or several separate repos eventually.

[karlitschek]

I hate it to move more stuff into the core and make everything less modular. But this might be a case where is actually make sense.

[DeepDiver1975]

I see no other possibility if we want to support external storage implementations to live isolated on their own apps.

[DeepDiver1975]

To be clear: no external storage should be moved to core

[RobinMcCorkell]

In addition, the configuration UI might want to stay in files_external? So only the backend stuff will go to core

[PVince81]

@Xenopathic which means that if a mount.json exists it would be read but there would be no UI to configure it then ?

[RobinMcCorkell]

@PVince81 Yes, in that circumstance that is what would happen. But in the majority of cases, mount.json would not get created, since the UI would not be present. It was an idea mentioned by @DeepDiver1975 some time ago...

[PVince81]

We might also want to add some public APIs to make it possible for apps to retrieve the list of external storage backends and also instantiate then. See pbek/nextbackup#3 (comment)

[PVince81]

I'm setting this to 9.1 to look into that.

There are some code paths in core that currently check if the app is enabled which is not nice, and bugs like #21784

[PVince81]

@Xenopathic do you have time for some fun refactoring ? 😄

[PVince81]

Hmmm, now I see that some apps might be using the \OCA\Files_External namespace already to access some of the APIs. So if we move stuff to core we'd need to add placeholders in the app itself to avoid breaking third party external storage apps during the deprecation period.

[PVince81]

CC @jvillafanez in case you want to give some input for the general idea of moving files_external service/backend/config classes to core.

Seems we'll also have to decide which are public and which are private.

So far my idea is to have a new namespace \OC\Files\External\Service in core for the services.

[PVince81]

PR here #25422 (WIP)

[jvillafanez]

Ideally, the files_external app should be part of the files app, and this one should be isolated. However, taking into account the amount of responsabilities absorbed by core regarding the files app, it make sense that some of the responsabilities from the files_external app are also absorbed.

What should we move to core? Being very general: all the interfaces, but none of the implementations.

• Interfaces to connect to the backends are already in core, so no need to change anything there. We might consider to separate each implementation into different apps, but not in the near future anyway. The implementation of each backend should be kept outside of core.
• Interfaces to configure the backends. The implementation of common components (textfields, checkboxes, etc) might also be moved to core, BUT custom components should be able to be created per each backend.
• Interfaces to provide authentication mechanisms. The same as the previous point.
• Additional services available (such as scanner, watcher, cache, etc). The default implementation might be in core, but custom implementations might be provided by others. (I think this needs some kind of rework because those services are very coupled with a DB implementation, and they're making several assumptions which depends on the specific implementation).
• General CLI actions over the storages. This includes configuration, listing, scanning, etc.

There are other things to consider:

• We should consider actions over the storage configuration and / or authentication mechanism (for example, password reset).
• We should also consider communication between storages (to be decided if the communication will be among the storages of the same type, or with any storage)

[PVince81]

@jvillafanez I'd say for now the goal is only to move all the code that is not storage-specific from files_external to core. And only leave the storage backend implementations in "files_external", which might also be moved to separate apps in the future.

As for the "GlobalStorageServices" and "UserGlobalStorageService" these implementations should also be in code, considering that they are exposed through the server container already. This might also imply moving the CLI commands from files_external to core as well to manage configurations.

The end result will be that "files_external" really becomes "here's a bunch of storage backends you might want to use" but is optional, and the external storage stuff and its config can then work as part of core.

[jvillafanez]

@PVince81 I completely agree, but I think it's a good time to look ahead and make plans for what's to come.

As for the "GlobalStorageServices" and "UserGlobalStorageService" these implementations should also be in code, considering that they are exposed through the server container already. This might also imply moving the CLI commands from files_external to core as well to manage configurations.

Yep. That part isn't storage-specific and as such, it should be in core. Those services (and the UserStorageService) access to already-configured mount points, and that responsability should fall under core.

We need to make sure that, even though the backends could provide custom configuration elements (a combobox, for example) that aren't defined in core, those elements are also acknowledged by core and can be configured using the "standard" CLI commands to configure the external storages. I think this is an important point: the core can provide standard components in order to use them easily, but it should be able to acknowledge other custom components and interact with them

[PVince81]

I think custom controls already work. When declaring a storage backend, you can specify a custom JS file that will manipulate the UI elements to replace some fields with custom elements (ex: date picker).

The CLI works in a generic manner so it would accept arbitrary values and should already work with custom fields.

[PVince81]

The backend and API stuff was already moved to core, so for all storages that use the core API the files_external app does not need to be enabled for the storages to work.

However the UI and OCC commands still needs to be ported to core for this to be more complete.

[PVince81]

Will move the web UI and OCC commands later

[PVince81]

There isn't enough time left to move GUI and OCC commands for now.
To avoid breaking the user experience, the files_external app will now be always enabled by default: #27051

This until we can finish these follow up tasks:

• move storage backends to separate apps: Move files_external backends to separate apps #27049
• move GUI / OCC commands to core: Move files_external GUI and OCC commands to core #27050

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.