[files_external] swift tests #14077

MorrisJobke · 2015-02-11T00:08:55Z

I just gave this a try, but I guess someone with more knowledge about Swift should have a look.

There seem to be some authentication problems. Is this url parameter the URL for getting the authentication?

In the README of the docker container it is used like this:

vagrant@host1:~$ swift -A http://127.0.0.1:12345/auth/v1.0 -U test:tester -K testing stat
       Account: AUTH_test
    Containers: 0
       Objects: 0
         Bytes: 0
  Content-Type: text/plain; charset=utf-8
   X-Timestamp: 1402463864.77057
    X-Trans-Id: tx4e7861ebab8244c09dad9-005397e678
X-Put-Timestamp: 1402463864.77057

(-A to specify the URL for the authentication token retrieval, -U for user and -K for the key)

Is this then the correct syntax in our files_external code?

cc @butonic @DeepDiver1975

MorrisJobke · 2015-02-11T00:09:51Z

apps/files_external/tests/env/start-swift-morrisjobke.sh

+    'key'=>'$password',
+    'bucket'=>'swift',
+    'region' => 'DFW',
+);


THe above lines are the ones where I'm not sure if I did that correct.

Maybe see ccollicutt/docker-swift-onlyone#1

PVince81 · 2015-09-01T11:42:59Z

What's the status ?

MorrisJobke · 2015-09-01T11:44:48Z

What's the status ?

Needs further work, because I didn't figured out how to connect from ownCloud to swift ... maybe @Xenopathic can help here.

Maybe we can use the ceph/demo docker container that also exposes swift.

RobinMcCorkell · 2015-09-09T15:20:13Z

So, it turns out the PHP library we use to connect to Swift only supports Keystone authentication (v2), not Swift built-in authentication (v1), which the current Docker image and Ceph support. It will be possible to get Keystone integrated with Ceph, but it could take some time...

butonic · 2015-09-09T15:31:34Z

@MorrisJobke nope. swift with ceph wont work because the libs we use don't support Keystone v1.

@Xenopathic I think time would be better invested in a devstack docker container. see https://wiki.openstack.org/wiki/Docker ... it seems several people are trying this: https://hub.docker.com/search/?q=devstack&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0

RobinMcCorkell · 2015-09-09T15:34:55Z

@butonic With a bit of luck, setting up Keystone and connecting it to Ceph should be relatively straight-forward. Hopefully, I can avoid needing a MySQL database...: http://ceph.com/docs/master/radosgw/keystone/

butonic · 2015-09-09T15:37:01Z

@Xenopathic AFAIU you need to set up an openstack instance with keystone to make ceph hand over auth to that service. So ... twice the trouble. But please prove me wrong ;)

RobinMcCorkell · 2015-09-09T18:31:56Z

@butonic Sorry to break the bad news, but I just got Keystone working with Ceph without needing the whole OpenStack full-stack. I nearly swallowed my own tongue when the little green dot appeared...

scrutinizer-notifier · 2015-09-09T21:16:22Z

A new inspection was created.

RobinMcCorkell · 2015-09-09T21:17:15Z

@MorrisJobke @butonic Get testing! The Docker image is available at https://hub.docker.com/r/xenopathic/ceph-keystone/, and I have full intentions of creating an S3 test around the same image.

MorrisJobke · 2015-09-09T21:30:50Z

I nearly swallowed my own tongue when the little green dot appeared...

? Jenkins needs to be configured to actually run this one ;)

RobinMcCorkell · 2015-09-09T21:43:46Z

@MorrisJobke No, in the admin interface, manually configuring it to see if it actually worked. But I can guarantee that Jenkins will fail, since one of the tests fails on Swift (good we've got unit testing now!)

RobinMcCorkell · 2015-09-14T13:48:23Z

@icewind1991 @MorrisJobke @butonic Review?

Note that the tests actually fail at the moment (two failures), but the tests do run, so this PR can be considered successful.

MorrisJobke · 2015-10-20T12:09:28Z

Can we reuse the objectstore primary test setup here somehow? cc @Xenopathic @butonic

RobinMcCorkell · 2015-10-20T12:15:54Z

@MorrisJobke The start/stop scripts are ever so slightly different, it might not be worth trying to combine them. These files are unlikely to change much, and when they do we can just port changes between them.

BTW, I've added you as a collaborator on the xenopathic/ceph-keystone image, so if any changes need to be made you can make them directly 😄

Use netcat to check when the port opens, rather than an arbitrary timeout. Hard limit of 60 seconds in case something breaks

RobinMcCorkell · 2015-11-04T15:23:21Z

Can we get this retested and merged?

MorrisJobke · 2015-11-04T15:43:43Z

Docker logs:

docker logs -f aac
creating /etc/ceph/ceph.client.admin.keyring
creating /etc/ceph/ceph.mon.keyring
monmaptool: monmap file /etc/ceph/monmap
monmaptool: set fsid to 8ab4d030-204c-4627-b073-3c9bf6fd39e5
monmaptool: writing epoch 0 to /etc/ceph/monmap (1 monitors)
creating /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
ceph-mon: set fsid to 8ab4d030-204c-4627-b073-3c9bf6fd39e5
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-aacb8faf079a for mon.aacb8faf079a
set pool 0 size to 1
0
2015-11-04 15:39:54.405959 7f6c59cf2900 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway




2015-11-04 15:39:56.372399 7f6c59cf2900 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-11-04 15:39:56.399258 7f6c59cf2900 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2015-11-04 15:39:57.550100 7f6c59cf2900 -1 created object store /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsid 8ab4d030-204c-4627-b073-3c9bf6fd39e5
add item id 0 name 'osd.0' weight 1 at location {host=aacb8faf079a,root=default} to crush map
starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
pool 'cephfs_data' created
pool 'cephfs_metadata' created
new fs with metadata pool 2 and data pool 1
2015-11-04 15:40:04.905476 7fa7e97cf800 -1 deprecation warning: MDS id 'mds.0' is invalid and will be forbidden in a future version.  MDS names may not start with a numeric digit.
starting mds.0 at :/0

autotest logs:

Status: Image is up to date for xenopathic/ceph-keystone:latest
xenopathic/ceph-keystone container: aacb8faf079a91a7d793cf304777a16009d072a24481b905ed8100dc4751084e
Waiting for ceph initialization............................................................
[ERROR] Waited 60 seconds, no response
stop: stop-swift-ceph.sh
Docker executable found - stop and remove docker containers
Stopping and removing docker container aacb8faf079a91a7d793cf304777a16009d072a24481b905ed8100dc4751084e
Error response from daemon: Cannot destroy container aacb8faf079a91a7d793cf304777a16009d072a24481b905ed8100dc4751084e: Driver btrfs failed to remove root filesystem aacb8faf079a91a7d793cf304777a16009d072a24481b905ed8100dc4751084e: Failed to destroy btrfs snapshot: directory not empty
Error: failed to remove containers: [aacb8faf079a91a7d793cf304777a16009d072a24481b905ed8100dc4751084e]
rm: das Entfernen von „./../apps/files_external/tests//config.swift.php“ ist nicht möglich: Datei oder Verzeichnis nicht gefunden
skip: start-webdav-ownCloud.sh

RobinMcCorkell · 2015-11-04T15:46:10Z

Is this that btrfs loopback thing with Ceph again?

MorrisJobke · 2015-11-04T15:46:50Z

Is this that btrfs loopback thing with Ceph again?

Could be.

PVince81 · 2015-11-16T16:24:09Z

Almost there:

Configuration read from /srv/www/htdocs/owncloud/tests/phpunit-autotest-external.xml

S................................................................ 65 / 73 ( 89%)
........

Time: 33.12 seconds, Memory: 24.00Mb

There was 1 skipped test:

1) Test\Files\Storage\Swift::testStat
Swift doesn't update the parents folder mtime

/srv/www/htdocs/owncloud/apps/files_external/tests/backends/swift.php:64

OK, but incomplete, skipped, or risky tests!
Tests: 73, Assertions: 323, Skipped: 1.
stop: stop-swift-ceph.sh
Docker executable found - stop and remove docker containers
Stopping and removing docker container 2659814329cba585215a939126e57e871b967ba95387d6bda99496ec86e35882
Error response from daemon: Cannot destroy container 2659814329cba585215a939126e57e871b967ba95387d6bda99496ec86e35882: Driver btrfs failed to remove root filesystem 2659814329cba585215a939126e57e871b967ba95387d6bda99496ec86e35882: Failed to destroy btrfs snapshot: directory not empty
Error: failed to remove containers: [2659814329cba585215a939126e57e871b967ba95387d6bda99496ec86e35882]
skip: start-webdav-ownCloud.sh

Tests passed, there was just an issue when cleaning up. Might not be critical.
This is with a BTRFS file system on the host.

PVince81 · 2015-11-16T16:26:34Z

The error code is still 0, so if this is run on CI it would like interpret it as a success.

TEST: ./autotest-external.sh sqlite swift-ceph
TEST: ./autotest-external.sh sqlite amazons3-ceph

👍 from me

icewind1991 · 2015-11-16T17:23:35Z

👍 looks good

RobinMcCorkell · 2015-11-16T18:53:57Z

@MorrisJobke Please confirm if we need to port the Btrfs thing into this branch?

icewind1991 · 2015-11-16T20:53:00Z

Improved the command to get the ip for the swift container

MorrisJobke · 2015-11-17T08:46:13Z

@MorrisJobke Please confirm if we need to port the Btrfs thing into this branch?

It doesn't work here directly but I need to investigate further because the startup of the container looks good (It needed a longer time to start than 60 seconds).

I would accept this, if this runs fine on Jenkins. @DeepDiver1975 Can you setup the CI run for this? Thanks

PVince81 · 2015-11-20T10:44:05Z

Can we move this forward ? There are at least 4 improvement PRs for SWIFT that are waiting for these tests.

MorrisJobke · 2015-11-20T10:52:51Z

I'm fine with merging. @DeepDiver1975 The CI job shouldn't be that hard to setup, right? 😉

DeepDiver1975 · 2015-11-20T10:53:45Z

I'm fine with merging. @DeepDiver1975 The CI job shouldn't be that hard to setup, right? 😉

no

MorrisJobke · 2015-11-20T12:56:41Z

no

Then please merge ;)

[files_external] swift tests

MorrisJobke added 3 - To Review 2 - Developing QA app:files_external and removed 3 - To Review labels Feb 11, 2015

MorrisJobke added this to the 8.1-next milestone Feb 11, 2015

MorrisJobke reviewed Feb 11, 2015
View reviewed changes

DeepDiver1975 modified the milestones: 8.1-current, 8.2-next Mar 2, 2015

MorrisJobke force-pushed the autotest-external-swift branch from 2fa9115 to 4529ff7 Compare July 7, 2015 11:19

RobinMcCorkell self-assigned this Sep 1, 2015

RobinMcCorkell force-pushed the autotest-external-swift branch from 4529ff7 to 3d9d47e Compare September 9, 2015 21:16

This was referenced Sep 10, 2015

Introduce CI for files_external #11717

Closed

Streaming download from Swift in files_external #19002

Merged

This was referenced Sep 14, 2015

Improving fopen behaviour for Swift in files_external #19001

Closed

Don't log expected Swift 404 responses in files_external #19000

Merged

MorrisJobke and others added 6 commits October 20, 2015 14:45

[files_external] swift tests

2577a62

Replace Swift Docker test with Ceph image

0de9819

Make Swift test robust if container didn't exist

70f4e94

Detect Docker initialization properly

8467dbf

Use netcat to check when the port opens, rather than an arbitrary timeout. Hard limit of 60 seconds in case something breaks

Don't attempt to run tests unless start file is successful

ad028c5

Skip Swift stat test

c98b303

DeepDiver1975 force-pushed the autotest-external-swift branch from 219a72e to c98b303 Compare October 20, 2015 12:45

fix ip detection for swift-ceph

d49fc9a

MorrisJobke added 4 - To release and removed 3 - To Review labels Nov 20, 2015

DeepDiver1975 added a commit that referenced this pull request Nov 20, 2015

Merge pull request #14077 from owncloud/autotest-external-swift

308aaf8

[files_external] swift tests

DeepDiver1975 merged commit 308aaf8 into master Nov 20, 2015

DeepDiver1975 deleted the autotest-external-swift branch November 20, 2015 12:57

lock bot locked as resolved and limited conversation to collaborators Aug 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[files_external] swift tests #14077

[files_external] swift tests #14077

MorrisJobke commented Feb 11, 2015

MorrisJobke Feb 11, 2015

MorrisJobke Feb 11, 2015

PVince81 commented Sep 1, 2015

MorrisJobke commented Sep 1, 2015

RobinMcCorkell commented Sep 9, 2015

butonic commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

butonic commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

scrutinizer-notifier commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

MorrisJobke commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

RobinMcCorkell commented Sep 14, 2015

MorrisJobke commented Oct 20, 2015

RobinMcCorkell commented Oct 20, 2015

RobinMcCorkell commented Nov 4, 2015

MorrisJobke commented Nov 4, 2015

RobinMcCorkell commented Nov 4, 2015

MorrisJobke commented Nov 4, 2015

PVince81 commented Nov 16, 2015

PVince81 commented Nov 16, 2015

icewind1991 commented Nov 16, 2015

RobinMcCorkell commented Nov 16, 2015

icewind1991 commented Nov 16, 2015

MorrisJobke commented Nov 17, 2015

PVince81 commented Nov 20, 2015

MorrisJobke commented Nov 20, 2015

DeepDiver1975 commented Nov 20, 2015

MorrisJobke commented Nov 20, 2015

[files_external] swift tests #14077

[files_external] swift tests #14077

Conversation

MorrisJobke commented Feb 11, 2015

MorrisJobke Feb 11, 2015

Choose a reason for hiding this comment

MorrisJobke Feb 11, 2015

Choose a reason for hiding this comment

PVince81 commented Sep 1, 2015

MorrisJobke commented Sep 1, 2015

RobinMcCorkell commented Sep 9, 2015

butonic commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

butonic commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

scrutinizer-notifier commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

MorrisJobke commented Sep 9, 2015

RobinMcCorkell commented Sep 9, 2015

RobinMcCorkell commented Sep 14, 2015

MorrisJobke commented Oct 20, 2015

RobinMcCorkell commented Oct 20, 2015

RobinMcCorkell commented Nov 4, 2015

MorrisJobke commented Nov 4, 2015

RobinMcCorkell commented Nov 4, 2015

MorrisJobke commented Nov 4, 2015

PVince81 commented Nov 16, 2015

PVince81 commented Nov 16, 2015

icewind1991 commented Nov 16, 2015

RobinMcCorkell commented Nov 16, 2015

icewind1991 commented Nov 16, 2015

MorrisJobke commented Nov 17, 2015

PVince81 commented Nov 20, 2015

MorrisJobke commented Nov 20, 2015

DeepDiver1975 commented Nov 20, 2015

MorrisJobke commented Nov 20, 2015