Input/Output Errors When Reading/Writing Files After Successfully Mounting CephFS Using go-ceph/cephfs

[herenli]

Hello,
I am encountering input/output errors when trying to read and write files after successfully mounting the Ceph File System (CephFS) using the go-ceph/cephfs library.

Context:
I have successfully mounted the CephFS manually, and I can create directories and files without any issues.
However, when using the go-ceph/cephfs interface to perform read and write operations, I encounter input/output errors.

Error Output:
go run -tags luminous test.go
Success creating CephFS mount
Success reading default config file
Success mounting CephFS
Directory created successfully.
Error writing to file: cephfs: ret=5, Input/output error

Details:
go-ceph v0.4.0
ceph luminous

Could anyone provide insights or suggestions on why this might be happening? Any help would be greatly appreciated!
Thank you!

the code is attached.
code.md

[nixpanic]

go-ceph v0.4.0
ceph luminous

Wow, these are both quite old and not maintained anymore.

There is a very similar existing test:

go-ceph/cephfs/file_test.go

Lines 80 to 96 in 2c86010

	t.Run("writeAndRead", func(t *testing.T) {
	// idempotent open for read and write
	f1, err := mount.Open(fname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
	n, err := f1.Write([]byte("yello world!"))
	assert.NoError(t, err)
	assert.EqualValues(t, 12, n)
	err = f1.Close()
	assert.NoError(t, err)
	defer func() { assert.NoError(t, mount.Unlink(fname)) }()
	buf := make([]byte, 1024)
	f2, err := mount.Open(fname, os.O_RDONLY, 0)
	n, err = f2.Read(buf)
	assert.NoError(t, err)
	assert.EqualValues(t, 12, n)
	assert.Equal(t, "yello world!", string(buf[:n]))
	})

I have added an extra mount.MakeDir() and put the fname in the new directory, like this:

	t.Run("writeAndReadInDir", func(t *testing.T) {
		dirname := "/TestDir"
		dfname := dirname + "/" + fname

		err := mount.MakeDir(dirname, 0777)
		assert.NoError(t, err)
		defer func() { assert.NoError(t, mount.RemoveDir(dirname)) }()

		// idempotent open for read and write
		f1, err := mount.Open(dfname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
		n, err := f1.Write([]byte("yello world!"))
		assert.NoError(t, err)
		assert.EqualValues(t, 12, n)
		err = f1.Close()
		assert.NoError(t, err)
		defer func() { assert.NoError(t, mount.Unlink(dfname)) }()

		buf := make([]byte, 1024)
		f2, err := mount.Open(dfname, os.O_RDONLY, 0)
		n, err = f2.Read(buf)
		assert.NoError(t, err)
		assert.EqualValues(t, 12, n)
		assert.Equal(t, "yello world!", string(buf[:n]))
	})

After that, running make test-container passes the new TestFileReadWrite/writeAndReadInDir test.

=== RUN   TestFileReadWrite
=== RUN   TestFileReadWrite/writeAndRead
=== RUN   TestFileReadWrite/writeAndReadInDir
=== RUN   TestFileReadWrite/openForWriteOnly
=== RUN   TestFileReadWrite/openForReadOnly
=== RUN   TestFileReadWrite/uninitializedFile
--- PASS: TestFileReadWrite (0.01s)
    --- PASS: TestFileReadWrite/writeAndRead (0.00s)
    --- PASS: TestFileReadWrite/writeAndReadInDir (0.00s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

This probably indicates a permission or other configuration mistake on the Ceph cluster. Maybe logs from the MDS or OSD processes give a hint on what could be wrong?

Have you used the same credentials for mounting with kernel or fuse CephFS, and did that work?

[phlogistonjohn]

I'll 2nd the "wow that's really old" comment from @nixpanic - it's so old that I hesitate to invest a lot of effort here. Please consider upgrading to something more recent & supported by the upstream projects.

I will add that ceph has various debugging option that can be loaded via a ceph.conf or config set functions. See https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/
The Ceph libraries will then emit debug logging (to the standard error IIRC).

[phlogistonjohn]

Oh, I should add I generally find it simpler to turn them all on to level 10. It produces a huge amount of output but I never seem to be able to guess the proper subsystem name for what I am looking for and so I just prefer to cast a wide net.

[herenli]

@nixpanic Thank you for your reply.
I did a similar existing test, but still have an error

--- FAIL: TestFileReadWrite (0.07s)
    --- FAIL: TestFileReadWrite/writeAndRead (0.01s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

And I used ceph-fuse to mount the cephfs manually and can read/write the file successfully.
So, I don't know the reasons.

[herenli]

@phlogistonjohn Thank you for your reply.
I updated the go-ceph version to v0.11 for using of nautilus.
But I still have the input/output errors when reading/writing files after successfully mounting cephfs using go-ceph/cephfs.

=== RUN   TestFileReadWrite/writeAndRead
    cephfs_test.go:336: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:336
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:337: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:337
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:345: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:345
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:346: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:346
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:347: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:347
                Error:          Not equal: 
                                expected: "yello world!"
                                actual  : ""
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -yello world!
                                +
                Test:           TestFileReadWrite/writeAndRead