Fix torch svd

[Daniel4078]

PR Description

Related Issue

Closes #28769

Checklist

• Did you add a function?
• Did you add the tests?
• Did you run your tests and are your tests passing?
• Did pre-commit not fail on any check?
• Did you follow the steps we provided?

Socials

[Daniel4078]

@Sam-Armstrong Can you tell me about how to get the device of the input received by the torch frontend function svd? Because the documentation requires the zero-filled matrixes in the output should be on the same device as input (https://pytorch.org/docs/stable/generated/torch.svd.html).
Also, why does the ground truth of the svd test shown here (https://github.com/Transpile-AI/ivy/actions/runs/9589504921/job/26443402936#step:6:184) is not in the correct form of (U,S,V) but only contain two matrixs?

[Sam-Armstrong]

@Daniel4078 you should just be able to do x.device right? I'm not sure about the ground truth, but it should be produced by the native torch function, so you should be able to reproduce the results manually in a python script

[Daniel4078]

@Sam-Armstrong I found something strange when I run the test locally. The test's error is like this

values = [[array([[0., 0.],
       [0., 0.]], dtype=float32), array([[4.89516249e-308, 0.00000000e+000],
       [4.89516249e-30...    [4.89516249e-308, 0.00000000e+000]]), array([[[0., 0.],
        [0., 0.]],

       [[0., 0.],
        [0., 0.]]])]]
this_key_chain = None

    def assert_same_type_and_shape(values, this_key_chain=None):
        x_, y_ = values
        for x, y in zip(x_, y_):
            if isinstance(x, np.ndarray):
                x_d = str(x.dtype).replace("longlong", "int64")
                y_d = str(y.dtype).replace("longlong", "int64")
>               assert (
                    x.shape == y.shape
                ), f"returned shape = {x.shape}, ground-truth returned shape = {y.shape}"
E               AssertionError: returned shape = (2, 2), ground-truth returned shape = (2, 2, 2)
E               Falsifying example: test_torch_svd(
E                   on_device='cpu',
E                   frontend='torch',
E                   backend_fw='jax',
E                   dtype_and_x=(['float64'], [array([[[-2.44758124e-308, -2.44758124e-308],
E                             [-2.44758124e-308, -2.44758124e-308]],
E                     
E                            [[-2.44758124e-308, -2.44758124e-308],
E                             [-2.44758124e-308, -2.44758124e-308]]])]),
E                   some=False,
E                   compute=False,
E                   test_flags=FrontendFunctionTestFlags(
E                       num_positional_args=0,
E                       with_out=False,
E                       with_copy=False,
E                       inplace=False,
E                       as_variable=[False],
E                       native_arrays=[False],
E                       test_trace=False,
E                       test_trace_each=False,
E                       generate_frontend_arrays=False,
E                       transpile=False,
E                       precision_mode=True,
E                   ),
E                   fn_tree='ivy.functional.frontends.torch.svd',

Here the return from both the ground truth function and ourfrontend are named tuples (U,S,V) where U and V should be zero matrixs (just like what is in the "values" variable). It seems like the assertion check somehow pick the first two part of one returned tuple and consider them as the frontend-groundtruth pair. I think maybe we should investigate this further as x_, y_ = values does not seem to work correctly here.

[Sam-Armstrong]

@Daniel4078 isn't the problem here that the shape of the tensors in the named tuple is different to the ground truth? Like the first element U of the tuple is 3d in the ground truth but 2d in the function return

[Daniel4078]

@Daniel4078 isn't the problem here that the shape of the tensors in the named tuple is different to the ground truth? Like the first element U of the tuple is 3d in the ground truth but 2d in the function return

@Sam-Armstrong I see, comparing to the document of ivy.svd, it seems like the document of torch.svd assume the input is 2D and is incomlete.

[Daniel4078]

@Sam-Armstrong the tests now sometime fails due to the fact that the correct output of svd is not unique and I am thinking of copying the test for ivy.linalg.svd(

ivy/ivy_tests/test_ivy/test_functional/test_core/test_linalg.py

Line 944 in 72fa169

# value test based on recreating the original matrix and testing the consistency

) instead as it test values by doing actual calcuations. Do you agree of this approach? And should I also apply this fix to other failing frontend svd functions in this PR?

[Sam-Armstrong]

@Daniel4078 sure, think that makes sense. yeah that would be great if you can fix the other failing frontend svd tests in this PR, thanks!

[Daniel4078]

@Daniel4078 sure, think that makes sense. yeah that would be great if you can fix the other failing frontend svd tests in this PR, thanks!

It is quite surprising that now torch.blas_and_lapack_ops.svd, torch.tensor.svd, jax.lax.linalg.svd, jax.numpy.linalg.svd, numpy.linalg.decompositions.svd are failing; while torch.linalg.svd , tensorflow.linalg.svd and tensorflow.raw_ops.svd are skipped. The only passing svd test is that of ivy functional API core.

[Sam-Armstrong]

@Daniel4078 you haven't made any changes to the tests here, did you force-push over them?

[Daniel4078]

@Daniel4078 you haven't made any changes to the tests here, did you force-push over them?

I accidently commited some other changes intended in another branch in here, so I thought it would be better to remove that commit.

[Daniel4078]

@Sam-Armstrong Why all the torch.svd frontend tests are skipping when I run it locally, showing that "Skipped: Function not implemented in backend." What should I do in this case?

[Daniel4078]

trying to solve the TypeError: iteration over a 0-d array error when comparing two (somehow empty) result from the matric calculation

[Sam-Armstrong]

The reason the test is failing is you're trying to convert a torch tensor to a numpy array, without detaching the gradients (using tensor.detach().numpy()). I don't really understand the reasoning behind changing the format of the tests, such as the comment I left below - can't we just change the atol_/rtol_ values with test_frontend_function, rather than adding these extra assertions? Or are there other reasons you did this?

[Daniel4078]

The reason the test is failing is you're trying to convert a torch tensor to a numpy array, without detaching the gradients (using tensor.detach().numpy()). I don't really understand the reasoning behind changing the format of the tests, such as the comment I left below - can't we just change the atol_/rtol_ values with test_frontend_function, rather than adding these extra assertions? Or are there other reasons you did this?

But the part that convert a torch tensor to a numpy array is not implemented by me and the same process works for tests of tenserflow.linalg.svd so I did not think I should not do the convertion. Or is there some way to do the same computation without converting? Thank you!

[Sam-Armstrong]

Looks like these svd tests are still failing, could you try to resolve these as well? But otherwise that makes sense 👍 thanks! 😊

• ivy_tests/test_ivy/test_frontends/test_torch/test_tensor.py::test_torch_svd
• ivy_tests/test_ivy/test_frontends/test_numpy/test_linalg/test_decompositions.py::test_numpy_svd
• ivy_tests/test_ivy/test_frontends/test_jax/test_numpy/test_linalg.py::test_jax_svd
• ivy_tests/test_ivy/test_frontends/test_jax/test_lax/test_linalg.py::test_jax_svd
• ivy_tests/test_ivy/test_frontends/test_tensorflow/test_raw_ops.py::test_tensorflow_Svd

[Daniel4078]

it is strange that with the same calculation to reconstruct the decomposited matrix (used in all other relevant tests), current jax.numpy.linalg.svd passes all tests while current jax.lax.linalg.svd fails with significantly different values produced. May be the decomposition of jax.lax.linalg.svd is different from the common way?

[Daniel4078]

found that the svd function of tenserflow backend does not support complex input, this may be the reason why some of the frontend tests for svd returns wrong results in tensorflow backend for complex value inputs. Also, I want to ask is the first component of the return of the helpers.test_frontend_function representing the groundtruth return?

[Daniel4078]

found that the svd function of tenserflow backend does not support complex input, this may be the reason why some of the frontend tests for svd returns wrong results in tensorflow backend for complex value inputs. Also, I want to ask is the first component of the return of the helpers.test_frontend_function representing the groundtruth return?