Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[API Proposal]: Generic overloads of existing TensorPrimitives methods #94553

Closed
stephentoub opened this issue Nov 9, 2023 · 8 comments · Fixed by #97192
Closed

[API Proposal]: Generic overloads of existing TensorPrimitives methods #94553

stephentoub opened this issue Nov 9, 2023 · 8 comments · Fixed by #97192
Assignees
@stephentoub
Labels
api-approved API was approved in API review, it can be implemented area-System.Numerics
Milestone
9.0.0

Comments

@stephentoub
Copy link
Member

stephentoub commented Nov 9, 2023 •
edited
Loading

Background and motivation

For .NET 8, we added the new TensorPrimitives type, with methods dedicated to handling float. For post-.NET 8, we're planning to augment this in three ways (#93286):

  1. Generic versions of these methods in order to handle other numerical types beyond float
  2. Additional generic methods for all the operations on the generic math interfaces that aren't currently on TensorPrimitives
  3. Additional generic methods to have good coverage of further relevant operations ala BLAS / LAPACK

This issue covers (1).

API Proposal

Exactly the same signatures as on TensorPrimitives in .NET 8, with an overload that takes a T instead of float.

namespace System.Numerics.Tensors;

public static class TensorPrimitives
{
    public static void Abs<T>(ReadOnlySpan<T> x, Span<T> destination) where T : INumberBase<T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, T y, Span<T> destination)  where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Cosh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
    public static T CosineSimilarity<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IRootFunctions<T>;
    public static T Distance<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>, IRootFunctions<T>;
    public static void Divide<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static void Divide<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static T Dot<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static void Exp<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunctions<T>;
    public static int IndexOfMax<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMin<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMinMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Log2<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;
    public static void Log<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;
    public static T MaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MaxMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T Max<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Max<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T MinMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MinMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T Min<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Min<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static void Negate<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IUnaryNegationOperators<T, T>;
    public static T Norm<T>(ReadOnlySpan<T> x) where T : IRootFunctions<T>;
    public static T ProductOfDifferences<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : ISubtractionOperators<T, T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static T ProductOfSums<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static T Product<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void Sigmoid<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunction<T>;
    public static void Sinh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunction<T>;
    public static void SoftMax<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExpoentialFunction<T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static T SumOfMagnitudes<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static T SumOfSquares<T>(ReadOnlySpan<T> x) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>;
    public static T Sum<T>(ReadOnlySpan<T> x) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Tanh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
}
  • Each method is constrained to what I believe is the smallest viable set of interfaces required (I think... we should double check). @tannergooding, is this the right thing to do? Or should we have everything either constrain to T to INumber<T> or IFloatingPointIeee754<T> for simplicity / consistency / future flexibility?

API Usage

double[] values1 = ..., values2 = ...;
double similarity = TensorPrimitives.CosineSimilarity(values1, values2);

Alternative Designs

No response

Risks

No response
@stephentoub stephentoub added api-suggestion Early API idea and discussion, it is NOT ready for implementation area-System.Numerics labels Nov 9, 2023
@stephentoub stephentoub added this to the 9.0.0 milestone Nov 9, 2023
@ghost
Copy link

ghost commented Nov 9, 2023

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

Background and motivation

For .NET 8, we added the new TensorPrimitives type, with methods dedicated to handling float. For post-.NET 8, we're planning to augment this in three ways (#93286):

  1. Generic versions of these methods in order to handle other numerical types beyond float
  2. Additional generic methods for all the operations on the generic math interfaces that aren't currently on TensorPrimitives
  3. Additional generic methods to have good coverage of further relevant operations ala BLAS / LAPACK

This issue covers (1).

API Proposal

Exactly the same signatures as on TensorPrimitives in .NET 8, with an overload that takes a T instead of float.

namespace System.Numerics.Tensors;

public static class TensorPrimitives
{
    public static void Abs<T>(ReadOnlySpan<T> x, Span<T> destination) where T : INumber<T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static void Add<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : INumber<T>;
    public static void ConvertToHalf(ReadOnlySpan<float> source, Span<Half> destination);
    public static void ConvertToSingle(ReadOnlySpan<Half> source, Span<float> destination);
    public static void Cosh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static T CosineSimilarity<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IFloatingPointIeee754<T>;
    public static T Distance<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>, IRootFunctions<T>;
    public static void Divide<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static void Divide<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static T Dot<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>;
    public static void Exp<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static void Log2<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static void Log<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static T MaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void MaxMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T Max<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Max<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T MinMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void MinMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T Min<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Min<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : INumberBase<T>;
    public static void Negate<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IUnaryNegationOperators<T, T>;
    public static T Norm<T>(ReadOnlySpan<T> x) where T : IRootFunctions<T>;
    public static T ProductOfDifferences<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>;
    public static T ProductOfSums<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>;
    public static T Product<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void Sigmoid<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static void Sinh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static void SoftMax<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static T SumOfMagnitudes<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static T SumOfSquares<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static T Sum<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void Tanh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IFloatingPointIeee754<T>;
}
  • Each method is constrained to what I believe is the smallest viable set of interfaces required (I think... we should double check). @tannergooding, is this the right thing to do? Or should we have everything either constrain to T to INumber<T> or IFloatingPointIeee754<T> for simplicity / consistency / future flexibility?

API Usage

double[] values1 = ..., values2 = ...;
double similarity = TensorPrimitives.CosineSimilarity(values1, values2);

Alternative Designs

No response

Risks

No response

Author: stephentoub
Assignees: -
Labels:

api-suggestion, area-System.Numerics
Milestone: 9.0.0

@stephentoub stephentoub self-assigned this Nov 9, 2023
@stephentoub stephentoub mentioned this issue Nov 9, 2023
Merged
@ghost ghost added in-pr There is an active PR which will close this issue when it is merged and removed in-pr There is an active PR which will close this issue when it is merged labels Nov 9, 2023
@tannergooding
Copy link
Member

@tannergooding, is this the right thing to do? Or should we have everything either constrain to T to INumber or IFloatingPointIeee754 for simplicity / consistency / future flexibility?
API Usage

A little bit torn here. But I think restricting to the smallest subset is sufficient and we can look into this again as we implement and before shipping to determine if it needs to be tweaked.

I do think there are smaller subsets then you've picked for a number of them.

I expect in practice INumberBase<T>, INumber<T>, and IFloatingPointIeee754<T> cover the vast majority of everyone's needs. Likewise, given we can't generically access static methods of more derived types via some type check, it gives us more freedom to support things generically without checking for very specific types.

Given that, we can then relax that in the future when a feature like bridging generic constraints is introduced, it is the ultimately safer option and shouldn't restrict any mainline scenarios.

However, some of the functionality like Add is also super simple, and really the only optimization there is to use SIMD for which we can just do if (Vector128<T>.IsSupported && Vector128.IsHardwareAccelerated) { .... }, so simply constraining to IAdditionOperators<T, T, T> or IAdditionOperators<T, T, T> + IAdditiveIdentity<T, T> should always be sufficient.

Likewise something such as Sinh is available via IHyperbolicFunctions<T> and we don't have to implement it ourselves, except for two explicit SIMD scenarios where checking for the type is warranted (float and double) and so I don't think we'd run into any problems by restricting to the smallest set in each case.

@tannergooding
Copy link
Member

I think the following are the APIs with the minimal set of interfaces needed on each. We could require that IAdditiveIdentity be used on anything taking IAdditionOperators/ISubtractionOperators and that IMultiplicativeIdentity be required on anything taking IMultiplicationOperators/IDivisionOperators, but I don' think its strictly necessary.

// IAdditionOperators<T, T, T>
    public static void Add<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IAdditionOperators<T, T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IAdditionOperators<T, T, T>;

// IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>
    public static T Sum<T>(ReadOnlySpan<T> x) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;

// IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>
    public static T Dot<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>;
    public static T SumOfSquares<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;

// IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;

// IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>
    public static T ProductOfSums<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;

// IDivisionOperators<T, T, T>
    public static void Divide<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static void Divide<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IDivisionOperators<T, T, T>;

// IExponentialFunctions<T> -- Implies INumberBase<T>, IFloatingPointConstants<T>
    public static void Exp<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunctions<T>;
    public static void SoftMax<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunctions<T>;
    public static void Sigmoid<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunctions<T>;

// IHyperbolicFunctions<T> -- Implies INumberBase<T>, IFloatingPointConstants<T>
    public static void Cosh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
    public static void Sinh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
    public static void Tanh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;

// ILogarithmicFunctions<T> -- Implies INumberBase<T>, IFloatingPointConstants<T>
    public static void Log<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;
    public static void Log2<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;

// IMultiplyOperators<T, T, T>
    public static void Multiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IMultiplyOperators<T, T, T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IMultiplyOperators<T, T, T>;

// IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>
    public static T Product<T>(ReadOnlySpan<T> x) where T : IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;

// INumber<T>
    public static int IndexOfMax<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMin<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static T Max<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Max<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T Min<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Min<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;

// INumberBase<T>
    public static void Abs<T>(ReadOnlySpan<T> x, Span<T> destination) where T : INumberBase<T>;
    public static int IndexOfMaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static int IndexOfMinMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static T MaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MaxMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T MinMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MinMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T SumOfMagnitudes<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;

// IRootFunctions<T> -- Implies INumberBase<T>, IFloatingPointConstants<T>
    public static T CosineSimilarity<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IRootFunctions<T>;
    public static T Distance<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IRootFunctions<T>;
    public static T Norm<T>(ReadOnlySpan<T> x) where T : IRootFunctions<T>;

// ISubtractionOperators<T, T, T>
    public static void Subtract<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;

// ISubtractionOperators<T, T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>
    public static T ProductOfDifferences<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>;

// IUnaryNegationOperators<T, T>
    public static void Negate<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IUnaryNegationOperators<T, T>;

// Other -- supporting arbitrary floating-point types is possible, but very difficult
    public static void ConvertToHalf(ReadOnlySpan<double> source, Span<Half> destination);
    public static void ConvertToSingle(ReadOnlySpan<Half> source, Span<double> destination);

@stephentoub
Copy link
Member Author

I do think there are smaller subsets then you've picked for a number of them.

Yeah, I wrote before I updated most of them to use INumber, so my comment was inconsistent.

That said, unless we change how things are implemented, we can't make many of them as fine-grained as you've outlined. For example, you've noted Add as only needing where T : IAdditionOperators<T, T, T>;, but in the implementation today it also needs access to T.Zero (as the same operator is used in operations like Sum), and T.Zero would require a constraint to at least INumberBase<T>. There are similar issues for other operators.

I can go through and make them the minimum possible with today's implementation and then we can possibly revise the implementation subsequently if we want to bring it down even further.

@tannergooding
Copy link
Member

you've noted Add as only needing where T : IAdditionOperators<T, T, T>;, but in the implementation today it also needs access to T.Zero (as the same operator is used in operations like Sum), and T.Zero would require a constraint to at least INumberBase. There are similar issues for other operators.

It would really just need IAdditiveIdentity<T, T> so it has access to T.AdditiveIdentity. In practice this is Zero (for number types), but it's the more correct thing for arbitrary types. -- We don't do this, but an example is that we could have string implement IAdditionOperators<string, string, string> in which case its AdditiveIdentity is string.Empty. This then ends up "well-behaved" for the purposes of TensorPrimitives.Add or TensorPrimitives.Sum. Some languages support this type of functionality, while, others don't.

But, the notably the need for IAdditiveIdentity<T, T> only comes from us reusing AddOperator in Sum. It could easily be tweaked or layered to avoid this dependence or we could simply say that Add needs an additional interface that isn't technically required, but simplifies the total implementation enough to make it worthwhile.

@stephentoub stephentoub added api-ready-for-review API is ready for review, it is NOT ready for implementation and removed api-suggestion Early API idea and discussion, it is NOT ready for implementation labels Nov 10, 2023
@stephentoub
Copy link
Member Author

It would really just need IAdditiveIdentity<T, T> so it has access to T.AdditiveIdentity.

Ok.

But, the notably the need for IAdditiveIdentity<T, T> only comes from us reusing AddOperator in Sum. It could easily be tweaked or layered to avoid this dependence or we could simply say that Add needs an additional interface that isn't technically required, but simplifies the total implementation enough to make it worthwhile.

Right, that's what I meant by "unless we change how things are implemented" and "we can possibly revise the implementation subsequently if we want to bring it down even further". I don't think it's worth adding more code here to reduce the constraints right now.

@bartonjs
Copy link
Member

bartonjs commented Nov 14, 2023 •
edited by dotnet-api-review bot
Loading

Video

Looks good as proposed (taking all of the constraints on faith)

namespace System.Numerics.Tensors;

public static partial class TensorPrimitives
{
    public static void Abs<T>(ReadOnlySpan<T> x, Span<T> destination) where T : INumberBase<T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void AddMultiply<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> multiplier, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Add<T>(ReadOnlySpan<T> x, T y, Span<T> destination)  where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Cosh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
    public static T CosineSimilarity<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IRootFunctions<T>;
    public static T Distance<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : INumberBase<T>, IRootFunctions<T>;
    public static void Divide<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static void Divide<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IDivisionOperators<T, T, T>;
    public static T Dot<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static void Exp<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunctions<T>;
    public static int IndexOfMax<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMin<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMinMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Log2<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;
    public static void Log<T>(ReadOnlySpan<T> x, Span<T> destination) where T : ILogarithmicFunctions<T>;
    public static T MaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MaxMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T Max<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Max<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static T MinMagnitude<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void MinMagnitude<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static T Min<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static void Min<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumber<T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, T addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void MultiplyAdd<T>(ReadOnlySpan<T> x, T y, ReadOnlySpan<T> addend, Span<T> destination) where T : IAdditionOperators<T, T, T>, IMultiplyOperators<T, T, T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : INumberBase<T>;
    public static void Multiply<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static void Negate<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IUnaryNegationOperators<T, T>;
    public static T Norm<T>(ReadOnlySpan<T> x) where T : IRootFunctions<T>;
    public static T ProductOfDifferences<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : ISubtractionOperators<T, T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static T ProductOfSums<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>, IMultiplicativeIdentity<T, T>;
    public static T Product<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static void Sigmoid<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExponentialFunction<T>;
    public static void Sinh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunction<T>;
    public static void SoftMax<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IExpoentialFunction<T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, ReadOnlySpan<T> y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static void Subtract<T>(ReadOnlySpan<T> x, T y, Span<T> destination) where T : ISubtractionOperators<T, T, T>;
    public static T SumOfMagnitudes<T>(ReadOnlySpan<T> x) where T : INumberBase<T>;
    public static T SumOfSquares<T>(ReadOnlySpan<T> x) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>, IMultiplyOperators<T, T, T>;
    public static T Sum<T>(ReadOnlySpan<T> x) where T : IAdditionOperators<T, T, T>, IAdditiveIdentity<T, T>;
    public static void Tanh<T>(ReadOnlySpan<T> x, Span<T> destination) where T : IHyperbolicFunctions<T>;
}

@bartonjs bartonjs added api-approved API was approved in API review, it can be implemented and removed api-ready-for-review API is ready for review, it is NOT ready for implementation labels Nov 14, 2023
@stephentoub stephentoub mentioned this issue Jan 3, 2024
Closed
@stephentoub
Copy link
Member Author

Everything here has been done, except:

    public static int IndexOfMax<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMaxMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMin<T>(ReadOnlySpan<T> x) where T : INumber<T>;
    public static int IndexOfMinMagnitude<T>(ReadOnlySpan<T> x) where T : INumber<T>;

Those still need to be added.

@stephentoub stephentoub mentioned this issue Jan 19, 2024
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jan 19, 2024
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jan 22, 2024
@333fred 333fred mentioned this issue Feb 3, 2024
Open
4 tasks
@github-actions github-actions bot locked and limited conversation to collaborators Feb 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@stephentoub stephentoub

Labels
api-approved API was approved in API review, it can be implemented area-System.Numerics
Projects
None yet
Milestone
9.0.0
Development

Successfully merging a pull request may close this issue.

Add lots more TensorPrimitives operations stephentoub/runtime
3 participants
@stephentoub @tannergooding @bartonjs