Tensor<T> and supporting types

[tannergooding]

Rationale

.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond.

In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, could benefit from additional improvements. Today, developers writing .NET applications, services, and libraries currently may need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.

API Proposal

This is extracted from dotnet/designs#316 and the design doc will be updated based on the API review here.

Native Index and Range Types

namespace System.Buffers;

public readonly struct NIndex : IEquatable<NIndex>
{
    public NIndex(nint value, bool fromEnd = false);
    public NIndex(Index value);

    public static NIndex End { get; }
    public static NIndex Start { get; }

    public bool IsFromEnd { get; }
    public nint Value { get; }

    public static implicit operator NIndex(nint value);
    public static implicit operator NIndex(Index value);

    public static explicit operator Index(NIndex value);
    public static explicit operator checked Index(NIndex value);

    public static NIndex FromEnd(nint value);
    public static NIndex FromStart(nint value);

    public static Index ToIndex();
    public static Index ToIndexUnchecked();

    public nint GetOffset(nint length);

    // IEquatable<NIndex>
    public bool Equals(NIndex other);
}

public readonly struct NRange : IEquatable<NRange>
{
    public NRange(NIndex start, NIndex end);
    public NRange(Range value);

    public static NRange All { get; }

    public NIndex End { get; }
    public NIndex Start { get; }

    public static explicit operator Range(NRange value);
    public static explicit operator checked Range(NRange value);

    public static NRange EndAt(NIndex end);
    public (nint Offset, nint Length) GetOffsetAndLength(nint length);
    public static NRange StartAt(NIndex start);

    public static Range ToRange();
    public static Range ToRangeUnchecked();

    // IEquatable<NRange>
    public bool Equals(NRange other);
}

Multi-dimensional Span Types

namespace System.Numerics.Tensors;

public ref struct TensorSpan<T>
{
    public TensorSpan(void* data, nint dataLength, ReadOnlySpan<nint> lengths);
    public TensorSpan(void* data, nint dataLength, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public TensorSpan(Span<T> span);
    public TensorSpan(Span<T> span, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public TensorSpan(T[]? array);
    public TensorSpan(T[]? array, int start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    public TensorSpan(T[]? array, Index startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    // Should start just be `nint` and represent the linear index?
    public TensorSpan(Array? array);
    public TensorSpan(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    public TensorSpan(Array? array, ReadOnlySpan<Index> startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    // Should we have variants for common ranks (consider 2-5):
    //   public TensorSpan(T[,]? array);
    //   public TensorSpan(T[,]? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    //   public TensorSpan(T[,]? array, ReadOnlySpan<Index> startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public static TensorSpan<T> Empty { get; }

    public bool IsEmpty { get; }
    public nint FlattenedLength { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Lengths { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref T this[params scoped ReadOnlySpan<nint> indexes] { get; }
    public ref T this[params scoped ReadOnlySpan<NIndex> indexes] { get; }
    public TensorSpan<T> this[params scoped ReadOnlySpan<NRange> ranges] { get; set; }

    // These work like Span<T>, comparing the byref, lengths, and strides, not element-wise
    public static bool operator ==(TensorSpan<T> left, TensorSpan<T> right);
    public static bool operator !=(TensorSpan<T> left, TensorSpan<T> right);

    public static explicit operator TensorSpan<T>(Array? array);
    public static implicit operator TensorSpan<T>(T[]? array);

    // Should we have variants for common ranks (consider 2-5):
    //   public static implicit operator TensorSpan<T>(T[,]? array);

    public static implicit operator ReadOnlyTensorSpan<T>(TensorSpan<T> span);

    public void Clear();
    public void CopyTo(TensorSpan<T> destination);
    public void Fill(T value);
    public void FlattenTo(Span<T> destination);
    public Enumerator GetEnumerator();
    public ref T GetPinnableReference();
    public TensorSpan<T> Slice(params ReadOnlySpan<NIndex> indexes);
    public TensorSpan<T> Slice(params ReadOnlySpan<NRange> ranges);
    public bool TryCopyTo(TensorSpan<T> destination);
    public bool TryFlattenTo(Span<T> destination);

    [ObsoleteAttribute("Equals() on TensorSpan will always throw an exception. Use the equality operator instead.")]
    public override bool Equals(object? obj);

    [ObsoleteAttribute("GetHashCode() on TensorSpan will always throw an exception.")]
    public override int GetHashCode();

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

public ref struct ReadOnlyTensorSpan<T>
{
    public ReadOnlyTensorSpan(void* data, nint dataLength, ReadOnlySpan<nint> lengths);
    public ReadOnlyTensorSpan(void* data, nint dataLength, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public ReadOnlyTensorSpan(T[]? array);
    public ReadOnlyTensorSpan(T[]? array, int start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    public ReadOnlyTensorSpan(T[]? array, Index startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    // Should start just be `nint` and represent the linear index?
    public ReadOnlyTensorSpan(Array? array);
    public ReadOnlyTensorSpan(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    public ReadOnlyTensorSpan(Array? array, ReadOnlySpan<Index> startIndices, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    // Should we have variants for common ranks (consider 2-5):
    //   public ReadOnlyTensorSpan(T[,]? array);
    //   public ReadOnlyTensorSpan(T[,]? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
    //   public ReadOnlyTensorSpan(T[,]? array, ReadOnlySpan<Index> startIndices, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public static ReadOnlyTensorSpan<T> Empty { get; }

    public bool IsEmpty { get; }
    public nint Length { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Lengths { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref readonly T this[params scoped ReadOnlySpan<nint> indexes] { get; }
    public ref readonly T this[params scoped ReadOnlySpan<NIndex> indexes] { get; }
    public ReadOnlyTensorSpan<T> this[params scoped ReadOnlySpan<NRange> ranges] { get; set; }

    // These work like ReadOnlySpan<T>, comparing the byref, lengths, and strides, not element-wise
    public static bool operator ==(ReadOnlyTensorSpan<T> left, ReadOnlyTensorSpan<T> right);
    public static bool operator !=(ReadOnlyTensorSpan<T> left, ReadOnlyTensorSpan<T> right);

    public static explicit operator ReadOnlyTensorSpan<T>(Array? array);
    public static implicit operator ReadOnlyTensorSpan<T>(T[]? array);

    // Should we have variants for common ranks (consider 2-5):
    //   public static implicit operator ReadOnlyTensorSpan<T>(T[,]? array);

    public void CopyTo(Span<T> destination);
    public void FlattenTo(Span<T> destination);
    public Enumerator GetEnumerator();
    public ref readonly T GetPinnableReference();
    public ReadOnlyTensorSpan<T> Slice(params ReadOnlySpan<NIndex> indexes);
    public ReadOnlyTensorSpan<T> Slice(params ReadOnlySpan<NRange> ranges);
    public bool TryCopyTo(TensorSpan<T> destination);
    public bool TryFlattenTo(Span<T> destination);

    [ObsoleteAttribute("Equals() on ReadOnlyTensorSpan will always throw an exception. Use the equality operator instead.")]
    public override bool Equals(object? obj);

    [ObsoleteAttribute("GetHashCode() on ReadOnlyTensorSpan will always throw an exception.")]
    public override int GetHashCode();

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

Tensor Type

public interface IReadOnlyTensor<TSelf, T> : IEnumerable<T>
    where TSelf : IReadOnlyTensor<TSelf, T>
{
    // Should there be APIs for creating from an array, TensorSpan, etc

    static abstract TSelf Empty { get; }

    bool IsEmpty { get; }
    bool IsPinned { get; }
    nint Length { get; }
    int Rank { get; }

    T this[params ReadOnlySpan<nint> indexes] { get; }
    T this[params ReadOnlySpan<NIndex> indexes] { get; }
    TSelf this[params scoped ReadOnlySpan<NRange> ranges] { get; }

    ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan();
    ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<nint> start);
    ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<NIndex> startIndex);
    ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<NRange> ranges);

    void CopyTo(TensorSpan<T> destination);
    void FlattenTo(TensorSpan<T> destination);

    // These are not properties so that structs can implement the interface without allocating:
    void GetLengths(Span<nint> destination);
    void GetStrides(Span<nint> destination);

    ref readonly T GetPinnableReference();
    TSelf Slice(params scoped ReadOnlySpan<nint> start);
    TSelf Slice(params scoped ReadOnlySpan<NIndex> startIndices);
    TSelf Slice(params scoped ReadOnlySpan<NRange> ranges);
    bool TryCopyTo(TensorSpan<T> destination);
    bool TryFlattenTo(TensorSpan<T> destination);
}

public interface ITensor<TSelf, T> : IReadOnlyTensor<TSelf, T>
    where TSelf : ITensor<TSelf, T>
{
    static TSelf Create(ReadOnlySpan<nint> lengths, bool pinned = false);
    static TSelf Create(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    static TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, bool pinned = false);
    static TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    bool IsReadOnly { get; }

    new T this[params ReadOnlySpan<nint> indexes] { get; set; }
    new T this[params ReadOnlySpan<NIndex> indexes] { get; set; }
    new TSelf this[params scoped ReadOnlySpan<NRange> ranges] { get; set; }

    TensorSpan<T> AsTensorSpan();
    TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<nint> start);
    TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<NIndex> startIndex);
    TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<NRange> ranges);

    void Clear();
    void Fill(T value);
    new ref T GetPinnableReference();
}

public sealed class Tensor<T> : ITensor<Tensor<T>, T>
{
    static TSelf ITensor<T>.Create(ReadOnlySpan<nint> lengths, bool pinned = false);
    static TSelf ITensor<T>.Create(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    static TSelf ITensor<T>.CreateUninitialized(ReadOnlySpan<nint> lengths, bool pinned = false);
    static TSelf ITensor<T>.CreateUninitialized(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    public static Tensor<T> Empty { get; }

    public bool IsEmpty { get; }
    public bool IsPinned { get; }
    public nint Length { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Lengths { get; }
    public ReadOnlySpan<nint> Strides { get; }

    bool ITensor<T>.IsReadOnly { get; }

    public ref T this[params ReadOnlySpan<nint> indexes] { get; }
    public ref T this[params ReadOnlySpan<NIndex> indexes] { get; }
    public Tensor<T> this[params ReadOnlySpan<NRange> ranges] { get; set; }

    T IReadOnlyTensor<T>.this[params ReadOnlySpan<nint> indexes] { get; }
    T IReadOnlyTensor<T>.this[params ReadOnlySpan<NIndex> indexes] { get; }
    Tensor<T> IReadOnlyTensor<T>.this[params ReadOnlySpan<NRange> ranges] { get; set; }

    public static implicit operator TensorSpan<T>(Tensor<T> value);
    public static implicit operator ReadOnlyTensorSpan<T>(Tensor<T> value);

    public TensorSpan<T> AsTensorSpan();
    public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<nint> start);
    public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<NIndex> startIndex);
    public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<NRange> ranges);

    public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan();
    public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<nint> start);
    public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<NIndex> startIndex);
    public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<NRange> ranges);

    public void Clear();
    public void CopyTo(TensorSpan<T> destination);
    public void FlattenTo(TensorSpan<T> destination);
    public void Fill(T value);
    public Enumerator GetEnumerator();
    public ref T GetPinnableReference();
    public Tensor<T> Slice(params ReadOnlySpan<nint> start);
    public Tensor<T> Slice(params ReadOnlySpan<NIndex> startIndices);
    public Tensor<T> Slice(params ReadOnlySpan<NRange> ranges);
    public bool TryCopyTo(TensorSpan<T> destination);
    public bool TryFlattenTo(TensorSpan<T> destination);

    void IReadOnlyTensor<T>.GetLengths(Span<nint> destination);
    void IReadOnlyTensor<T>.GetStrides(Span<nint> destination);
    ref readonly T IReadOnlyTensor<T>.GetPinnableReference();

    // The behavior of Equals, GetHashCode, and ToString needs to be determined

    // For ToString, is the following sufficient to help mitigate potential issues:
    //   public string ToString(params ReadOnlySpan<nint> maximumLengths);

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

public static partial class Tensor
{
    public static TSelf Create(ReadOnlySpan<nint> lengths, bool pinned = false);
    public static TSelf Create(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    public static TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, bool pinned = false);
    public static TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

    // Effectively mirror the TensorPrimitives surface area. Following the general pattern
    // where we will return a new tensor and an overload that takes in the destination explicitly.
    //   public static Tensor<T> Add<T>(Tensor<T> left, Tensor<T> right)
    //       where T : IAdditionOperators<T, T, T>;
    //   public static void Add<T>(TensorSpan<T> left, ReadOnlyTensorSpan<T> right)
    //       where T : IAdditionOperators<T, T, T>;
    //   public static TTensor Add<TTensor, T>(TensorSpan<T> left, ReadOnlyTensorSpan<T> right)
    //       where T : IAdditionOperators<T, T, T>
    //       where TTensor : ITensor<TTensor, T>;

    // Consider whether we should have one named `*InPlace`, for easier chaining, such as:
    //   public static Span<T> AddInPlace<T>(TensorSpan<T> left, ReadOnlyTensorSpan<T> right)
    //       where T : IAdditionOperators<T, T, T>;

    // The language ideally has extension operators such that we can define `operator +` instead of `Add`
    // Without this support, we end up having to have `TensorNumber<T>`, `TensorBinaryInteger<T>`, etc
    // as we otherwise cannot correctly expose the operators based on what `T` supports

    // APIs that would return `bool` like `GreaterThan` are split into 3. Following the general
    // pattern already established for our SIMD vector types.
    // * public static ITensor<T> GreaterThan(ITensor<T> x, ITensor<T> y);
    // * public static bool GreaterThanAny<T>(ITensor<T> x, ITensor<T> y);
    // * public static bool GreaterThanAll<T>(ITensor<T> x, ITensor<T> y);
}

Additional Notes

We know we need to support having the backing memory allocated in native so that large array allocations can work. Our current thinking is that this will be a distinct/separate type (possibly named NativeTensor<T>), but are still finalizing the details on how lifetimes and ownership will work correctly.

By default, operations will allocate and return a Tensor<T> from operations. As part of the above support for NativeTensor<T> we are looking at ways the users can override this default behavior, it will likely come with some ThreadStatic member that allows users to provide a custom allocator.

[bartonjs]

Video

• Added NativeIndex..ctor(Index), NativeIndex.ToIndex, and similar to NativeRange.
• Added a Span-accepting overload to SpanND to mirror the Array one, and similar to ReadOnlySpanND
• We're not sure if System is the right namespace for NativeIndex and NativeRange, but want more people in the room to discuss.
• The name SpanND did not meet with universal love and harmony, needs more people to discuss.
• The namespaces are also something to further discuss
• We added indexers on SpanND/ROSpanND to accept params NativeIndex
• We added (Try)FlattenTo to copy a SpanND into a Span.

namespace System;

public readonly struct NativeIndex : IEquatable<NativeIndex>
{
    public NativeIndex(nint value, bool fromEnd = false);
    public NativeIndex(Index value);

    public static NativeIndex End { get; }
    public static NativeIndex Start { get; }

    public bool IsFromEnd { get; }
    public nint Value { get; }

    public static implicit operator NativeIndex(nint value);
    public static implicit operator NativeIndex(Index value);

    public static explicit operator Index(NativeIndex value);
    public static explicit operator checked Index(NativeIndex value);

    public Index ToIndex();

    public static NativeIndex FromEnd(nint value);
    public static NativeIndex FromStart(nint value);

    public nint GetOffset(nint length);
}

public readonly struct NativeRange : IEquatable<NativeRange>
{
    public NativeRange(NativeIndex start, NativeIndex end);
    public NativeRange(Range range);

    public static NativeRange All { get; }

    public NativeIndex End { get; }
    public NativeIndex Start { get; }

    public static explicit operator Range(NativeRange value);
    public static explicit operator checked Range(NativeRange value);

    public Range ToRange();

    public static NativeRange EndAt(NativeIndex end);
    public (nint Offset, nint Length) GetOffsetAndLength(nint length);
    public static NativeRange StartAt(NativeIndex start);
}

namespace System;

public readonly ref struct SpanND<T>
{
    public SpanND(void* pointer, ReadOnlySpan<nint> lengths);
    public SpanND(void* pointer, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public SpanND(Span<T> span);
    public SpanND(Span<T> span, ReadOnlySpan<nint> offsets, ReadOnlySpan<nint> lengths);

    public SpanND(Array? array);
    public SpanND(Array? array, ReadOnlySpan<nint> offsets, ReadOnlySpan<nint> lengths);

    // Do we want overloads for common dimensions like `T[,]` and `T[,,]`?

    public static SpanND<T> Empty { get; }

    public bool IsEmpty { get; }
    public ReadOnlySpan<nint> Lengths { get; }
    public nint LinearLength { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref T this[params ReadOnlySpan<nint> indices] { get; }
    public ref T this[params ReadOnlySpan<NativeIndex> indices] { get; }

    public static bool operator ==(SpanND<T> left, SpanND<T> right);
    public static bool operator !=(SpanND<T> left, SpanND<T> right);

    public static implicit operator SpanND<T>(Array array);
    public static implicit operator SpanND<T>(Span<T> span);
    public static implicit operator ReadOnlySpanND<T>(SpanND<T> span);

    public void Clear();

    public void CopyTo(SpanND<T> destination);
    public bool TryCopyTo(SpanND<T> destination);
    
    public void FlattenTo(Span<T> destination);
    public bool TryFlattenTo(Span<T> destination);
    
    public void Fill(T value);
    public Enumerator GetEnumerator();
    public ref T GetPinnableReference();
    public SpanND<T> Slice(params ReadOnlySpan<NativeIndex> indices);
    public SpanND<T> Slice(params ReadOnlySpan<NativeRange> ranges);

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

public readonly ref struct ReadOnlySpanND<T>
{
    public ReadOnlySpanND(void* pointer, ReadOnlySpan<nint> lengths);
    public ReadOnlySpanND(void* pointer, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

    public ReadOnlySpanND(ReadOnlySpan<T> span);
    public ReadOnlySpanND(ReadOnlySpan<T> span, ReadOnlySpan<nint> starts, ReadOnlySpan<nint> lengths);

    public ReadOnlySpanND(Array? array);
    public ReadOnlySpanND(Array? array, ReadOnlySpan<nint> starts, ReadOnlySpan<nint> lengths);

    // Do we want overloads for common dimensions like `T[,]` and `T[,,]`?

    public static ReadOnlySpanND<T> Empty { get; }

    public bool IsEmpty { get; }
    public ReadOnlySpan<nint> Lengths { get; }
    public nint LinearLength { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref readonly T this[params ReadOnlySpan<nint> indices] { get; }
    public ref readonly T this[params ReadOnlySpan<NativeIndex> indices] { get; }

    public static bool operator ==(ReadOnlySpanND<T> left, ReadOnlySpanND<T> right);
    public static bool operator !=(ReadOnlySpanND<T> left, ReadOnlySpanND<T> right);

    public static implicit operator ReadOnlySpanND<T>(Array array);

    public void CopyTo(SpanND<T> destination);
    public bool TryCopyTo(SpanND<T> destination);

    public void FlattenTo(Span<T> destination);
    public bool TryFlattenTo(Span<T> destination);
    
    public Enumerator GetEnumerator();
    public ref readonly T GetPinnableReference();
    public ReadOnlySpanND<T> Slice(params ReadOnlySpan<NativeIndex> indices);
    public ReadOnlySpanND<T> Slice(params ReadOnlySpan<NativeRange> ranges);

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

[Clockwork-Muse]

... Isn't the ability to return buffers backed by a native buffer part of the point of MemoryManager?

[Neme12]

I really, really don't think we should expose NativeIndex and NativeRange (and potentially NativeSpan for that matter). Instead, the existing types should be updated to accept nint and also to be able to return nint using the new language feature that is coming of BinaryCompatOnly/overload resolution preference, which should allow overloading by return type. This would be a source breaking change, but not a binary breaking change since existing binaries would call the original methods. Otherwise, it will be really weird why there's Span and NativeSpan but no NativeSpanND

[Neme12]

If we're going with abbreviations in the name (i.e. not something like SpanMultiDimensional), I would definitely go with SpanMD over SpanND. I can immediately understand what the MD in SpanMD means, but I would have no idea what a SpanND is - I would have to look at the documentation to understand that it's a multi-dimensional span... and even after I look at the documentation and learn that, I would still have no idea what the ND stands for (and I would probably think the N stands for native, so native something), whereas SpanMD is immediately clear.

[Neme12]

As far as the namespace goes, System.Numerics might be better than System (for SpanMD, not for NativeIndex & NativeRange), since I imagine this will be a lot less common than regular Span, and only used in specialized domains related to numerics.

[Neme12]

public SpanND(Span<T> span, ReadOnlySpan<nint> offsets, ReadOnlySpan<nint> lengths);

How would this constructor work? It mirrors the Array one, but the array one calculates the strides based on the array, and there are no strides here. How are the strides determined here? Or is this more like the void* one where the span is from the first element, as opposed to the whole thing we're slicing? It's really unclear.

[Neme12]

I would also add the equivalent of the void* constructor but with a ref instead of a pointer. But that would be more appropriate on MemoryMarshal next to CreateSpan, so:

namespace System.Runtime.InteropServices;

public static class MemoryMarshal
{
    // Existing method:
    public static Span<T> CreateSpan<T>(ref T reference, int length);

    // New methods:
    public static SpanND<T> CreateSpanND<T>(ref T reference, ReadOnlySpan<nint> lengths);
    public static SpanND<T> CreateSpanND<T>(ref T reference, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
}

(I'll start putting everything in one comment so that people don't have many pings).

---

Also, why do the void* constructors take void* instead of T*?

---

And is NativeRange missing an implicit conversion from Range, similar to how NativeIndex has an implicit conversion from Index? And does NativeIndex need an implicit conversion from int, or will it "just work" and the int will be converted to nint and then the implicit conversion from nint will be called?

I would also expect the ToIndex and ToRange methods to be checked and throw if it cannot fit, I hope that will be the case.

Note: In the API review, Jeremy mentioned that there's a guideline that there should be named methods in addition to (or instead of) conversion operators because operators aren't discoverable. This isn't really the case anymore, IntelliSense now shows operators:

And when you hit tab, it conveniently replaces i. with ((int)i). I'm just mentioning that since it might change the calculus of when named methods should be added in addition to operators, although named methods might still be nice because it reads a little nicer, especially when you want the conversion to be checked and you'd have to surround the cast with a checked expression.

[tannergooding]

Instead, the existing types should be updated to accept nint and also to be able to return nint using the new language feature that is coming of BinaryCompatOnly/overload resolution preference, which should allow overloading by return type. This would be a source breaking change, but not a binary breaking change since existing binaries would call the original methods. Otherwise, it will be really weird why there's Span and NativeSpan but no NativeSpanND

This is not going to happen. It has been suggested and discussed in depth for years and it is "too breaking". There is simply too much existing code that assumes the lengths can only be int in length and it would lead to subtle bugs, corner cases, and a non-trivial perf pessimization to almost all existing code.

[tannergooding]

A lot of the other feedback given was already covered in API review and the proposal will be updated accordingly before it goes back to review.

Most of the questions of "why" can be answered by prior precedent, convenience, or it being historical artifacts based on the point in time the prior precedent was implemented. .NET then tries to stay overall consistent unless there is sufficient reason to deviate.

For example, we use void* because T* used to be completely illegal. It was then relaxed only for where T : unmanaged and so wouldn't have been usable for some Span<T> constructor given T is unconstrained. It was then relaxed again even more recently to allow any T and to warn if T was unconstrained. void* avoids the need to suppress the diagnostic and makes several other concepts simpler, so it ends up being more than sufficient and overall consistent with prior art.

Cases where we have opted to explicitly deviate include places that don't have more widespread prior art, like Array using LongLength and GetLength and it being well known that these are historical points of confusion for users that lead them into potential pits of failure and where slightly different names and accounting for modern features gives us something overall better.

[neon-sunset]

Somewhat unrelated but is there a reason for SpanND over NDSpan (like numpy's ndarray)? 😄

[jubruckne]

What about HyperSpan since it's a view into a hypercube, basically...

NativeIndex / NativeRange... I feel that "Native" implies to me that it's doing something with unmanaged memory (as NativeTensor is envisioned to do).

As for FlattenTo... What is this method envisioned to be used for? If it's for interop or persistence, it would probably need a parameter to specify the target layout (row/column major).

[tannergooding]

Somewhat unrelated but is there a reason for SpanND over NDSpan (like numpy's ndarray)?

Also covered in the review. The general premise is that the natural name for a type of known size would be Span2D, Span3D, etc.. It then becomes consistent to use SpanND to represent a type of arbitrary dimensionality. N was picked because that is the common way to refer to this data, it is n-dimensional, and many people preferred it over MD (hence also why languages like python have choosen ndarray, as nd is a very common term used in the domain space).

NativeIndex / NativeRange... I feel that "Native" implies to me that it's doing something with unmanaged memory (as NativeTensor is envisioned to do).

Native has many different meanings in many different contexts, but historically in .NET it has meant platform-sized. For example, nint is named such because it is a native integer which corresponds to the official IL name for the type going back to .NET Framework v1.0. Unmanaged is typically the term used to mean things that are done in native code or which are compatible with native code or otherwise manually managed/handled, as they are not managed by the GC or runtime.

What is this method envisioned to be used for? If it's for interop or persistence, it would probably need a parameter to specify the target layout (row/column major).

It is a common API for libraries providing such functionality and is simply meant to get a linear view of the data via a copy operation (as compared to a reshape operation that may make data access effectively random).

You cannot trivially specify target layout via parameterization as it is not as simple as "row vs column major", especially as you get to higher dimensionality. As per API review, this would be done via the explicit Reshape APIs that are will be covered in a future API review as we knew we couldn't get to everything in 2 hours. Thus, you would tensor.Reshape(...).FlattenTo(...) (or another API like Transpose if you wanted one of the well defined Reshape operations) if you needed it stored in a particular format.

Tensors themselves will stored "row-major" as that is how .NET is explicitly documented to work and how most other languages/runtimes (and physical memory) actually work. Which is to say given an index [a, b, c, d]. It is a-major and d-minor, effectively ordered from the largest factor first to the smallest factor of the index last. Rehsape operations will allow you to create alternative views without copying such that d may not have a stride of 1 and this will allow you to, for example, get a column-major view of a row-major stored dataset without copy, but whether that is better or worse is algorithm dependent since random memory access may be worse overall for performance.

[Neme12]

If there was an implicit conversion from an index to a range, I don't think it should create a range of length one, but instead a range from the specified index to the end, because that's how span's Slice behaves when you only pass an index and not a range.

[DaZombieKiller]

To throw a name suggestion out there, one that comes to mind is RankSpan<T>. Rank is much shorter than Dimension/Dimensional while still carrying the same meaning and not being an abbreviation.

---

public SpanND<T> Slice(params ReadOnlySpan<NativeIndex> indices);
public SpanND<T> Slice(params ReadOnlySpan<NativeRange> ranges);

Should there be an additional params ReadOnlySpan<nint> overload of Slice for parity with the indexer?

---

// Do we want `Array ToArray()` to mirror Span<T>?

Judging by the return type, I assume this would allocate a multi-dimensional array? Would it make sense to also have T[] ToFlattenedArray() or T[] ToLinearArray()?

[frankbuckley]

• The name SpanND did not meet with universal love and harmony, needs more people to discuss.

Why not TensorSpan<T> and ReadOnlyTensorSpan<T> as in the design doc?

(Though MatrixSpan<T> might be preferable to Span2D<T>)

[colejohnson66]

I'm partial to [ReadOnly]SpanND simply because I'd expect I could use them for normal access of a 2D array, not just tensors/matrices. However, while I understand the intent behind the name (a span for an "n" dimensional array) and how it would follow a convention ("span 2D" -> "span ND"), I'm in agreement with the API review comment: the name isn't good, but not necessarily for the same reasons. NDSpan feels more natural at first glance. While #DSpan type names wouldn't be allowed, NDSpan would.

Ultimately, while I agree with @tannergooding that the name should be short, "ND" is too short and unclear IMO. I'd suggest either [ReadOnly]NDimSpan or [ReadOnly]SpanNDim. That would clarify their purpose a bit better.

[tannergooding]

To throw a name suggestion out there, one that comes to mind is RankSpan

RankSpan is too confusing imo. It requires users to be familiar with the term rank, which I'd guess is even less familiar to users and may be confused for other terminology. Additionally, users may think that it is effectively similar to a custom type Span<Rank>.

Why not TensorSpan and ReadOnlyTensorSpan as in the

We determined these types should not be restricted to only Tensor<T> and they should be general purpose span types.

(Though MatrixSpan might be preferable to Span2D)

MatrixSpan has similar issues as detailed above. Many users may assume it is a specialized form of Span<Matrix> and that itself may lead to assumptions around dimensionality as you typically consider it scalar (0D), vector (1D), matrix (2D), ... where Tensor is the domain specific term encompassing all of them

---

It's worth explicitly noting that one of the reasons that ND was preferred by almost everyone that took an initial look is that its the almost ubiquitous terminology used in this domain and is the common term used in other languages (like python and ndarray).

This isn't some arbitrary thing that .NET would be opting to do that differs from the rest of the broader ecosystem, it would only differ initially from the more standard .NET convention. However, .NET has often used abbreviations where it does make sense, especially for more core or primitive types (it is Int32 not Integer32) and there are places we've been more verbose that people have voiced opinions around (ReadOnlySpan could have probably been ROSpan given its prevalence).

I disagree that ND is too unclear and rather that its simply a term one needs to learn when working in the broader context of ML, AI, Intelligent Apps, and the types of programming languages, samples, etc that get used in that space. I expect that not using ND will cause overall more long term confusion, especially for people who are primarily .NET developers but who want to interact with other prominent libraries (TorchSharp, TensorFlow.NET, ONNYX, etc) and who may be following samples from other languages (particularly python/numpy). -- That is to say, us not using nd doesn't fix the problem of people not familiar with the domain space needing to learn what nd means, it is going to be front and center regardless.

[bartonjs]

Video

• After a very long debate on names, we ended up with NativeIndex => NIndex and NativeRange => NRange
• After a similarly long debate, we think SpanND => TensorSpan
• NIndex/NRange go to System.Buffers
• NIndex/NRange are probably now considered complete
• For pairing To-prefixed methods with explicit conversions, the type-only name (ToIndex) means checked, and the alternative named one gets the suffix -Unchecked (ToIndexUnchecked)
• TensorSpan probably goes to System.Numerics.Tensors
• TensorSpan did not get reviewed today other than its name and namespace, so this issue will need a round 3.

namespace System.Buffers;

public readonly struct NIndex : IEquatable<NIndex>
{
    public NIndex(nint value, bool fromEnd = false);
    public NIndex(Index value);

    public static NIndex End { get; }
    public static NIndex Start { get; }

    public bool IsFromEnd { get; }
    public nint Value { get; }

    public static implicit operator NIndex(nint value);
    public static implicit operator NIndex(Index value);

    public static explicit operator Index(NIndex value);
    public static explicit operator checked Index(NIndex value);

    public static NIndex FromEnd(nint value);
    public static NIndex FromStart(nint value);

    // Should this always do checked or unchecked, how do users get the other?
    public static Index ToIndex();
    public static Index ToIndexUnchecked();

    public nint GetOffset(nint length);

    // IEquatable<NIndex>
    public bool Equals(NIndex other);
}

public readonly struct NRange : IEquatable<NRange>
{
    public NRange(NIndex start, NIndex end);
    public NRange(Range value);

    public static NRange All { get; }

    public NIndex End { get; }
    public NIndex Start { get; }

    public static explicit operator Range(NRange value);
    public static explicit operator checked Range(NRange value);

    public static NRange EndAt(NIndex end);
    public (nint Offset, nint Length) GetOffsetAndLength(nint length);
    public static NRange StartAt(NIndex start);

    // Should this always do checked or unchecked, how do users get the other?
    public static Range ToRange();
    public static Range ToRangeUnchecked();

    // IEquatable<NRange>
    public bool Equals(NRange other);
}

namespace System.Numerics.Tensors;

public ref struct TensorSpan<T>
{
    // Should this be in the shape of the following instead:
    //   public SpanND(void* pointer, nint length, ReadOnlySpan<nint> shape);
    //   public SpanND(void* pointer, nint length, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public SpanND(void* pointer, ReadOnlySpan<nint> shape);
    public SpanND(void* pointer, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public SpanND(Span<T> span);
    public SpanND(Span<T> span, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public SpanND(T[]? array);
    public SpanND(T[]? array, int start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    public SpanND(T[]? array, Index startIndex, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    // Should start just be `nint` and represent the linear index?
    public SpanND(Array? array);
    public SpanND(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    public SpanND(Array? array, ReadOnlySpan<Index> startIndex, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    // Should we have variants for common ranks (consider 2-5):
    //   public SpanND(T[,]? array);
    //   public SpanND(T[,]? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    //   public SpanND(T[,]? array, ReadOnlySpan<Index> startIndex, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public static SpanND<T> Empty { get; }

    public bool IsEmpty { get; }
    public nint Length { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Shape { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref T this[params scoped ReadOnlySpan<nint> indices] { get; }
    public ref T this[params scoped ReadOnlySpan<NativeIndex> indices] { get; }
    public SpanND<T> this[params scoped ReadOnlySpan<NativeRange> ranges] { get; set; }

    // These work like Span<T>, comparing the byref, shape, and strides, not element-wise
    public static bool operator ==(SpanND<T> left, SpanND<T> right);
    public static bool operator !=(SpanND<T> left, SpanND<T> right);

    public static explicit operator SpanND<T>(Array? array);
    public static implicit operator SpanND<T>(T[]? array);

    // Should we have variants for common ranks (consider 2-5):
    //   public static implicit operator SpanND<T>(T[,]? array);

    public static implicit operator ReadOnlySpanND<T>(SpanND<T> span);

    public void Clear();
    public void CopyTo(SpanND<T> destination);
    public void Fill(T value);
    public void FlattenTo(SpanND<T> destination);
    public Enumerator GetEnumerator();
    public ref T GetPinnableReference();
    public SpanND<T> Slice(params ReadOnlySpan<NativeIndex> indices);
    public SpanND<T> Slice(params ReadOnlySpan<NativeRange> ranges);
    public bool TryCopyTo(SpanND<T> destination);
    public bool TryFlattenTo(SpanND<T> destination);

    [ObsoleteAttribute("Equals() on SpanND will always throw an exception. Use the equality operator instead.")]
    public override bool Equals(object? obj);

    [ObsoleteAttribute("GetHashCode() on SpanND will always throw an exception.")]
    public override int GetHashCode();

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

public ref struct ReadOnlySpanND<T>
{
    // Should this be in the shape of the following instead:
    //   public ReadOnlySpanND(void* pointer, nint length, ReadOnlySpan<nint> shape);
    //   public ReadOnlySpanND(void* pointer, nint length, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public ReadOnlySpanND(void* pointer, ReadOnlySpan<nint> shape);
    public ReadOnlySpanND(void* pointer, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public ReadOnlySpanND(T[]? array);
    public ReadOnlySpanND(T[]? array, int start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    public ReadOnlySpanND(T[]? array, Index startIndex, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    // Should start just be `nint` and represent the linear index?
    public ReadOnlySpanND(Array? array);
    public ReadOnlySpanND(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    public ReadOnlySpanND(Array? array, ReadOnlySpan<Index> startIndices, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    // Should we have variants for common ranks (consider 2-5):
    //   public ReadOnlySpanND(T[,]? array);
    //   public ReadOnlySpanND(T[,]? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);
    //   public ReadOnlySpanND(T[,]? array, ReadOnlySpan<Index> startIndices, ReadOnlySpan<nint> shape, ReadOnlySpan<nint> strides);

    public static ReadOnlySpanND<T> Empty { get; }

    public bool IsEmpty { get; }
    public nint Length { get; }
    public int Rank { get; }
    public ReadOnlySpan<nint> Shape { get; }
    public ReadOnlySpan<nint> Strides { get; }

    public ref readonly T this[params scoped ReadOnlySpan<nint> indices] { get; }
    public ref readonly T this[params scoped ReadOnlySpan<NativeIndex> indices] { get; }
    public ReadOnlySpanND<T> this[params scoped ReadOnlySpan<NativeRange> ranges] { get; set; }

    // These work like ReadOnlySpan<T>, comparing the byref, shape, and strides, not element-wise
    public static bool operator ==(ReadOnlySpanND<T> left, ReadOnlySpanND<T> right);
    public static bool operator !=(ReadOnlySpanND<T> left, ReadOnlySpanND<T> right);

    public static explicit operator ReadOnlySpanND<T>(Array? array);
    public static implicit operator ReadOnlySpanND<T>(T[]? array);

    // Should we have variants for common ranks (consider 2-5):
    //   public static implicit operator ReadOnlySpanND<T>(T[,]? array);

    public void CopyTo(ReadOnlySpanND<T> destination);
    public void FlattenTo(ReadOnlySpanND<T> destination);
    public Enumerator GetEnumerator();
    public ref readonly T GetPinnableReference();
    public ReadOnlySpanND<T> Slice(params ReadOnlySpan<NativeIndex> indices);
    public ReadOnlySpanND<T> Slice(params ReadOnlySpan<NativeRange> ranges);
    public bool TryCopyTo(SpanND<T> destination);
    public bool TryFlattenTo(SpanND<T> destination);

    [ObsoleteAttribute("Equals() on ReadOnlySpanND will always throw an exception. Use the equality operator instead.")]
    public override bool Equals(object? obj);

    [ObsoleteAttribute("GetHashCode() on ReadOnlySpanND will always throw an exception.")]
    public override int GetHashCode();

    // Do we want `Array ToArray()` to mirror Span<T>?

    public ref struct Enumerator
    {
        public ref readonly T Current { get; }
        public bool MoveNext();
    }
}

[tannergooding]

It really feels to me like the Tensor class is just a workaround against having to do a few runtime changes to improve multi-dimensional arrays. But why not just improve the runtime, and everyone will benefit?

Because "just" is massively complex, potentially breaking, error prone, and doesn't really solve the problem across the considerations that are actually required.

As much as it would be nice if we could "just" fix Span or T[] to use nint, that's basically impossible at this point for all the reasons I detailed in the actual API review. As a summary, it is not pay for play and would regress perf for nearly every app in existence for something which is almost never needed. It introduces new arbitrary failure points for everything that hasn't updated to actually be nint aware yet (with no way to determine what is actually going to work or not).

You can't really extend System.Array for the same reasons, but there are even more complexities involved there that make this difficult. There's also the reasons I've gotten into on other threads that specifically pertain to doing large (85kb or greater) or huge (megabyte or larger) allocations in general and how those need to be very carefully handled/considered.

A new type and interface is the simplest/cleanest way to do this. The term tensor is being used because that's more consistent with other ecosystems and the scenarios where you typically need to work with such data. It not being in System also matches how often .NET developers (across the entire ecosystem) are going to use/need such types. There are specific apps, libraries and scenarios that will use these tons and so its beneficial to provide. But the vast majority of applications will never even know they exist (much as multi-dimensional arrays already are in .NET).

[Neme12]

Because "just" is massively complex, potentially breaking, error prone, and doesn't really solve the problem across the considerations that are actually required.

How would the idea of an intermediate base class between all arrays and System.Array be breaking? Such a type doesn't currently exist so noone is using it, so how could anyone be broken? This should be a much easier effort than changing Span to use nint, which would break every for loop that iterates over a span. This should break nobody, unless I'm missing something? And it's not like the idea that arrays would magically derive from something would be a whole new concept for the runtime - the runtime already does it with System.Array, and a whole bunch of interfaces as well, so it shouldn't be a huge amount of work to do it with an additional intermediate class as well.

I don't know why it would be error-prone or massively complex either, unless the runtime code that handles arrays deriving from System.Array and from IList<T>, IReadOnlyList<T> etc is written in such a bad way that it cannot be extended with just one additional class. Even though it was extended when IReadOnlyList<T> came out.

Anyway, I guess I said enough for this argument, thanks for the discussion, I just hope that the idea that such a change would be massively complex is really verified with the runtime folks. (Unless you work on the runtime as well in which case I apologize for not trusting your judgment enough, it just says you're on the .NET Libraries team in your profile. and from what I've seen in API review videos, runtime costs of various things are always determined by someone like Jan - e.g. "let's add these methods to the class itself unless Jan yells at us" 😄).

[tannergooding]

Such a thing doesn't provide any benefit and makes the general type system of .NET more complex with more considerations.

System.Array is itself limited to int in any single dimension, so you can't change it to (or have a derived type) now support nint without all the same considerations that Span<T> requires.

The runtime synthesizing interfaces and types behind the scenes requires languages to understand this and to specially handle that premise, that has cost. It also impacts the cost for dynamic type checks, interface casts, and general complexity of the ecosystem elsewhere. -- There's also, notably, a difference between a new interface and a new concrete base type in terms of considerations/complexity there.

Unless you work on the runtime as well in which case I apologize for not trusting your judgment enough, it just says you're on the .NET Libraries team in your profile. and from what I've seen in API review videos, runtime costs of various things are always determined by someone like Jan

I'm on the libraries team, but due to my areas of ownership in the libraries I'm fairly familiar with the runtime side of things and regularly contribute to the JIT (that's needed to provide the SIMD support for intrinsics or to version the primitive types for example). I do defer to people like Jan (the runtime architecture) and people who are actually on the JIT team in most cases, but this is one of the areas where it's been pretty heavily discussed and so I believe I'm in the generally correct area with the summary I've given here

[Neme12]

Fair enough. It just seems to me like if .NET were designed today, after generics and everything, multi-dimensional arrays would have been designed in a better way, with all of these use cases in mind, and I'm pretty sure someone even mentioned this in this API review - as deriving from some sort of MultidimensionalArray<T>, so that's why I'm more in favor of going that route.

[tannergooding]

Yes, I expect that if we were to design .NET from the ground up there would be a number of changes made to how things were defined/exposed.

But, we're on a 20+ year old ecosystem and so backwards compatiblity and not resetting existing source code and/or binaries fully to 0 is important.

[Neme12]

But, we're on a 20+ year old ecosystem and so backwards compatiblity and not resetting existing source code and/or binaries fully to 0 is important.

Yes, that's the argument against changing Span<T> from int to nint. I'm not making that argument though. Or is there an issue with my proposal that I missed that would break backwards compatibility in any way? I'm really confused, because your arguments seem like you're talking about the Span thing, not what I'm talking about, arrays.

[Neme12]

I mentioned Jeremy talking about the Span thing simply because the same arguments would apply in favor of it. In case of Span/nint, there are also strong arguments against it - it would break every current for loop that iterates over a span. In this case of arrays, such argument doesn't apply.

[tannergooding]

Arrays are in the same bucket with largely the same considerations.

Array.Length is exposed by an int, neither a new base type nor a new derived type could change that to nint. While the JIT could effectively devirtualize for T[] (this is essentially what ldlen IL isntruction does) since it knows is single-dimensional and therefore cannot overflow, the access of the same property for non szarray or on System.Array itself (where it can't be effectively devirtualized) requires a checked cast, which can throw and adds implicit overhead.

Array is then likewise a GC tracked type, and there are negative implications from having huge allocations, especially if they are known to be used for scenarios like temporaries and therefore are likely short lived. Adding new members can then increase the costs for dynamic type checks and some casts, as there's more types that need to be checked.

You do still have to consider the breakage of existing loops since they may miss indices or not be able to represent all indices. We already need to be considerate (due to muscle memory and other bits) that someone may write for (int i = 0; i < someNInt; i++) and that this is a bug, where i can overflow to a negative value and cause a AV or IndexOutOfRange exception.

---

Beyond that, Array also has a lot of APIs that we wouldn't want to expose on Tensor<T>, there's considerations of exposing methods on generic types and what that does to the VM metadata, to the number of generic instantiations and potential specializations needed, and so on.

Having a net new type is an overall "better" direction and allows things to be done in a manner which is more beneficial to the ecosystem long term.

[Neme12]

I guess there was a misunderstanding, I was arguing for multi-dimensional arrays deriving from Array<T>, not for changing arrays to have their length be nint, sorry if I wasn't clear enough. If that's a requirement for Tensor<T>, then yes I agree it can't be done without massive breaks. Although we might still want to think about whether we'll ever need nint-sized single-dimensional arrays as well in the future. If there is such a possibility, maybe this would be better called NArray<T> in such a scenario and single-dimensional native-sized arrays would derive from the same type, having the inheritance hierachy I argued for and the one you yourself said in API review would have been the case if you were designing .NET today with generics in mind. EDIT: Actually no, sorry, you said multi-dimensional arrays would have been called Tensor<T> and would be separate from single-dimensional arrays. Just that multi-dimensional arrays would derive from Tensor<T>, which is similar but not the quite the same thing as what I talked about, only half of it. (https://www.youtube.com/live/AuqUmrNwSLA?si=j9Y0zR5-IOjhiIaE&t=3605).

I do still wonder whether there will ever be a need for single-dimensional nint-sized arrays though and whether there will ever be regret about always using Tensor<T> in such scenarios, even for single-dimensional ones 🤔

[bartonjs]

• TensorSpan: Shape => Lengths
• TensorSpan: Length => FlattenedLength
• TensorSpan..ctor: (void* pointer, ...) => (T* data, nint dataLength, ...) to apply a total range limit on the pointer (effectively being BiggerSpan<T>, but flattened out)
• indices => indexes, to match FDG
• All of the mdArray overloads are cut for now
• ToArray is cut for now
• We had a debate around the defaulted parameters on ITensor.Create
  • whether pinned needed to be that high of a concept: yes
  • whether we wanted to violate the guideline of not defaulting parameters in interfaces: yes, because statics in interfaces are different from "interface methods".

namespace System.Buffers
{
    public readonly struct NIndex : IEquatable<NIndex>
    {
        public NIndex(nint value, bool fromEnd = false);
        public NIndex(Index value);

        public static NIndex End { get; }
        public static NIndex Start { get; }

        public bool IsFromEnd { get; }
        public nint Value { get; }

        public static implicit operator NIndex(nint value);
        public static implicit operator NIndex(Index value);

        public static explicit operator Index(NIndex value);
        public static explicit operator checked Index(NIndex value);

        public static NIndex FromEnd(nint value);
        public static NIndex FromStart(nint value);

        public static Index ToIndex();
        public static Index ToIndexUnchecked();

        public nint GetOffset(nint length);

        // IEquatable<NIndex>
        public bool Equals(NIndex other);
    }

    public readonly struct NRange : IEquatable<NRange>
    {
        public NRange(NIndex start, NIndex end);
        public NRange(Range value);

        public static NRange All { get; }

        public NIndex End { get; }
        public NIndex Start { get; }

        public static explicit operator Range(NRange value);
        public static explicit operator checked Range(NRange value);

        public static NRange EndAt(NIndex end);
        public (nint Offset, nint Length) GetOffsetAndLength(nint length);
        public static NRange StartAt(NIndex start);

        public static Range ToRange();
        public static Range ToRangeUnchecked();

        // IEquatable<NRange>
        public bool Equals(NRange other);
    }
}

namespace System.Numerics.Tensors
{
    public ref struct TensorSpan<T>
    {
        public TensorSpan(T* data, nint dataLength, ReadOnlySpan<nint> lengths);
        public TensorSpan(T* data, nint dataLength, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public TensorSpan(Span<T> span);
        public TensorSpan(Span<T> span, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public TensorSpan(T[]? array);
        public TensorSpan(T[]? array, int start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
        public TensorSpan(T[]? array, Index startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        // Should start just be `nint` and represent the linear index?
        public TensorSpan(Array? array);
        public TensorSpan(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
        public TensorSpan(Array? array, ReadOnlySpan<Index> startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public static TensorSpan<T> Empty { get; }

        public bool IsEmpty { get; }
        public nint FlattenedLength { get; }
        public int Rank { get; }
        public ReadOnlySpan<nint> Lengths { get; }
        public ReadOnlySpan<nint> Strides { get; }

        public ref T this[params scoped ReadOnlySpan<nint> indexes] { get; }
        public ref T this[params scoped ReadOnlySpan<NIndex> indexes] { get; }
        public TensorSpan<T> this[params scoped ReadOnlySpan<NRange> ranges] { get; set; }

        public static bool operator ==(TensorSpan<T> left, TensorSpan<T> right);
        public static bool operator !=(TensorSpan<T> left, TensorSpan<T> right);

        public static explicit operator TensorSpan<T>(Array? array);
        public static implicit operator TensorSpan<T>(T[]? array);

        public static implicit operator ReadOnlyTensorSpan<T>(TensorSpan<T> span);

        public void Clear();
        public void CopyTo(TensorSpan<T> destination);
        public void Fill(T value);
        public void FlattenTo(Span<T> destination);
        public Enumerator GetEnumerator();
        public ref T GetPinnableReference();
        public TensorSpan<T> Slice(params ReadOnlySpan<NIndex> indexes);
        public TensorSpan<T> Slice(params ReadOnlySpan<NRange> ranges);
        public bool TryCopyTo(TensorSpan<T> destination);
        public bool TryFlattenTo(Span<T> destination);

        [EditorBrowsable(EditorBrowsableState.Never)]
        [ObsoleteAttribute("Equals() on TensorSpan will always throw an exception. Use the equality operator instead.")]
        public override bool Equals(object? obj);

        [EditorBrowsable(EditorBrowsableState.Never)]
        [ObsoleteAttribute("GetHashCode() on TensorSpan will always throw an exception.")]
        public override int GetHashCode();

        public ref struct Enumerator
        {
            public ref readonly T Current { get; }
            public bool MoveNext();
        }
    }

    public ref struct ReadOnlyTensorSpan<T>
    {
        public ReadOnlyTensorSpan(T* data, nint dataLength, ReadOnlySpan<nint> lengths);
        public ReadOnlyTensorSpan(T* data, nint dataLength, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public ReadOnlyTensorSpan(ReadOnlySpan<T> span);
        public ReadOnlyTensorSpan(ReadOnlySpan<T> span, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public ReadOnlyTensorSpan(T[]? array);
        public ReadOnlyTensorSpan(T[]? array, int start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
        public ReadOnlyTensorSpan(T[]? array, Index startIndex, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public ReadOnlyTensorSpan(Array? array);
        public ReadOnlyTensorSpan(Array? array, ReadOnlySpan<int> start, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);
        public ReadOnlyTensorSpan(Array? array, ReadOnlySpan<Index> startIndexes, ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides);

        public static ReadOnlyTensorSpan<T> Empty { get; }

        public bool IsEmpty { get; }
        public nint FlattenedLength { get; }
        public int Rank { get; }
        public ReadOnlySpan<nint> Lengths { get; }
        public ReadOnlySpan<nint> Strides { get; }

        public ref readonly T this[params scoped ReadOnlySpan<nint> indexes] { get; }
        public ref readonly T this[params scoped ReadOnlySpan<NIndex> indexes] { get; }
        public ReadOnlyTensorSpan<T> this[params scoped ReadOnlySpan<NRange> ranges] { get; }

        public static bool operator ==(ReadOnlyTensorSpan<T> left, ReadOnlyTensorSpan<T> right);
        public static bool operator !=(ReadOnlyTensorSpan<T> left, ReadOnlyTensorSpan<T> right);

        public static explicit operator ReadOnlyTensorSpan<T>(Array? array);
        public static implicit operator ReadOnlyTensorSpan<T>(T[]? array);

        public void CopyTo(TensorSpan<T> destination);
        public void FlattenTo(Span<T> destination);
        public Enumerator GetEnumerator();
        public ref readonly T GetPinnableReference();
        public ReadOnlyTensorSpan<T> Slice(params ReadOnlySpan<NIndex> indexes);
        public ReadOnlyTensorSpan<T> Slice(params ReadOnlySpan<NRange> ranges);
        public bool TryCopyTo(TensorSpan<T> destination);
        public bool TryFlattenTo(Span<T> destination);

        [EditorBrowsable(EditorBrowsableState.Never)]
        [ObsoleteAttribute("Equals() on ReadOnlyTensorSpan will always throw an exception. Use the equality operator instead.")]
        public override bool Equals(object? obj);

        [EditorBrowsable(EditorBrowsableState.Never)]
        [ObsoleteAttribute("GetHashCode() on ReadOnlyTensorSpan will always throw an exception.")]
        public override int GetHashCode();

        public ref struct Enumerator
        {
            public ref readonly T Current { get; }
            public bool MoveNext();
        }
    }

    public interface IReadOnlyTensor<TSelf, T> : IEnumerable<T>
        where TSelf : IReadOnlyTensor<TSelf, T>
    {
        static abstract TSelf Empty { get; }

        bool IsEmpty { get; }
        bool IsPinned { get; }
        nint Length { get; }
        int Rank { get; }

        T this[params ReadOnlySpan<nint> indexes] { get; }
        T this[params ReadOnlySpan<NIndex> indexes] { get; }
        TSelf this[params scoped ReadOnlySpan<NRange> ranges] { get; }

        ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan();
        ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<nint> start);
        ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<NIndex> startIndex);
        ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params scoped ReadOnlySpan<NRange> range);

        void CopyTo(TensorSpan<T> destination);
        void FlattenTo(Span<T> destination);

        // These are not properties so that structs can implement the interface without allocating:
        void GetLengths(Span<nint> destination);
        void GetStrides(Span<nint> destination);

        ref readonly T GetPinnableReference();
        TSelf Slice(params scoped ReadOnlySpan<nint> start);
        TSelf Slice(params scoped ReadOnlySpan<NIndex> startIndex);
        TSelf Slice(params scoped ReadOnlySpan<NRange> range);
        bool TryCopyTo(TensorSpan<T> destination);
        bool TryFlattenTo(Span<T> destination);
    }

    public interface ITensor<TSelf, T> : IReadOnlyTensor<TSelf, T>
        where TSelf : ITensor<TSelf, T>
    {
        static abstract TSelf Create(ReadOnlySpan<nint> lengths, bool pinned = false);
        static abstract TSelf Create(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

        static abstract TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, bool pinned = false);
        static abstract TSelf CreateUninitialized(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

        bool IsReadOnly { get; }

        new T this[params ReadOnlySpan<nint> indexes] { get; set; }
        new T this[params ReadOnlySpan<NIndex> indexes] { get; set; }
        new TSelf this[params scoped ReadOnlySpan<NRange> ranges] { get; set; }

        TensorSpan<T> AsTensorSpan();
        TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<nint> start);
        TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<NIndex> startIndex);
        TensorSpan<T> AsTensorSpan(params scoped ReadOnlySpan<NRange> range);

        void Clear();
        void Fill(T value);
        new ref T GetPinnableReference();
    }

    public sealed class Tensor<T> : ITensor<Tensor<T>, T>
    {
        static TSelf ITensor<T>.Create(ReadOnlySpan<nint> lengths, bool pinned);
        static TSelf ITensor<T>.Create(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned);

        static TSelf ITensor<T>.CreateUninitialized(ReadOnlySpan<nint> lengths, bool pinned);
        static TSelf ITensor<T>.CreateUninitialized(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned);

        public static Tensor<T> Empty { get; }

        public bool IsEmpty { get; }
        public bool IsPinned { get; }
        public nint Length { get; }
        public int Rank { get; }
        public ReadOnlySpan<nint> Lengths { get; }
        public ReadOnlySpan<nint> Strides { get; }

        bool ITensor<T>.IsReadOnly { get; }

        public ref T this[params ReadOnlySpan<nint> indexes] { get; }
        public ref T this[params ReadOnlySpan<NIndex> indexes] { get; }
        public Tensor<T> this[params ReadOnlySpan<NRange> ranges] { get; set; }

        T IReadOnlyTensor<T>.this[params ReadOnlySpan<nint> indexes] { get; }
        T IReadOnlyTensor<T>.this[params ReadOnlySpan<NIndex> indexes] { get; }
        Tensor<T> IReadOnlyTensor<T>.this[params ReadOnlySpan<NRange> ranges] { get; set; }

        public static implicit operator TensorSpan<T>(Tensor<T> value);
        public static implicit operator ReadOnlyTensorSpan<T>(Tensor<T> value);

        public TensorSpan<T> AsTensorSpan();
        public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<nint> start);
        public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<NIndex> startIndex);
        public TensorSpan<T> AsTensorSpan(params ReadOnlySpan<NRange> ranges);

        public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan();
        public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<nint> start);
        public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<NIndex> startIndex);
        public ReadOnlyTensorSpan<T> AsReadOnlyTensorSpan(params ReadOnlySpan<NRange> ranges);

        public void Clear();
        public void CopyTo(TensorSpan<T> destination);
        public void FlattenTo(TensorSpan<T> destination);
        public void Fill(T value);
        public Enumerator GetEnumerator();
        public ref T GetPinnableReference();
        public Tensor<T> Slice(params ReadOnlySpan<nint> start);
        public Tensor<T> Slice(params ReadOnlySpan<NIndex> startIndexes);
        public Tensor<T> Slice(params ReadOnlySpan<NRange> ranges);
        public bool TryCopyTo(TensorSpan<T> destination);
        public bool TryFlattenTo(TensorSpan<T> destination);

        void IReadOnlyTensor<T>.GetLengths(Span<nint> destination);
        void IReadOnlyTensor<T>.GetStrides(Span<nint> destination);
        ref readonly T IReadOnlyTensor<T>.GetPinnableReference();

        // The behavior of Equals, GetHashCode, and ToString needs to be determined

        // For ToString, is the following sufficient to help mitigate potential issues:
        //   public string ToString(params ReadOnlySpan<nint> maximumLengths);

        public ref struct Enumerator
        {
            public ref readonly T Current { get; }
            public bool MoveNext();
        }
    }

    public static partial class Tensor
    {
        public static Tensor<T> Create<T>(ReadOnlySpan<nint> lengths, bool pinned = false);
        public static Tensor<T> Create<T>(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);

        public static Tensor<T> CreateUninitialized<T>(ReadOnlySpan<nint> lengths, bool pinned = false);
        public static Tensor<T> CreateUninitialized<T>(ReadOnlySpan<nint> lengths, ReadOnlySpan<nint> strides, bool pinned = false);
    }
}

[bartonjs]

The ITensor<T>.Create* methods weren't declared as abstract, but I believe the intent is that they should be. So I patched that into the approved shape just now. (Thanks, @halter73)

[vpenades]

Sorry, I'm late to the party,
@tannergooding I would like you to take a look at my InteropTypes.Tensors

They look a lot like TensorSpan and ReadOnlyTensorSpan with one major difference: the rank is hardcoded, so I have TensorSpan1, TensorSpan2, TensorSpan3, and so on. This approach has a number of advantages:

• the dimensions are stored as struct fields, so the struct only has one Span reference.
• structs with rank 2 and 3 have additional methods specialised for images.

Also, I have some interesting methods there: for example, you can cast a Tensor2 to a Tensor3 and back.

I've been using this library for a while now, and I've been adding more methods as needed, so there's probably more interesting stuff there.

[tannergooding]

Not hardcoding the rank is an explicit design decision and mirrors what most other major tensor frameworks have done.

While there are advantages for having something like a Tensor2D and Tensor3D type, there are also many disadvantages, overall composability issues, etc. We might decide to also expose fixed dimension tensors in the future, but they would be additive on top of the baseline general-purpose functionality being added here.

[vpenades]

Not hardcoding the rank is an explicit design decision and mirrors what most other major tensor frameworks have done.

Yes, I agree, I was not advocating for that, I was just pointing you to yet another tensors library you might pick ideas from,

I do a lot of vision computing with onnxruntime, so these are some of the concepts that I came up with that seems to be useful

Yes, Tensor2D and Tensor3D would be really welcome for image/vision processing.

[gaviny82]

structs with rank 2 and 3 have additional methods specialised for images.

Could we use the new extensions feature in C# 13 for rank-specific operations? For example, matrices and vectors can define the operations as

public explicit extension Matrix<T> for Tensor<T> { ... }
public explicit extension Vector<T> for Tensor<T> { ... }

This avoids defining new types that either derive from Tensor<T> or implemented separately to represent tensors of a particular rank, and it still allows matrices and vectors to be used as tensors as the underlying type is still Tensor<T>.

[marhja]

Also a bit late to the party, but since I haven't heard it mentioned here nor in the API review videos I'll just ask the question - was MultiSpan<T> ever considered when discussing the name for TensorSpan<T>? I liked the original idea (though not the name SpanND<T> in particular) that it didn't include the word tensor since it potentially represents a broader concept.

[tannergooding]

MultiSpan<T> itself has potential confusion in that it could be thought of to be more like Span<Span<T>> or potentially (Span<T>, Span<T>), etc.

Multi by itself isn't sufficient to provide the context and while Tensor isn't the greatest name, it is at least relatively unambiguous and does apply to the general concept (as tensors can represent scalars, vectors, matrices, etc).

[hez2010]

Another reason for using TensorSpan instead of NdSpan/MdSpan is that in machine learning, we have extensive usage to strided buffers, while such a thing doesn't exist in a traditional span of contiguous memory. We are expecting to use TensorSpan to express things like data[..,1..3,1, new NRange(start: 1, step: 2)] without copying the underlying data.
The NRange in the current proposal doesn't have Step which is pretty limited.
For example, in pytorch we are able to transpose a matrix without copying the underlying data, instead, we only need to create a different view. I don't see how we can do this with the current proposal.

[tannergooding]

TensorSpan itself has support for tracking the strides between elements. It is not part of NRange, by design.

Working with strided buffers, however, is notably very expensive and while it works well enough for local experimentation and working with small data sets; it can have significant cost when working with large data sets because it results in effectively random memory access which the CPU or GPU often cannot correctly predict and account for. This is true of all languages that expose the ability to create views in which indices are no longer guaranteed to be linearly sequenced in memory (such as via copy free transpose).