RFC-0030: FP8 dtype introduction to PyTorch

[australopitek]

This RFC proposes adding 8-bit floating point data types to PyTorch.

[jakeh-gc]

For completeness, these formats are proposed by Graphcore, AMD, and Qualcomm.

[australopitek]

I'll correct it when more comments are there.

[YinglinSun]

Curious what the progress for fp8 support looks like? Thanks!

[australopitek]

@jakeh-gc ,
Is there any progress in the fp8 datatypes area?

[jakeh-gc]

@australopitek I've been working more on the XLA side. The only activity I've seen in PyTorch was this pytorch/pytorch#97798, which didn't get merged.

[albanD]

Hey!
Wrote an update at pytorch/pytorch#91577 (comment) if you want to comment there!

[timljj]

Hi @australopitek, in your md file, you mentioned that for E5M2 "there are many models that can be trained only with this variant".

May I know what models/type of models you are referring to?

Also, does your statement mean that those models would not be able to be trained with E4M3?

[australopitek]

Hi @timljj ,
my wording was not precise. I wanted to say that some models can be trained with E5M2 without "help" of E4M3, i.e. E5M2 is sufficient (despite being "Applicable mainly for gradients in the backward pass of training").
Such experiments are described in the https://arxiv.org/pdf/1905.12334.pdf - Resnet18, Resnet34, Resnet50, GNMT, Transformer.

[maxpain]

Any updates?

[australopitek]

@maxpain,
float8 dtypes are already delivered to pytorch - pytorch/pytorch#104242 - I guess they will be released in PT2.1
@albanD,
should we commit this RFC for the sake of completeness?

[albanD]

Yes I think this one is good.
Note that this only covers the basic (unscaled) data types and so we only expect "power users" to be able to use this at this point. End-user facing UX are being looked into but not ready right now.