Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor[next]: cleaner toolchain #1537

Merged
merged 38 commits into from
May 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
b0f7e3a
workflowify past linting and args injection
DropD Mar 26, 2024
445b35d
workflowify func -> FOAST
DropD Apr 2, 2024
3561a0c
fix missing attribute rename.
DropD Apr 2, 2024
2842cad
workflowify `FieldOperator.as_program`
DropD Apr 2, 2024
3357d11
make sure scan operator attributes are properly hashed
DropD Apr 3, 2024
3aab3c8
Merge branch 'main' into c20-workflowify-step3
DropD Apr 4, 2024
44b3060
Merge branch 'main' into c20-workflowify-step3
DropD Apr 4, 2024
ebf164c
add closure vars to hash for program definition
DropD Apr 4, 2024
d3a7851
[wip] integrating fieldop workflows
DropD Apr 4, 2024
a99fc48
update foast pretty printer doctest
DropD Apr 5, 2024
c71e0bf
fix code quality issues
DropD Apr 5, 2024
df648e8
reuse content hashing code in ffront.stages
DropD Apr 5, 2024
1f5c336
remove erroneously committed .python-version
DropD Apr 5, 2024
2cab5b7
Merge remote-tracking branch 'upstream/main' into c20-workflowify-step3
DropD Apr 8, 2024
0a54c3d
re-apply Program.itir fix after merge
DropD Apr 9, 2024
e1ed8d2
move backend transforms to `next.backend`
DropD Apr 11, 2024
3e30ca2
add tested toolchain workthrough notebook
DropD Apr 12, 2024
9e50bd8
fix toolchain walkthrough notebook
DropD Apr 12, 2024
678bd67
put default toolchain steps into definitions
DropD Apr 18, 2024
fad81c0
replace content_hash with dedicated cache key gen for ffront stages
DropD Apr 18, 2024
af1a016
add typeignores for hash algorithms
DropD Apr 18, 2024
556e8c5
downgrade singledispatch type hints for py < 310
DropD Apr 19, 2024
fba07f1
docstrings for AST based decorator wrappers
DropD Apr 19, 2024
50bdea2
todos for linting step calls in decorator wrappers
DropD Apr 19, 2024
63121fd
comment first occurrence of backwards compat backend pattern
DropD Apr 19, 2024
04e407b
stages hasher: avoid recursing into non-stage dataclasses
DropD Apr 19, 2024
ba838f1
Merge remote-tracking branch 'upstream/main' into c20-workflowify-step3
DropD Apr 19, 2024
9bb341a
rename `ffront.stages.cache_key` -> `ffront.stages.fingerprint_stage`
DropD Apr 22, 2024
888017d
improve ffront.stage fingerprinting
DropD Apr 22, 2024
345ce8e
update HashlibAlgorithm in eve
DropD Apr 22, 2024
7be23de
remove redundant singledispatch methods
DropD Apr 23, 2024
250b26d
[wip] make toolchains static with args as inputs
DropD Apr 23, 2024
c6458aa
Merge branch 'main' into c21-cleaner-toolchain
DropD Apr 29, 2024
8bb5f66
update and expand toolchain documentation notebooks
DropD Apr 29, 2024
ef80571
update stage fingerprinting tests
DropD Apr 30, 2024
5ae7f90
Merge branch 'main' into c21-cleaner-toolchain
DropD May 3, 2024
85a2d03
Merge branch 'main' into c21-cleaner-toolchain
DropD May 14, 2024
f6291b5
Merge branch 'main' into c21-cleaner-toolchain
DropD May 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions docs/user/next/advanced/HackTheToolchain.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
```python
import dataclasses
import typing

from gt4py import next as gtx
from gt4py.next.otf import workflow
from gt4py import eve
```

<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"><script src="https://spcl.github.io/dace/webclient2/dist/sdfv.js"></script>
<link href="https://spcl.github.io/dace/webclient2/sdfv.css" rel="stylesheet">

## Replace Steps

```python
cached_lowering_toolchain = gtx.backend.DEFAULT_PROG_TRANSFORMS.replace(
past_to_itir=workflow.CachedStep(
step=gtx.ffront.past_to_itir.PastToItirFactory(),
hash_function=eve.utils.content_hash
)
)
```

## Skip Steps / Change Order

```python
gtx.backend.DEFAULT_PROG_TRANSFORMS.step_order
```

['func_to_past',
'past_lint',
'past_inject_args',
'past_transform_args',
'past_to_itir']

```python
@dataclasses.dataclass(frozen=True)
class SkipLinting(gtx.backend.ProgramTransformWorkflow):
@property
def step_order(self):
return [
"func_to_past",
# not running "past_lint"
"past_inject_args",
"past_transform_args",
"past_to_itir",
]

same_steps = dataclasses.asdict(gtx.backend.DEFAULT_PROG_TRANSFORMS)
skip_linting_transforms = SkipLinting(
**same_steps
)
```

## Alternative Factory

```python
class MyCodeGen:
...

class Cpp2BindingsGen:
...

class PureCpp2WorkflowFactory(gtx.program_processors.runners.gtfn.GTFNCompileWorkflowFactory):
translation: workflow.Workflow[
gtx.otf.stages.ProgramCall, gtx.otf.stages.ProgramSource] = MyCodeGen()
bindings: workflow.Workflow[
gtx.otf.stages.ProgramSource, gtx.otf.stages.CompilableSource] = Cpp2BindingsGen()

PureCpp2WorkflowFactory(cmake_build_type=gtx.config.CMAKE_BUILD_TYPE.DEBUG)
```

## Invent new Workflow Types

````mermaid
graph LR

IN_T --> i{{split}} --> A_T --> a{{track_a}} --> B_T --> o{{combine}} --> OUT_T
i --> X_T --> x{{track_x}} --> Y_T --> o


```python
IN_T = typing.TypeVar("IN_T")
A_T = typing.TypeVar("A_T")
B_T = typing.TypeVar("B_T")
X_T = typing.TypeVar("X_T")
Y_T = typing.TypeVar("Y_T")
OUT_T = typing.TypeVar("OUT_T")

@dataclasses.dataclass(frozen=True)
class FullyModularDiamond(
workflow.ChainableWorkflowMixin[IN_T, OUT_T],
workflow.ReplaceEnabledWorkflowMixin[IN_T, OUT_T],
typing.Protocol[IN_T, OUT_T, A_T, B_T, X_T, Y_T]
):
split: workflow.Workflow[IN_T, tuple[A_T, X_T]]
track_a: workflow.Workflow[A_T, B_T]
track_x: workflow.Workflow[X_T, Y_T]
combine: workflow.Workflow[tuple[B_T, Y_T], OUT_T]

def __call__(self, inp: IN_T) -> OUT_T:
a, x = self.split(inp)
b = self.track_a(a)
y = self.track_x(x)
return self.combine((b, y))


@dataclasses.dataclass(frozen=True)
class PartiallyModularDiamond(
workflow.ChainableWorkflowMixin[IN_T, OUT_T],
workflow.ReplaceEnabledWorkflowMixin[IN_T, OUT_T],
typing.Protocol[IN_T, OUT_T, A_T, B_T, X_T, Y_T]
):
track_a: workflow.Workflow[A_T, B_T]
track_x: workflow.Workflow[X_T, Y_T]

def split(inp: IN_T) -> tuple[A_T, X_T]:
...

def combine(b: B_T, y: Y_T) -> OUT_T:
...

def __call__(inp: IN_T) -> OUT_T:
a, x = self.split(inp)
return self.combine(
b=self.track_a(a),
y=self.track_x(x)
)
````
Loading
Loading