Fix module initialization with other dtypes and simplify module registration #47
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes and improves initializing modules with other parameter types.
This is currently inconsistent with tensor creation ops like
torch.ones
where we use a default parameter for the dtype, meaning it's easier to override at runtime, but we have a fixed default. Both designs have tradeoffs and we need to test what works well in practice. Perhaps we can event find a way to combine both approaches.Type of all modules fixed at creation time
To make container classes like
Sequential
work, we currently fix the dtype on module initialization even for parameterless modules likeSoftmax
:Instead of a generic apply method:
If we're able to find a better way, we might be able to change this back in the future.
DType conversion of modules
Module type conversion will need rethinking the module design taking into account (im)mutability of modules and things like mixed precision training and perhaps quantization. For now the type is fixed on module creation.
Other changes