Skip to content

Commit

Permalink
Wrong import in zero (#169)
Browse files Browse the repository at this point in the history
## Title

Prevent from using torch 2.0

## Description

- Some of feature have changed in torch 2.0. and oslo has dependency on
torch._six which no longer support by torch 2.0.

olso Dependency
-
https://github.com/EleutherAI/oslo/blob/910c789e7f46d2876b964c221d31984b7924974f/oslo/torch/nn/parallel/data_parallel/zero/sharded_optim/_utils.py#L19

other issues
- microsoft/DeepSpeed#2845

## Linked Issues

- resolved #00
  • Loading branch information
jinwonkim93 authored and dyanos committed Jun 8, 2023
1 parent d62bc18 commit d87c5ba
Showing 1 changed file with 2 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@

from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
import torch.distributed as dist
from torch._six import inf
import math
from math import sqrt, inf
import torch
from typing import Optional, Iterable, List, Union
from oslo.torch.distributed import ParallelMode
Expand Down Expand Up @@ -86,7 +85,7 @@ def calculate_global_norm_from_list(norm_list: List[float]) -> float:
total_norm = 0.0
for norm in norm_list:
total_norm += norm**2.0
return math.sqrt(total_norm)
return sqrt(total_norm)


def reduce_tensor_dp_group(
Expand Down

0 comments on commit d87c5ba

Please sign in to comment.