Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sync support for dict collections of metrics #98

Closed
wants to merge 2 commits into from
Closed

Add sync support for dict collections of metrics #98

wants to merge 2 commits into from

Conversation

bobakfb
Copy link
Contributor

@bobakfb bobakfb commented Dec 7, 2022

Summary: Added support for two new sync methods sync_and_compute_collection and get_synced_state_dicts_collection. These methods use only a single data transfer per sync rather than one per metric.

Differential Revision: D41674853

@facebook-github-bot facebook-github-bot added CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported labels Dec 7, 2022
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D41674853

bobakfb added a commit to bobakfb/torcheval that referenced this pull request Dec 7, 2022
@bobakfb @facebook-github-bot
Add sync support for dict collections of metrics (pytorch#98)
9e99a48 
Summary:
Pull Request resolved: pytorch#98

Added support for two new sync methods `sync_and_compute_collection` and `get_synced_state_dicts_collection`. These methods use only a single data transfer per sync rather than one per metric.

Differential Revision: D41674853

fbshipit-source-id: 369b2586b371d7b46bddb4192537c6c87390c2fc
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D41674853

@codecov
Copy link

codecov bot commented Dec 7, 2022 •
edited
Loading

Codecov Report

Merging #98 (ddc8d1f) into main (043ef07) will increase coverage by 0.01%.
The diff coverage is 95.04%.

@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
+ Coverage   95.29%   95.30%   +0.01%     
==========================================
  Files         143      143              
  Lines        7991     8100     +109     
==========================================
+ Hits         7615     7720     +105     
- Misses        376      380       +4
Impacted Files Coverage Δ
torcheval/metrics/toolkit.py 94.95% <91.93%> (-0.64%) ⬇️
tests/metrics/test_toolkit.py 98.91% <98.30%> (-0.30%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

bobakfb added a commit to bobakfb/torcheval that referenced this pull request Dec 12, 2022
@bobakfb @facebook-github-bot
Add sync support for dict collections of metrics (pytorch#98)
db444b5 
Summary:
Pull Request resolved: pytorch#98

Added support for two new sync methods `sync_and_compute_collection` and `get_synced_state_dicts_collection`. These methods use only a single data transfer per sync rather than one per metric.

Reviewed By: ananthsub

Differential Revision: D41674853

fbshipit-source-id: 1df55ca544b1d910e76408c75bf0cc96f554001a
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D41674853

bobakfb and others added 2 commits December 12, 2022 17:00
@bobakfb @facebook-github-bot
Fix docstring examples in get_synced_metric
a88b990 
Summary: Docstring has code that would throw an error incorrectly reported as returning `None`

Differential Revision: https://internalfb.com/D41830353

fbshipit-source-id: 46a9aa516c0bea23c9164b50d14cec7bc37be372
@bobakfb @facebook-github-bot
Add sync support for dict collections of metrics (pytorch#98)
ddc8d1f 
Summary:
Pull Request resolved: pytorch#98

Added support for two new sync methods `sync_and_compute_collection` and `get_synced_state_dicts_collection`. These methods use only a single data transfer per sync rather than one per metric.

Reviewed By: ananthsub

Differential Revision: D41674853

fbshipit-source-id: e020fab12b1647bf9ea40e02ed611c2a62f2e4d1
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D41674853

bobakfb added a commit to bobakfb/torcheval that referenced this pull request Jan 25, 2023
@bobakfb @facebook-github-bot
Bumping to Version 0.0.6
8383a0e 
Summary:
# TorchEval Version 0.0.6

## Change Log

 - New metrics:
   - AUC
   - Binary, Multiclass, Multilabel AUPRC (also called Average Precision) pytorch#108 pytorch#109
   - Multilabel Precision Recall Curve pytorch#87
   - Recall at Fixed Precision pytorch#88 pytorch#91
   - Windowed Mean Square Error pytorch#72 pytorch#86
   - Blue Score pytorch#93 pytorch#95
   - Perplexity pytorch#90
   - Word Error Rate pytorch#97
   - Word Information Loss pytorch#111
   - Word Information Preserved pytorch#110
 - Features
   - Added Sync for Dictionaries of Metrics pytorch#98
   - Improved FLOPS counter pytorch#81
   - Improved Module Summary, added forward elapsed times pytorch#100 pytorch#103 pytorch#104 pytorch#105 pytorch#114
   - AUROC now supports weighted inputs pytorch#94
 - Other
   - Improved Documentation pytorch#80 pytorch#117 pytorch#121
   - Added Module Summary to Quickstart pytorch#113
   - Updates several unit tests pytorch#77 pytorch#96 pytorch#101 pytorch#73
   - Docs Automatically Add New Metrics pytorch#118
   - Several Aggregation Metrics now Support fp64 pytorch#116 pytorch#123

### [BETA] Sync Dictionaries of Metrics

We're looking forward to building tooling for metric collections. The first important feature towards this end is collective syncing of groups of metrics. In the example below, we show how easy it is to sync all your metrics at the same time with `sync_and_compute_collection`.

This method is not only for convenience, on the backend we only use one torch distributed sync collective for the entire group of metrics, meaning that the overhead from repeated network directives is maximally reduced.

```python
import torch
from torcheval.metrics import BinaryAUPRC, BinaryAUROC, BinaryAccuracy
from torcheval.metrics.toolkit import sync_and_compute_collection, reset_metrics

# Collections should be Dict[str, Metric]
train_metrics = {
    "train_auprc": BinaryAUPRC(),
    "train_auroc": BinaryAUROC(),
    "train_accuracy": BinaryAccuracy(),
}

# Hydrate metrics with some random data
preds = torch.rand(size=(100,))
targets = torch.randint(low=0, high=2, size=(100,))

for name, metric in train_metrics.items():
    metric.update(preds, targets)

# Sync the whole group with a single gather
print(sync_and_compute_collection(train_metrics))
>>> {'train_auprc': tensor(0.5913), 'train_auroc': tensor(0.5161, dtype=torch.float64), 'train_accuracy': tensor(0.5100)}

# reset all metrics in collection
reset_metrics(train_metrics.values())
```

Be on the lookout for more metric collection code coming in future releases.

## Contributors

We're grateful for our community, which helps us improving torcheval by highlighting issues and contributing code. The following persons have contributed patches for this release: Rohit Alekar lindawangg Julia Reinspach jingchi-wang Ekta Sardana williamhufb @\andreasfloros Erika Lal samiwilf

Reviewed By: ananthsub

Differential Revision: D42737308

fbshipit-source-id: 4c9d72ce73a35636d7cd6421926a23a80250e267
@bobakfb bobakfb mentioned this pull request Jan 25, 2023
Closed
facebook-github-bot pushed a commit that referenced this pull request Jan 25, 2023
@bobakfb @facebook-github-bot
Bumping to Version 0.0.6 (#124)
8d46858 
Summary:
Pull Request resolved: #124

# TorchEval Version 0.0.6

## Change Log

 - New metrics:
   - AUC
   - Binary, Multiclass, Multilabel AUPRC (also called Average Precision) #108 #109
   - Multilabel Precision Recall Curve #87
   - Recall at Fixed Precision #88 #91
   - Windowed Mean Square Error #72 #86
   - Blue Score #93 #95
   - Perplexity #90
   - Word Error Rate #97
   - Word Information Loss #111
   - Word Information Preserved #110
 - Features
   - Added Sync for Dictionaries of Metrics #98
   - Improved FLOPS counter #81
   - Improved Module Summary, added forward elapsed times #100 #103 #104 #105 #114
   - AUROC now supports weighted inputs #94
 - Other
   - Improved Documentation #80 #117 #121
   - Added Module Summary to Quickstart #113
   - Updates several unit tests #77 #96 #101 #73
   - Docs Automatically Add New Metrics #118
   - Several Aggregation Metrics now Support fp64 #116 #123

### [BETA] Sync Dictionaries of Metrics

We're looking forward to building tooling for metric collections. The first important feature towards this end is collective syncing of groups of metrics. In the example below, we show how easy it is to sync all your metrics at the same time with `sync_and_compute_collection`.

This method is not only for convenience, on the backend we only use one torch distributed sync collective for the entire group of metrics, meaning that the overhead from repeated network directives is maximally reduced.

```python
import torch
from torcheval.metrics import BinaryAUPRC, BinaryAUROC, BinaryAccuracy
from torcheval.metrics.toolkit import sync_and_compute_collection, reset_metrics

# Collections should be Dict[str, Metric]
train_metrics = {
    "train_auprc": BinaryAUPRC(),
    "train_auroc": BinaryAUROC(),
    "train_accuracy": BinaryAccuracy(),
}

# Hydrate metrics with some random data
preds = torch.rand(size=(100,))
targets = torch.randint(low=0, high=2, size=(100,))

for name, metric in train_metrics.items():
    metric.update(preds, targets)

# Sync the whole group with a single gather
print(sync_and_compute_collection(train_metrics))
>>> {'train_auprc': tensor(0.5913), 'train_auroc': tensor(0.5161, dtype=torch.float64), 'train_accuracy': tensor(0.5100)}

# reset all metrics in collection
reset_metrics(train_metrics.values())
```

Be on the lookout for more metric collection code coming in future releases.

## Contributors

We're grateful for our community, which helps us improving torcheval by highlighting issues and contributing code. The following persons have contributed patches for this release: Rohit Alekar lindawangg Julia Reinspach jingchi-wang Ekta Sardana williamhufb @\andreasfloros Erika Lal samiwilf

Reviewed By: ananthsub

Differential Revision: D42737308

fbshipit-source-id: dfd852345e1a9f3069ea33b056f5a60a3adde5aa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bobakfb @facebook-github-bot