[Update] layer prefix to be set at model level #1778

rahul-tuli · 2023-10-19T14:06:39Z

This PR allows layer_prefix field to be set at ModelMetaData level rather than at individual Modifier level

Changes include:

Propagating metadata when recipes are combined (includes updates needed in session.py)
Attaching layer_prefix to ModifiableModel level before calling initialize
Use layer_prefix from ModifiableModel instance in the Modifier

Test Recipe:

# llama_small_recipe.yaml

metadata:
  target_model:
    layer_prefix: "decoder"
    architecture: "opt"
test_stage:
  obcq_modifiers:
    QuantizationModifier:
      ignore: ["lm_head", "Embedding", "OPTLearnedPositionalEmbedding", "QuantizableBatchMatMul", "BMMLeftInput_QK", "BMMRightInput_QK", "BMMOutput_QK", "BMMLeftInput_PV", "BMMRightInput_PV", "BMMOutput_PV"]
      post_oneshot_calibration: True
      scheme_overrides:
        ReLU:
          input_activations: null
          output_activations: null
        LayerNorm:
          input_activations: null
          output_activations: null
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: False
      quantize: True
      percdamp: 0.01
      prunen: 0
      prunem: 0
      targets: [
        "model.decoder.layers.0"
      ]
      target_ids: ["attention_mask"]

Test Output:

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b
 c4 --recipe local/llama_small_recipe.yaml --eval wikitext2 
Found cached dataset json (/home/rahul/.cache/huggingface/datasets/allenai___json/allenai--c4-6fbe877195f42de5/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)
2023-10-19 09:40:34 sparseml.modifiers.quantization.pytorch INFO     Running quantization calibration using calibration_dataloader

rahul at office-desktop in ~/projects/sparseml (attach-layer-prefix-to-the-model●●) (.venv) 
$  cd /home/rahul/projects/sparseml ; /usr/bin/env /home/rahul/projects/.venv/bin/python /home/rahul/.vscode-server/extensions/ms-python.python-2023
.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 52037 -- src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b
 c4 --recipe local/llama_small_recipe.yaml --eval wikitext2 
Found cached dataset json (/home/rahul/.cache/huggingface/datasets/allenai___json/allenai--c4-6fbe877195f42de5/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)
2023-10-19 09:43:52 sparseml.modifiers.quantization.pytorch INFO     Running quantization calibration using calibration_dataloader

rahul at office-desktop in ~/projects/sparseml (attach-layer-prefix-to-the-model●●) (.venv) 
$  cd /home/rahul/projects/sparseml ; /usr/bin/env /home/rahul/projects/.venv/bin/python /home/rahul/.vscode-server/extensions/ms-python.python-2023
.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 43113 -- src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b
 c4 --recipe local/llama_small_recipe.yaml --eval wikitext2 
Found cached dataset json (/home/rahul/.cache/huggingface/datasets/allenai___json/allenai--c4-6fbe877195f42de5/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)
2023-10-19 09:49:13 sparseml.modifiers.quantization.pytorch INFO     Running quantization calibration using calibration_dataloader
2023-10-19 09:50:16 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 0/0 =====
2023-10-19 09:50:17 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj.module...
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.55
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 31129.56
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj.module...
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.45
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 7156.07
2023-10-19 09:50:18 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj.module...
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.45
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 31903.02
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj.module...
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.45
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 25.32
2023-10-19 09:50:19 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1.module...
2023-10-19 09:50:20 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.46
2023-10-19 09:50:20 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 31037.05
2023-10-19 09:50:20 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2.module...
2023-10-19 09:50:22 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 2.00
2023-10-19 09:50:22 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 284.96
2023-10-19 09:50:22 sparseml.modifiers.quantization.pytorch INFO     Running quantization calibration using calibration_dataloader
Found cached dataset wikitext (/home/rahul/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
2023-10-19 09:50:34 sparseml.modifiers.obcq.utils.helpers INFO     Evaluating perplexity...
2023-10-19 09:50:38 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.4565, device='cuda:0')
2023-10-19 09:50:43 sparseml.modifiers.obcq.utils.helpers INFO     tensor(16.6085, device='cuda:0')
2023-10-19 09:50:47 sparseml.modifiers.obcq.utils.helpers INFO     tensor(17.0098, device='cuda:0')
2023-10-19 09:50:52 sparseml.modifiers.obcq.utils.helpers INFO     tensor(16.6345, device='cuda:0')
2023-10-19 09:50:56 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.3444, device='cuda:0')
2023-10-19 09:51:01 sparseml.modifiers.obcq.utils.helpers INFO     tensor(16.2035, device='cuda:0')
2023-10-19 09:51:05 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.3615, device='cuda:0')
2023-10-19 09:51:09 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.3593, device='cuda:0')
2023-10-19 09:51:14 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.1442, device='cuda:0')
2023-10-19 09:51:18 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.1357, device='cuda:0')
2023-10-19 09:51:22 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.1277, device='cuda:0')
2023-10-19 09:51:27 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.5246, device='cuda:0')
2023-10-19 09:51:31 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.7131, device='cuda:0')
2023-10-19 09:51:35 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.6395, device='cuda:0')
2023-10-19 09:51:39 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.5373, device='cuda:0')
2023-10-19 09:51:44 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.4753, device='cuda:0')
2023-10-19 09:51:48 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.4658, device='cuda:0')
2023-10-19 09:51:52 sparseml.modifiers.obcq.utils.helpers INFO     tensor(15.5323, device='cuda:0')
2023-10-19 09:51:52 sparseml.modifiers.obcq.utils.helpers INFO     Perplexity: 15.532339

to use layer prefix for from model - Remove `layer_prefix` from `SparseGPTModifier` base - Update ModelMetaData to include layer_prefix - Added a convenience function to update missing values in RecipeMetaData instance from another RecipeMetaData instance - Update simplify recipe to also include metadata - Update simplify_combine_recipes to include metadata - Add layer_prefix property to `ModifiableModel` - propagate `layer_prefix` to superclass - update session.py to set_layer_prefix on the model before initializing modifiers - Update example recipe to include layer_prefix in metadata

Satrat

Nice work on this, I think including the prefix under metadata makes a lot of sense. Before this gets merged I'd like to see a unit test propogating the prefix from the recipe to the modifiable model, making sure the model.prefix is None if no prefix is set

src/sparseml/core/model/pytorch.py

src/sparseml/core/recipe/metadata.py

- update docstring - add test for `update_missing_metadata`

Satrat

Looks good to me, I tested locally that this runs on OPT and Llama. When this merges though, can you ping out to ml-engineering or ml-research about the UX change? Since they'll have to update their recipes to work with this change

rahul-tuli · 2023-10-26T17:23:04Z

Looks good to me, I tested locally that this runs on OPT and Llama. When this merges though, can you ping out to ml-engineering or ml-research about the UX change? Since they'll have to update their recipes to work with this change

Absolutely will do!

@Satrat

* Initial commit * Add end to end tests * Add e2e tests for constant pruning modifier * Move imports inside the test fuctions so that torch isn't imported unless running the tests * Update setup.py to not run modifier tests unless pytorch is specified * [Bugfix] .dict() method on Recipe (#1753) * Bugfix .dict() method on Recipe * Remove extraneous local test, [faulty commit] * [modifier refactor] Add serialization tests (#1755) * Add serialization tests * Clean up * Keep original stage and group names Clean up _get_yaml_dict * fix comment * Typo * [Unit Tests][Modifier Refactor] (#1756) * Move valid recipes to a helper file Add tests for session.py * Increase test coverage of src/sparseml/core/session.py to 100% Run Style Add logs to .gitignore * Increase coverage of tests/sparseml/core/test_state.py to 100% * add tests for lifecycle/event.py * Increase code coverage of lifecycle/event to 100% * increase lifecycle/session.py code coverage to 93% * Address review comments from @Satrat * Address review comments on 1752 (#1772) Update makefile to only ignore *pytorch.py files in modifier dir Fix order in test Add regex to makefile Add helper function to determine if torch tests should be run Check masks Make transformers import optional in sparsegpt.py * Fix merge conflict * Add more tests to check valid modifiers are created (#1774) * [Bug][ConstantPruningModifier] Fix mask de register bug (#1773) * Fix mask de-register logic * forgot to remove commented out line * Move tests inside pytorch directory as requested * Fix session reset (#1790)

* Add kvcache config for Mistral * Update configs.py * Update configs.py

@Satrat

* - Update `src/sparseml/modifiers/obcq/pytorch.py` to use layer prefix for from model - Remove `layer_prefix` from `SparseGPTModifier` base - Update ModelMetaData to include layer_prefix - Added a convenience function to update missing values in RecipeMetaData instance from another RecipeMetaData instance - Update simplify recipe to also include metadata - Update simplify_combine_recipes to include metadata - Add layer_prefix property to `ModifiableModel` - propagate `layer_prefix` to superclass - update session.py to set_layer_prefix on the model before initializing modifiers - Update example recipe to include layer_prefix in metadata * Add missing docstring * - address review comment - update docstring - add test for `update_missing_metadata` * Add test * Style * Fix tests * Style * [modifier refactor] Add constant pruning tests (#1752) * Initial commit * Add end to end tests * Add e2e tests for constant pruning modifier * Move imports inside the test fuctions so that torch isn't imported unless running the tests * Update setup.py to not run modifier tests unless pytorch is specified * [Bugfix] .dict() method on Recipe (#1753) * Bugfix .dict() method on Recipe * Remove extraneous local test, [faulty commit] * [modifier refactor] Add serialization tests (#1755) * Add serialization tests * Clean up * Keep original stage and group names Clean up _get_yaml_dict * fix comment * Typo * [Unit Tests][Modifier Refactor] (#1756) * Move valid recipes to a helper file Add tests for session.py * Increase test coverage of src/sparseml/core/session.py to 100% Run Style Add logs to .gitignore * Increase coverage of tests/sparseml/core/test_state.py to 100% * add tests for lifecycle/event.py * Increase code coverage of lifecycle/event to 100% * increase lifecycle/session.py code coverage to 93% * Address review comments from @Satrat * Address review comments on 1752 (#1772) Update makefile to only ignore *pytorch.py files in modifier dir Fix order in test Add regex to makefile Add helper function to determine if torch tests should be run Check masks Make transformers import optional in sparsegpt.py * Fix merge conflict * Add more tests to check valid modifiers are created (#1774) * [Bug][ConstantPruningModifier] Fix mask de register bug (#1773) * Fix mask de-register logic * forgot to remove commented out line * Move tests inside pytorch directory as requested * Fix session reset (#1790) * fix datasets version to be compatible with fsspec (#1797) * Add kvcache config for Mistral (#1766) * Add kvcache config for Mistral * Update configs.py * Update configs.py * Fix reset logic * Style after resolving merge conflicts --------- Co-authored-by: Sara Adkins <[email protected]> Co-authored-by: Michael Goin <[email protected]>

@Satrat

* - Update `src/sparseml/modifiers/obcq/pytorch.py` to use layer prefix for from model - Remove `layer_prefix` from `SparseGPTModifier` base - Update ModelMetaData to include layer_prefix - Added a convenience function to update missing values in RecipeMetaData instance from another RecipeMetaData instance - Update simplify recipe to also include metadata - Update simplify_combine_recipes to include metadata - Add layer_prefix property to `ModifiableModel` - propagate `layer_prefix` to superclass - update session.py to set_layer_prefix on the model before initializing modifiers - Update example recipe to include layer_prefix in metadata * Add missing docstring * - address review comment - update docstring - add test for `update_missing_metadata` * Add test * Style * Fix tests * Style * [modifier refactor] Add constant pruning tests (#1752) * Initial commit * Add end to end tests * Add e2e tests for constant pruning modifier * Move imports inside the test fuctions so that torch isn't imported unless running the tests * Update setup.py to not run modifier tests unless pytorch is specified * [Bugfix] .dict() method on Recipe (#1753) * Bugfix .dict() method on Recipe * Remove extraneous local test, [faulty commit] * [modifier refactor] Add serialization tests (#1755) * Add serialization tests * Clean up * Keep original stage and group names Clean up _get_yaml_dict * fix comment * Typo * [Unit Tests][Modifier Refactor] (#1756) * Move valid recipes to a helper file Add tests for session.py * Increase test coverage of src/sparseml/core/session.py to 100% Run Style Add logs to .gitignore * Increase coverage of tests/sparseml/core/test_state.py to 100% * add tests for lifecycle/event.py * Increase code coverage of lifecycle/event to 100% * increase lifecycle/session.py code coverage to 93% * Address review comments from @Satrat * Address review comments on 1752 (#1772) Update makefile to only ignore *pytorch.py files in modifier dir Fix order in test Add regex to makefile Add helper function to determine if torch tests should be run Check masks Make transformers import optional in sparsegpt.py * Fix merge conflict * Add more tests to check valid modifiers are created (#1774) * [Bug][ConstantPruningModifier] Fix mask de register bug (#1773) * Fix mask de-register logic * forgot to remove commented out line * Move tests inside pytorch directory as requested * Fix session reset (#1790) * fix datasets version to be compatible with fsspec (#1797) * Add kvcache config for Mistral (#1766) * Add kvcache config for Mistral * Update configs.py * Update configs.py * Fix reset logic * Style after resolving merge conflicts --------- Co-authored-by: Sara Adkins <[email protected]> Co-authored-by: Michael Goin <[email protected]>

rahul-tuli marked this pull request as ready for review October 19, 2023 14:13

rahul-tuli self-assigned this Oct 19, 2023

rahul-tuli requested review from bfineran, Satrat, dsikka and dbogunowicz October 19, 2023 14:13

rahul-tuli added the mle-team label Oct 19, 2023

Base automatically changed from rename-compress-layers to main October 19, 2023 14:19

rahul-tuli added 2 commits October 19, 2023 10:20

Add missing docstring

9fa0374

rahul-tuli force-pushed the attach-layer-prefix-to-the-model branch from bca23d5 to 9fa0374 Compare October 19, 2023 14:20

Satrat suggested changes Oct 20, 2023

View reviewed changes

src/sparseml/core/model/pytorch.py Outdated Show resolved Hide resolved

Satrat reviewed Oct 20, 2023

View reviewed changes

src/sparseml/core/recipe/metadata.py Show resolved Hide resolved

rahul-tuli added 6 commits October 25, 2023 09:37

- address review comment

046b85d

- update docstring - add test for `update_missing_metadata`

Add test

1bcf3a0

Style

fe367d3

Merge branch 'main' into attach-layer-prefix-to-the-model

8206969

Fix tests

c45c745

Style

2c8f8a8

Satrat self-requested a review October 26, 2023 15:49

Satrat previously approved these changes Oct 26, 2023

View reviewed changes

Merge branch 'main' into attach-layer-prefix-to-the-model

19188fb

rahul-tuli dismissed Satrat’s stale review via 19188fb October 27, 2023 12:53

Satrat previously approved these changes Oct 27, 2023

View reviewed changes

rahul-tuli and others added 3 commits October 30, 2023 09:15

fix datasets version to be compatible with fsspec (#1797)

da27e82

Add kvcache config for Mistral (#1766)

037e302

* Add kvcache config for Mistral * Update configs.py * Update configs.py

rahul-tuli dismissed Satrat’s stale review via 037e302 October 30, 2023 13:16

Merge branch 'main' into attach-layer-prefix-to-the-model

aaf80c3

Fix reset logic

3b0b319

bfineran previously approved these changes Oct 30, 2023

View reviewed changes

Satrat previously approved these changes Oct 31, 2023

View reviewed changes

Merge branch 'main' into attach-layer-prefix-to-the-model

8b7d859

rahul-tuli dismissed stale reviews from Satrat and bfineran via 8b7d859 October 31, 2023 12:43

Style after resolving merge conflicts

9734ef6

Satrat approved these changes Oct 31, 2023

View reviewed changes

bfineran approved these changes Oct 31, 2023

View reviewed changes

rahul-tuli merged commit 24a61a9 into main Oct 31, 2023
11 checks passed

rahul-tuli deleted the attach-layer-prefix-to-the-model branch October 31, 2023 13:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Update] layer prefix to be set at model level #1778

[Update] layer prefix to be set at model level #1778

rahul-tuli commented Oct 19, 2023 •

edited

Loading

Satrat left a comment

Satrat left a comment

rahul-tuli commented Oct 26, 2023

[Update] layer prefix to be set at model level #1778

[Update] layer prefix to be set at model level #1778

Conversation

rahul-tuli commented Oct 19, 2023 • edited Loading

Satrat left a comment

Choose a reason for hiding this comment

Satrat left a comment

Choose a reason for hiding this comment

rahul-tuli commented Oct 26, 2023

rahul-tuli commented Oct 19, 2023 •

edited

Loading