Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mistral to obcq #1798

Merged
merged 3 commits into from
Oct 31, 2023
Merged

Add mistral to obcq #1798

merged 3 commits into from
Oct 31, 2023

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Oct 28, 2023

Adds mistral support for the obcq script using the latest nm-transformers. This diff experiments with loading with the simple AutoModel interface and finding the model arch name from the config file's model_type

Here is a sample recipe I used for zephyr-beta

test_stage:
  obcq_modifiers:
    QuantizationModifier:
      ignore:
        - MistralRotaryEmbedding
        - MistralRMSNorm
        - SiLUActivation
        - model.layers.1.mlp.down_proj
        - model.layers.31.mlp.down_proj
        - model.layers.30.mlp.down_proj
        - model.layers.30.mlp.gate_proj
        - model.layers.30.mlp.up_proj
      post_oneshot_calibration: True
      scheme_overrides:
        Embedding:
          input_activations: null
          weights:
            num_bits: 8
            symmetric: False
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: False
      quantize: True
      percdamp: 0.01
      prunen: 0
      prunem: 0
      targets: [
        "model.layers.0",
        "model.layers.1",
        "model.layers.2",
        "model.layers.3",
        "model.layers.4",
        "model.layers.5",
        "model.layers.6",
        "model.layers.7",
        "model.layers.8",
        "model.layers.9",
        "model.layers.10",
        "model.layers.11",
        "model.layers.12",
        "model.layers.13",
        "model.layers.14",
        "model.layers.15",
        "model.layers.16",
        "model.layers.17",
        "model.layers.18",
        "model.layers.19",
        "model.layers.20",
        "model.layers.21",
        "model.layers.22",
        "model.layers.23",
        "model.layers.24",
        "model.layers.25",
        "model.layers.26",
        "model.layers.27",
        "model.layers.28",
        "model.layers.29",
        "model.layers.30",
        "model.layers.31",
      ]
      target_ids: ["attention_mask", "position_ids"]  

@mgoin mgoin requested a review from Satrat October 28, 2023 21:59
Satrat
Satrat previously approved these changes Oct 30, 2023
@mgoin mgoin marked this pull request as ready for review October 30, 2023 19:40
@mgoin mgoin merged commit b622bba into main Oct 31, 2023
11 checks passed
@mgoin mgoin deleted the mistral-obcq branch October 31, 2023 01:02
bfineran pushed a commit that referenced this pull request Nov 16, 2023
* Add mistral to obcq

* Update with model_type from config
bfineran pushed a commit that referenced this pull request Nov 16, 2023
* Add mistral to obcq

* Update with model_type from config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants