Model Developer
- DongGeon Lee [HuggingFace] [GitHub]
Model Architecture
- KoSOLAR-v0.2-gugutypus-10.7B is an instruction fine-tuned auto-regressive language model, based on the SOLAR transformer architecture.
Base Model
Training Dataset
- Ko-LLM leaderboard (2024/03/01) [link]
| Model | Average | Ko-ARC | Ko-HellaSwag | Ko-MMLU | Ko-TruthfulQA | Ko-CommonGen V2 |
|---|---|---|---|---|---|---|
| oneonlee/KoSOLAR-v0.2-gugutypus-10.7B | 51.17 | 47.78 | 58.29 | 47.27 | 48.31 | 54.19 |
| oneonlee/LDCC-SOLAR-gugutypus-10.7B | 49.45 | 45.9 | 55.46 | 47.96 | 48.93 | 49 |
- (KOR) AI-Harness evaluation [link]
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| KMMLU | N/A | none | 0 | acc | 0.3335 | ± | 0.0475 |
| KMMLU | N/A | none | 5 | acc | 0.3938 | ± | 0.0823 |
| KoBEST-HellaSwag | 0 | none | 0 | acc | 0.4360 | ± | 0.0222 |
| KoBEST-HellaSwag | 0 | none | 5 | acc | 0.4420 | ± | 0.0222 |
| KoBEST-BoolQ | 0 | none | 0 | acc | 0.5064 | ± | 0.0133 |
| KoBEST-BoolQ | 0 | none | 5 | acc | 0.8583 | ± | 0.0093 |
| KoBEST-COPA | 0 | none | 0 | acc | 0.6040 | ± | 0.0155 |
| KoBEST-COPA | 0 | none | 5 | acc | 0.7610 | ± | 0.0135 |
| KoBEST-SentiNeg | 0 | none | 0 | acc | 0.5844 | ± | 0.0248 |
| KoBEST-SentiNeg | 0 | none | 5 | acc | 0.9471 | ± | 0.0112 |
- (ENG) AI-Harness evaluation [link]
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| MMLU | N/A | none | 0 | acc | 0.5826 | ± | 0.1432 |
| MMLU | N/A | none | 5 | acc | 0.5885 | ± | 0.1285 |
| HellaSwag | 1 | none | 0 | acc | 0.6075 | ± | 0.0049 |
| HellaSwag | 1 | none | 5 | acc | 0.6098 | ± | 0.0049 |
| BoolQ | 2 | none | 0 | acc | 0.8737 | ± | 0.0058 |
| BoolQ | 2 | none | 5 | acc | 0.8826 | ± | 0.0056 |
| COPA | 1 | none | 0 | acc | 0.8300 | ± | 0.0378 |
| COPA | 1 | none | 5 | acc | 0.9100 | ± | 0.0288 |
| truthfulqa | N/A | none | 0 | acc | 0.4249 | ± | 0.0023 |
| truthfulqa | N/A | none | 5 | acc | - | ± | - |
### KoSOLAR-gugutypus
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "oneonlee/KoSOLAR-v0.2-gugutypus-10.7B"
model = AutoModelForCausalLM.from_pretrained(
repo,
return_dict=True,
torch_dtype=torch.float16,
device_map='auto'
)
tokenizer = AutoTokenizer.from_pretrained(repo)@misc {donggeon_lee_2024,
author = { {DongGeon Lee} },
title = { KoSOLAR-v0.2-gugutypus-10.7B (Revision 56841d5) },
year = 2024,
url = { https://huggingface.co/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B },
doi = { 10.57967/hf/1735 },
publisher = { Hugging Face }
}