Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to get OTIS dataset to run? #1763

Open
dsputnikk opened this issue Feb 27, 2024 · 0 comments
Open

Unable to get OTIS dataset to run? #1763

dsputnikk opened this issue Feb 27, 2024 · 0 comments
Labels
bug Something isn't working unconfirmed

Comments

@dsputnikk
Copy link

dsputnikk commented Feb 27, 2024

LocalAI version:
quay.io/go-skynet/local-ai:latest

Environment, CPU architecture, OS, and Version:
Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 15 07:49:10 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
When I download this dataset: OTIS Official Spam Model

and then run:

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:latest --models-path /models --context-size 700 --threads 4

... it launches. I do:

curl -s http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ 
     "backend": "piper", "model": "model.onnx",                            
     "input": "this is a test 2 suspcisious activity detected. click on https://hack.me"
   }'

and receive:

{
  "created": 1708993374,
  "object": "text_completion",
  "id": "9ace87d3-1e8c-4271-b6ce-37eebf1e1674",
  "model": "model.onnx",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

I expect to get more output.

The backend piper log says it loaded OK

12:17AM INF [piper] Attempting to load
12:17AM INF Loading model 'model.onnx' with backend piper
12:17AM INF [piper] Loads OK

To Reproduce
git clone https://huggingface.co/Titeiiko/OTIS-Official-Spam-Model into the models folder and then run

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:latest --models-path /models --context-size 700 --threads 4

followed by an attempted test (model.onnx is in the models/OTIS.... directory) :

curl -s http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ 
     "backend": "piper", "model": "model.onnx",                            
     "input": "this is a test 2 suspcisious activity detected. click on https://hack.me"
   }'

Expected behavior
I get an AI response

Logs

12:28AM DBG Request received: 
12:28AM DBG `input`: &{PredictionOptions:{Model:model.onnx Language: N:0 TopP:0 TopK:0 Temperature:0 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Context:context.Background.WithCancel Cancel:0x4a9d40 File: ResponseFormat:{Type:} Size: Prompt:<nil> Instruction: Input:this is a test 2 suspcisious activity detected. click on https://hack.me Stop:<nil> Messages:[] Functions:[] FunctionCall:<nil> Tools:[] ToolsChoice:<nil> Stream:false Mode:0 Step:0 Grammar: JSONFunctionGrammarObject:<nil> Backend:piper ModelBaseName:}
12:28AM DBG Parameter Config: &{PredictionOptions:{Model:model.onnx Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:512 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name: F16:false Threads:4 Debug:true Roles:map[] Embeddings:false Backend:piper TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions:} PromptStrings:[] InputStrings:[this is a test 2 suspcisious activity detected. click on https://hack.me] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:700 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:}
12:28AM DBG Response: {"created":1708993689,"object":"text_completion","id":"2b735f7a-1ffc-4639-ae5e-dcf10a315eed","model":"model.onnx","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
[172.17.0.1]:52006 200 - POST /v1/completions

Additional context

@dsputnikk dsputnikk added bug Something isn't working unconfirmed labels Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working unconfirmed
Projects
None yet
Development

No branches or pull requests

1 participant