Nvidia NIM

[Dual-0]

Hello,

Nvidia offers a "NIM access through the NVIDIA Developer Program". This has some limits but I think it can be integrated easily cause its based on OpenAI API. Good news who wanna try Nvidias llama-3.1-nemotron-70b, it already works via custom endpoint

URL: https://integrate.api.nvidia.com/
Endpoint: POST /v1/chat/completions

custom endpoint

# Definition of custom endpoints
endpoints:
  custom:
    - name: "Nvidia"
      apiKey: "${NVIDIA_API_KEY}"
      baseURL: "https://integrate.api.nvidia.com/v1/"
      models:
        default: [
          "nvidia/llama-3.1-nemotron-51b-instruct",
          "nvidia/llama-3.1-nemotron-70b-instruct",
          "nvidia/nemotron-mini-4b-instruct",
          "nvidia/nemotron-4-340b-instruct",
        ]
        fetch: false
      titleConvo: true
      titleModel: "nvidia/nemotron-mini-4b-instruct"
      modelDisplayLabel: "Nvidia"

example with curl:

curl --request POST \
     --url https://integrate.api.nvidia.com/v1/chat/completions \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "model": "nvidia/llama-3.1-nemotron-70b-instruct",
  "max_tokens": 1024,
  "stream": false,
  "temperature": 0.5,
  "top_p": 1,
  "stop": null,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "seed": 0,
  "messages": [
    {
      "role": "user",
      "content": "Write a limerick about the wonders of GPU computing."
    }
  ]
}
'

output:

{
  "id": "string",
  "object": "chat.completion",
  "created": 0,
  "model": "nvidia/llama-3.1-nemotron-70b-instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "string",
        "content": "string"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0,
    "completion_tokens": 0
  }
}

[berry-13]

thanks @Dual-0!

@fuegovic we can add this to the demo and also add it to the know endpoint list

[fuegovic]

Freshly added to the demo!