Skip to main content

How to self-host a model

You can deploy a model in your AWS, GCP, Azure, Lambda, or other clouds using:

Self-hosting an open-source model​

For many cases, either Continue will have a built-in provider or the API you use will be OpenAI-compatible, in which case you can use the "openai" provider and change the "baseUrl" to point to the server.

However, if neither of these are the case, you will need to wire up a new LLM object.

Authentication​

Basic authentication can be done with any provider using the apiKey field:

config.json
{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "llama2-7b",
      "apiKey": "xxx"
    }
  ]
}

This translates to the header "Authorization": "Bearer xxx".

If you need to send custom headers for authentication, you may use the requestOptions.headers property like in this example with Ollama:

config.json
{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "llama2-7b",
      "requestOptions": {
        "headers": {
          "X-Auth-Token": "xxx"
        }
      }
    }
  ]
}

Similarly if your model requires a Certificate for authentication, you may use the requestOptions.clientCertificate property like in the example below:

config.json
{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "llama2-7b",
      "requestOptions": {
        "clientCertificate": {
          "cert": "C:\tempollama.pem",
          "key": "C:\tempollama.key",
          "passphrase": "c0nt!nu3"
        }
      }
    }
  ]
}