Chat Role

A "chat model" is an LLM that is trained to respond in a conversational format. Because they should be able to answer general questions and generate complex code, the best chat models are typically large, often 405B+ parameters.

In Continue, these models are used for Chat and Actions. The selected chat model will also be used for Edit and Apply if no edit or apply models are specified, respectively.

Recommended Chat models

Best overall experience

For the best overall Chat experience, you will want to use a 400B+ parameter model or one of the frontier models.

Claude Sonnet 3.5 from Anthropic

Our current top recommendation is Claude 3.7 Sonnet from Anthropic.

Hub

View the Claude 3.7 Sonnet model block on the hub.

YAML

config.yaml

models:
  - name: Claude 3.7 Sonnet
    provider: anthropic
    model: claude-3-7-sonnet-latest
    apiKey: <YOUR_ANTHROPIC_API_KEY>

JSON

config.json

{
  "models": [
    {
      "title": "Claude 3.5 Sonnet",
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-latest",
      "apiKey": "<YOUR_ANTHROPIC_API_KEY>"
    }
  ]
}

Llama 3.1 405B from Meta

If you prefer to use an open-weight model, then Llama 3.1 405B from Meta is your best option right now. You will need to decide if you use it through a SaaS model provider (e.g. Together or Groq) or self-host it (e.g. using vLLM or Ollama).

YAML

Together

config.yaml

models:
  - name: "Llama 3.1 405B"
    provider: "together"
    model: "llama3.1-405b"
    apiKey: <YOUR_TOGETHER_API_KEY>

Groq

config.yaml

models:
  - name: "Llama 3.1 405B"
    provider: "groq"
    model: "llama3.1-405b"
    apiKey: <YOUR_GROQ_API_KEY>

vLLM

config.yaml

models:
  - name: "Llama 3.1 405B"
    provider: "vllm"
    model: "llama3.1-405b"

Ollama

config.yaml

models:
  - name: "Llama 3.1 405B"
    provider: "ollama"
    model: "llama3.1:405b"

JSON

Together

config.json

{
  "models": [
    {
      "title": "Llama 3.1 405B",
      "provider": "together",
      "model": "llama3.1-405b",
      "apiKey": "<YOUR_TOGETHER_API_KEY>"
    }
  ]
}

Groq

config.json

{
  "models": [
    {
      "title": "Llama 3.1 405B",
      "provider": "groq",
      "model": "llama3.1-405b",
      "apiKey": "<YOUR_GROQ_API_KEY>"
    }
  ]
}

vLLM

config.json

{
  "models": [
    {
      "title": "Llama 3.1 405B",
      "provider": "vllm",
      "model": "llama3.1-405b"
    }
  ]
}

Ollama

config.json

{
  "models": [
    {
      "title": "Llama 3.1 405B",
      "provider": "ollama",
      "model": "llama3.1:405b"
    }
  ]
}

GPT-4o from OpenAI

If you prefer to use a model from OpenAI, then we recommend GPT-4o.

Hub

Add the OpenAI GPT-4o block from the hub

YAML

config.yaml

models:
  - name: GPT-4o
    provider: openai
    model: ''
    apiKey: <YOUR_OPENAI_API_KEY>

JSON

config.json

{
  "models": [
    {
      "title": "GPT-4o",
      "provider": "openai",
      "model": "",
      "apiKey": "<YOUR_OPENAI_API_KEY>"
    }
  ]
}

Grok-2 from xAI

If you prefer to use a model from xAI, then we recommend Grok-2.

Hub

Add the xAI Grok-2 block from the hub

YAML

config.yaml

models:
  - name: Grok-2
    provider: xAI
    model: grok-2-latest
    apiKey: <YOUR_XAI_API_KEY>

JSON

config.json

{
  "models": [
    {
      "title": "Grok-2",
      "provider": "xAI",
      "model": "grok-2-latest",
      "apiKey": "<YOUR_XAI_API_KEY>"
    }
  ]
}

Gemini 2.0 Flash from Google

If you prefer to use a model from Google, then we recommend Gemini 2.0 Flash.

Hub

Add the Gemini 2.0 Flash block from the hub

YAML

config.yaml

models:
  - name: Gemini 2.0 Flash
    provider: gemini
    model: gemini-2.0-flash
    apiKey: <YOUR_GEMINI_API_KEY>

JSON

config.json

{
  "models": [
    {
      "title": "Gemini 2.0 Flash",
      "provider": "gemini",
      "model": "gemini-2.0-flash",
      "apiKey": "<YOUR_GEMINI_API_KEY>"
    }
  ]
}

Local, offline experience

For the best local, offline Chat experience, you will want to use a model that is large but fast enough on your machine.

Llama 3.1 8B

If your local machine can run an 8B parameter model, then we recommend running Llama 3.1 8B on your machine (e.g. using Ollama or LM Studio).

Hub

Ollama

Add the Ollama Llama 3.1 8b block from the hub

YAML

Ollama

config.yaml

models:
  - name: Llama 3.1 8B
    provider: ollama
    model: llama3.1:8b

LM Studio

config.yaml

models:
  - name: Llama 3.1 8B
    provider: lmstudio
    model: llama3.1:8b

JSON

Ollama

config.json

{
  "models": [
    {
      "title": "Llama 3.1 8B",
      "provider": "ollama",
      "model": "llama3.1:8b"
    }
  ]
}

LM Studio

config.json

{
  "models": [
    {
      "title": "Llama 3.1 8B",
      "provider": "lmstudio",
      "model": "llama3.1-8b"
    }
  ]
}

DeepSeek Coder 2 16B

If your local machine can run a 16B parameter model, then we recommend running DeepSeek Coder 2 16B (e.g. using Ollama or LM Studio).

YAML

Ollama

config.yaml

models:
  - name: DeepSeek Coder 2 16B
    provider: ollama
    model: deepseek-coder-v2:16b

LM Studio

config.yaml

models:
  - name: DeepSeek Coder 2 16B
    provider: lmstudio
    model: deepseek-coder-v2:16b

JSON

Ollama

config.json

{
  "models": [
    {
      "title": "DeepSeek Coder 2 16B",
      "provider": "ollama",
      "model": "deepseek-coder-v2:16b",
      "apiBase": "http://localhost:11434"
    }
  ]
}

LM Studio

config.json

{
  "models": [
    {
      "title": "DeepSeek Coder 2 16B",
      "provider": "lmstudio",
      "model": "deepseek-coder-v2:16b"
    }
  ]
}

Other experiences

There are many more models and providers you can use with Chat beyond those mentioned above. Read more here