Since Ollama/llama.cpp do not currently YARN for context length
extension, the context length is limited to `32768`. This can be
confirmed by the Ollama model card.
See corresponding issue on Ollama repo :
https://github.com/ollama/ollama/issues/6865
Co-authored-by: Patrick Samson <1416027+patricksamson@users.noreply.github.com>
This removes the `low_speed_timeout` setting from all providers as a
response to issue #19509.
Reason being that the original `low_speed_timeout` was only as part of
#9913 because users wanted to _get rid of timeouts_. They wanted to bump
the default timeout from 5sec to a lot more.
Then, in the meantime, the meaning of `low_speed_timeout` changed in
#19055 and was changed to a normal `timeout`, which is a different thing
and breaks slower LLMs that don't reply with a complete response in the
configured timeout.
So we figured: let's remove the whole thing and replace it with a
default _connect_ timeout to make sure that we can connect to a server
in 10s, but then give the server as long as it wants to complete its
response.
Closes#19509
Release Notes:
- Removed the `low_speed_timeout` setting from LLM provider settings,
since it was only used to _increase_ the timeout to give LLMs more time,
but since we don't have any other use for it, we simply remove the
setting to give LLMs as long as they need.
---------
Co-authored-by: Antonio <antonio@zed.dev>
Co-authored-by: Peter Tripp <peter@zed.dev>
This PR is a refactor to pave the way for allowing the user to view and
edit workflow step resolutions. I've made tool calls work more like
normal streaming completions for all providers. The `use_any_tool`
method returns a stream of strings (which contain chunks of JSON). I've
also done some minor cleanup of language model providers in general,
removing the duplication around handling streaming responses.
Release Notes:
- N/A
- [x] OpenAI
- [ ] ~Google~ Moved into a separate branch at:
https://github.com/zed-industries/zed/tree/tool-calls-in-google-ai I've
ran into issues with having the API digest our schema without tripping
over itself - the function call parameters are malformed and whatnot. We
can resume from that branch if needed.
- [x] Ollama
- [x] Cloud
- [ ] ~Copilot Chat (?)~
Release Notes:
- Added tool calling capabilities to OpenAI and Ollama models.
This adds the ability to set the keep alive as an integer, including
`-1` for staying alive indefinitely until a new model is loaded or
Ollama exits. I've also set the default to `-1` so that models stay
ready to go for Zed to use.
Release Notes:
- N/A
This PR adds a missing LICENSE file to the recently-added `ollama`
crate.
Also added the missing `lints.workspace = true` to the `Cargo.toml`.
Release Notes:
- N/A
Closes#4424.
A few design decisions that may need some rethinking or later PRs:
* Other providers have a check for authentication. I use this
opportunity to fetch the models which doubles as a way of finding out if
the Ollama server is running.
* Ollama has _no_ API for getting the max tokens per model
* Ollama has _no_ API for getting the current token count
https://github.com/ollama/ollama/issues/1716
* Ollama does allow setting the `num_ctx` so I've defaulted this to
4096. It can be overridden in settings.
* Ollama models will be "slow" to start inference because they're
loading the model into memory. It's faster after that. There's no UI
affordance to show that the model is being loaded.
Release Notes:
- Added an Ollama Provider for the assistant. If you have
[Ollama](https://ollama.com/) running locally on your machine, you can
enable it in your settings under:
```jsonc
"assistant": {
"version": "1",
"provider": {
"name": "ollama",
// Recommended setting to allow for model startup
"low_speed_timeout_in_seconds": 30,
}
}
```
Chat like usual
<img width="1840" alt="image"
src="https://github.com/zed-industries/zed/assets/836375/4e0af266-4c4f-4d9e-9d74-1a91f76a12fe">
Interact with any model from the [Ollama
Library](https://ollama.com/library)
<img width="587" alt="image"
src="https://github.com/zed-industries/zed/assets/836375/87433ac6-bf87-4a99-89e1-96a93bf8de8a">
Open up the terminal to download new models via `ollama pull`:
