mirror of
https://github.com/vegu-ai/talemate.git
synced 2025-09-04 03:19:16 +00:00
35 lines
1.2 KiB
Markdown
35 lines
1.2 KiB
Markdown
## ChromaDB
|
|
|
|
If you want chromaDB to use the more accurate (but much slower) instructor embeddings add the following to `config.yaml`:
|
|
|
|
```yaml
|
|
chromadb:
|
|
embeddings: instructor
|
|
instructor_device: cpu
|
|
instructor_model: hkunlp/instructor-xl
|
|
```
|
|
|
|
You will need to restart the backend for this change to take effect.
|
|
|
|
Note that the first time you do this it will need to download the instructor model you selected. This may take a while, and the talemate backend will be un-responsive during that time.
|
|
|
|
Once the download is finished, if talemate is still un-responsive, try reloading the front-end to reconnect. When all fails just restart the backend as well.
|
|
|
|
### GPU support
|
|
|
|
If you want to use the instructor embeddings with GPU support, you will need to install pytorch with CUDA support.
|
|
|
|
To do this on windows, run `install-pytorch-cuda.bat` from the project root. Then change your device in the config to `cuda`:
|
|
|
|
```yaml
|
|
chromadb:
|
|
embeddings: instructor
|
|
instructor_device: cuda
|
|
instructor_model: hkunlp/instructor-xl
|
|
```
|
|
|
|
Instructor embedding models:
|
|
|
|
- `hkunlp/instructor-base` (smallest / fastest)
|
|
- `hkunlp/instructor-large`
|
|
- `hkunlp/instructor-xl` (largest / slowest) - requires about 5GB of GPU memory
|