The token immediately before an eot token was lost when SSE streaming
was enabled if that token was contained entirely within a stop sequence.
As an example of when this could happen, consider this prompt:
Type the phrase 'pleas' once.
In a Llama 3-derived model, 'pleas' tokenizes as 'ple' 'as'. The token
'as' is contained within this instruct mode stop sequence:
<|eot_id|><|start_header_id|>assistant<|end_header_id|>
due to the word 'assistant'. Since `string_contains_sequence_substring`
returns True for 'as', this token is added to `tokenReserve` instead of
being streamed immediately. If the '<|eot_id|>' token was generated
next, the text in `tokenReserve` would be discarded.
* Henk's version of the fsize algo
This is the current version of the fsize algo based on Pyro's algorithm with added padding.
* Update koboldcpp.py
Add debugs and bump padding
* Pyro version
Pyro didn't agree with my version, so here is a test with his version
* Polish new auto layers
This one cleans up some debug prints, restores the max behavior in case the old alg suits someone better and changes the 200 layers to be the actual max for all backends so users have a better feel for the models.
* Remove 10% margin
The new version has been much more accurate, for low vram systems I only notice 1 layer difference. Getting rid of it so users can test if its still in safe margins like I expect. On a 6GB system it results in 18 layers instead of 17 being chosen for Tiefighter.
* Restore 500MB buffer to play it safe
I'm not feeling confident most people keep their vram usage under 1GB with background tasks. For now since we are aiming to have it work on as many systems as possible I restore the 500MB extra space since the fsize inflation is gone.
* Cap layers at maximum
When using the auto predict we don't want to go over the maximum amount of layers. Users should have a realistic feel for how large the model is.
For example when I was using the new auto guesser to communicate if a larger model would fit on someone's system at a higher context, it originally made me think that the model had 60 layers. In reality it had less.
This commit will take the layers of the model, and add 3 extra since that is the highest amount of additional layers a backend adds for the context handling (Most its 1).
* Remove old max layer code
Turns out at extreme contexts on new models such as Nemo the old code is incorrectly assuming we can offload everything. Its also redundant to check for max layers the old way since I capped our new guesses.
Old code is now removed to simplify it, and it changed the nemo guess from 43 layers to 15 layers. Still looking into the 15 part, still seems to high but can be the old algo taking over.
* Restructure algorithm into multiple parts
As requested the different calculations in the algorithm now have their own sections and names so its easier to understand what parts are being used. This also fixes the typo that was caused as a result of it being harder to read, the typo made no difference during execution and the algorithm is confirmed to still work the same.
* Rudimentary support of openai chat completions tools calls
-Most small models are not smart enough to do this, especially a combined tool call + role play response, but at least this allows experimentation along these lines with koboldcpp
* try to also support specified function and tool choice set to none
Allow tools start and end messages to be configured in adapter
Try to force grammar to specific function call if specified (untested)
* ensure tools get listed right after user content and before end of user message content
* omit grammars approach try prompting instead
-use more extensive json parsing and direct instructions to models to try to obtain the desired result
-seems to work relatively well with Mistral-7B-Instruct-v.0.3.Q4_K_M.gguf and neuralhermes-2.5-mistral-7b.Q4_K_M.gguf
-question of whether this is too opinionated of an approach, should the instructions be things that can be passed with the prompt template?
* add back llamacpp recommended json grammar
Go back to adding grammar but use "official" llamacpp grammar only not a custom one just for openai
* Tidy up, remove unnecessary globals
* clarity
* fix missing local variable error
This worked to fix the error I mentioned on my last comment
---------
Co-authored-by: Concedo <39025047+LostRuins@users.noreply.github.com>
