* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* Fix Issue #157
Add chroma_utils.py to manage chromaDB and added docs for explanation
* Add logging and debugging process
- Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist.
- Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization.
- Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization.
- Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval.
- Improved logging throughout the code to provide more context in error messages, making debugging easier.
* change progress order
* feat: change progress frontend type
* feat: update _load_progress for progress's new format
* fix: fix formatUnderscoreToName
* fix:simplify TrainProgress with direct JSON structure and mapping
* fix：add necessary accessors (getters and setters) to maintain compatibility with existing code while adopting the new data structure.
* fix: train log jump
* fix: fix current step update failed
* fix:fix stop problem
* fix trainprogress has no attribute status problem
* fix: merge master
---------
Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai>
* feat(trainprocess):add receive & get training params
* feat: support low/standard option for L2 data generate
* feat:improve training script invocation with direct bash command and internationalize comments
* fix: restore code
* feat: add error handling for GraphRAG index and modify model save frequency(epoch -> steps)
* feat: support low/medium/high mode for L2 data generation
* feat: set defaut value
* feat: add train params
* feat: use ScriptExecutor for training process and check return code
* fix: fix params round
* feat:use subprocess instead of script_executor to keep logs
* feat:hot fix name 'monitor_result' is not defined
* fix: delete useless message
* fix: fix params status
* feat: add status suspended
* feat:fix stop status
* fix: fix resume status
* fix: disable error
---------
Co-authored-by: Crabboss Mr <1123357821@qq.com>
Co-authored-by: ryangyuan <ryangyuan@mail.mindverse.ai>
* feat: support LongCoT mode for DeepSeek-R1 data synthesis and model training
* fix: restore .env and setting.yaml configuration files
* fix: restore .env from L2 configuration files
---------
Co-authored-by: Xiang Ying <yingxiang835@gmail.com>
This fix:
- Add text chunking in LLMClient to handle long texts
- Remove truncation in document_service.py
- Add EMBEDDING_MAX_TEXT_LENGTH config to control chunk size
- Average embeddings of chunks to maintain semantic representation
The fix ensures that:
1. No content is lost for long texts
2. API requests don't fail due to text length
3. Complete semantic information is preserved
* Potential fix for code scanning alert no. 110: Uncontrolled data used in path expression
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 109: Uncontrolled data used in path expression
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
