* feature: use uv to setup python environment
* TrainProcessService add singleten method: get_instance
* feat: fix code
* Added CUDA support (#228)
* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* fix: train service singlten
---------
Co-authored-by: Zachary Pitroda <30330004+zpitroda@users.noreply.github.com>
* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* Fix Issue #157
Add chroma_utils.py to manage chromaDB and added docs for explanation
* Add logging and debugging process
- Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist.
- Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization.
- Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization.
- Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval.
- Improved logging throughout the code to provide more context in error messages, making debugging easier.
* split train process service
* split train process service
* split train param
* save train param to file
---------
Co-authored-by: kevin-mindverse <kevin@mindverse.ai>
* change progress order
* feat: change progress frontend type
* feat: update _load_progress for progress's new format
* fix: fix formatUnderscoreToName
* fix:simplify TrainProgress with direct JSON structure and mapping
* fix：add necessary accessors (getters and setters) to maintain compatibility with existing code while adopting the new data structure.
* fix: train log jump
* fix: fix current step update failed
* fix:fix stop problem
* fix trainprogress has no attribute status problem
* fix: merge master
---------
Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai>
* feat(DPO_support): add DPO workflow script and documentation
- Introduce dpo_pipeline.sh for automated DPO process execution.
- Add README with overview and getting started guide (auto & manual modes).
- Include steps for SFT model deployment, data synthesis, training, and LoRA weight merging.
* feat(DPO_support): update API key, global bio config, pipeline script, and README
- Enhanced security by refining API key management.
- Simplified global bio configuration process.
- Improved pipeline script initialization for better reliability.
- Updated README with comprehensive documentation.
* feat(feature/DPO_support): Translate Chinese comments and README to English
- Translated all Chinese comments in the script to English for better readability and consistency.
- Updated the project README to English to ensure clarity and accessibility for all contributors and users.
---------
Co-authored-by: mcx_gt <moral_compass24@163.com>
* feat(trainprocess):add receive & get training params
* feat: support low/standard option for L2 data generate
* feat:improve training script invocation with direct bash command and internationalize comments
* fix: restore code
* feat: add error handling for GraphRAG index and modify model save frequency(epoch -> steps)
* feat: support low/medium/high mode for L2 data generation
* feat: set defaut value
* feat: add train params
* feat: use ScriptExecutor for training process and check return code
* fix: fix params round
* feat:use subprocess instead of script_executor to keep logs
* feat:hot fix name 'monitor_result' is not defined
* fix: delete useless message
* fix: fix params status
* feat: add status suspended
* feat:fix stop status
* fix: fix resume status
* fix: disable error
---------
Co-authored-by: Crabboss Mr <1123357821@qq.com>
Co-authored-by: ryangyuan <ryangyuan@mail.mindverse.ai>
