* Add code for data filtering llm judge
* Ignore log file created on root (mainly for synthetic_data_generation.log)
* Fix metadata API compatibility issues by commenting out metadata tags in LLM API calls
- Commented out metadata.tags parameters in all LLM API calls across the codebase
- This fixes compatibility issues with custom LLM providers that don't support metadata
- Affects shades generation, topics generation, wiki generation, bio QA, and question generation
- Preserves the original code structure for future re-enabling if needed
* feat: add data filtering pipeline with Ollama integration
- Add MergedDataJudge class for intelligent data filtering using Ollama Gemma
- Integrate automatic Ollama CLI installation into project setup process
- Add DATA_FILTERING step to training pipeline with concurrent processing
- Include testing for MergedDataJudge in its local main() function
- Add Ollama dependency to pyproject.toml
* feat: add automatic Ollama model cleanup after data filtering
* Add logging for outputting data filtering parameters
* fix: adjust error handling for MergedDataJudge:
- Keep original merged.json unchanged when any error occurs
- Exit filtering process immediately on errors instead of continuing with defaults
- Ensure training pipeline continues safely even if data filtering fails
* Add frontend for data filtering pipeline
* resolve data filtering quality_level error by commenting out problematic fields, change TrainProcessService back to original class definition
* fix: quote unquoted shade icons to prevent JSON parsing errors
* Fixed wiki_res.json missing due to no database connection at wiki/base.py module import
* Added scoring reasoning as part of the merged data
* fix: filter ANSI escape sequences from Ollama logs in data filtering step
* fix: Add data filtering steps to cloud training to resolve KeyError
- Added 'Data Filtering' step to cloud training progress holder
- Added data filtering step execution in cloud training service
- Added data filtering parameters to cloud training routes
- Updated frontend to send data filtering parameters
- Fixed missing except clause in cloud training service
This resolves the KeyError: 'data_filtering' when switching from cloud to local training.
* feat: Updated the cloud service model list and fixed the error message in the training process service
* cloud serive add
* add deployed_name
* add name,deploy model
* add model，name
* feat: Update cloud model deployment information, add deployment status and related parameters
* remove qwen2.5
* feat: Updated the model name to Qwen3, adjusted related configurations and default values
* refactor: simplify chat data processing and update model training configuration
* Qwen 3
* feat: Add cloud service status check function to optimize model status management
* chore: update API endpoints and file paths for local development environment
* feat: store complete training parameters in local storage
* Fix/cloud service stop (#384)
* fix cloud service stop
* add pending status
* feature: Added pause status polling function and updated cloud training stop logic
* Stop logic modification
* feature: Added cloud training pause status check function to optimize training process control
* fix stop
* feature: Add language parameter to support multi-language training configuration
* add Chinese language, fix top
---------
Co-authored-by: wyx-hhhh <1360479992@qq.com>
* feat: update originPrompt to simplify response structure and enhance clarity
* load error
---------
Co-authored-by: yanmuyuan <2216646664@qq.com>
Co-authored-by: doubleBlack2 <108928143+doubleBlack2@users.noreply.github.com>
* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* feat(DPO_support): add DPO workflow script and documentation
- Introduce dpo_pipeline.sh for automated DPO process execution.
- Add README with overview and getting started guide (auto & manual modes).
- Include steps for SFT model deployment, data synthesis, training, and LoRA weight merging.
* feat(DPO_support): update API key, global bio config, pipeline script, and README
- Enhanced security by refining API key management.
- Simplified global bio configuration process.
- Improved pipeline script initialization for better reliability.
- Updated README with comprehensive documentation.
* feat(feature/DPO_support): Translate Chinese comments and README to English
- Translated all Chinese comments in the script to English for better readability and consistency.
- Updated the project README to English to ensure clarity and accessibility for all contributors and users.
---------
Co-authored-by: mcx_gt <moral_compass24@163.com>
* feat(trainprocess):add receive & get training params
* feat: support low/standard option for L2 data generate
* feat:improve training script invocation with direct bash command and internationalize comments
* fix: restore code
* feat: add error handling for GraphRAG index and modify model save frequency(epoch -> steps)
* feat: support low/medium/high mode for L2 data generation
* feat: set defaut value
* feat: add train params
* feat: use ScriptExecutor for training process and check return code
* fix: fix params round
* feat:use subprocess instead of script_executor to keep logs
* feat:hot fix name 'monitor_result' is not defined
* fix: delete useless message
* fix: fix params status
* feat: add status suspended
* feat:fix stop status
* fix: fix resume status
* fix: disable error
---------
Co-authored-by: Crabboss Mr <1123357821@qq.com>
Co-authored-by: ryangyuan <ryangyuan@mail.mindverse.ai>
* feat: support LongCoT mode for DeepSeek-R1 data synthesis and model training
* fix: restore .env and setting.yaml configuration files
* fix: restore .env from L2 configuration files
---------
Co-authored-by: Xiang Ying <yingxiang835@gmail.com>
* feat(mlx_training_support): init training pipeline with support for training, merging, serving, and testing
* feat(feature/mlx_training_support): Enhance data transformation script, modify training script, and encapsulate model training parameters in YAML file
- Improved the data conversion script to handle additional edge cases and ensure better data integrity.
- Updated the training script to support both YAML configuration and command-line parameters, with a preference for YAML for LoRA fine-tuning.
- Encapsulated model training parameters in a YAML file to simplify configuration and management.
- Added detailed instructions and explanations in the README.md file to guide users through the data conversion, training, and testing processes.
---------
Co-authored-by: mcx_gt <moral_compass24@163.com>
* Potential fix for code scanning alert no. 110: Uncontrolled data used in path expression
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 109: Uncontrolled data used in path expression
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
