* Add code for data filtering llm judge
* Ignore log file created on root (mainly for synthetic_data_generation.log)
* Fix metadata API compatibility issues by commenting out metadata tags in LLM API calls
- Commented out metadata.tags parameters in all LLM API calls across the codebase
- This fixes compatibility issues with custom LLM providers that don't support metadata
- Affects shades generation, topics generation, wiki generation, bio QA, and question generation
- Preserves the original code structure for future re-enabling if needed
* feat: add data filtering pipeline with Ollama integration
- Add MergedDataJudge class for intelligent data filtering using Ollama Gemma
- Integrate automatic Ollama CLI installation into project setup process
- Add DATA_FILTERING step to training pipeline with concurrent processing
- Include testing for MergedDataJudge in its local main() function
- Add Ollama dependency to pyproject.toml
* feat: add automatic Ollama model cleanup after data filtering
* Add logging for outputting data filtering parameters
* fix: adjust error handling for MergedDataJudge:
- Keep original merged.json unchanged when any error occurs
- Exit filtering process immediately on errors instead of continuing with defaults
- Ensure training pipeline continues safely even if data filtering fails
* Add frontend for data filtering pipeline
* resolve data filtering quality_level error by commenting out problematic fields, change TrainProcessService back to original class definition
* fix: quote unquoted shade icons to prevent JSON parsing errors
* Fixed wiki_res.json missing due to no database connection at wiki/base.py module import
* Added scoring reasoning as part of the merged data
* fix: filter ANSI escape sequences from Ollama logs in data filtering step
* fix: Add data filtering steps to cloud training to resolve KeyError
- Added 'Data Filtering' step to cloud training progress holder
- Added data filtering step execution in cloud training service
- Added data filtering parameters to cloud training routes
- Updated frontend to send data filtering parameters
- Fixed missing except clause in cloud training service
This resolves the KeyError: 'data_filtering' when switching from cloud to local training.
* feat: Updated the cloud service model list and fixed the error message in the training process service
* cloud serive add
* add deployed_name
* add name,deploy model
* add model，name
* feat: Update cloud model deployment information, add deployment status and related parameters
* remove qwen2.5
* feat: Updated the model name to Qwen3, adjusted related configurations and default values
* refactor: simplify chat data processing and update model training configuration
* Qwen 3
* feat: Add cloud service status check function to optimize model status management
* chore: update API endpoints and file paths for local development environment
* feat: store complete training parameters in local storage
* Fix/cloud service stop (#384)
* fix cloud service stop
* add pending status
* feature: Added pause status polling function and updated cloud training stop logic
* Stop logic modification
* feature: Added cloud training pause status check function to optimize training process control
* fix stop
* feature: Add language parameter to support multi-language training configuration
* add Chinese language, fix top
---------
Co-authored-by: wyx-hhhh <1360479992@qq.com>
* feat: update originPrompt to simplify response structure and enhance clarity
* load error
---------
Co-authored-by: yanmuyuan <2216646664@qq.com>
Co-authored-by: doubleBlack2 <108928143+doubleBlack2@users.noreply.github.com>
* fix cloud service stop
* add pending status
* feature: Added pause status polling function and updated cloud training stop logic
* Stop logic modification
* feature: Added cloud training pause status check function to optimize training process control
* fix stop
* feature: Add language parameter to support multi-language training configuration
* add Chinese language, fix top
---------
Co-authored-by: wyx-hhhh <1360479992@qq.com>
* Enhance GGUF model handling with timestamps, metadata and memory training status
* Check if is_trained exists
* fix
* cloud service
* Change the data type of the is_trained field to boolean and update the related logic to reflect this change
* Change the data type of the is_trained field to boolean and update the related logic to reflect this change
* Add gguf path to json file
* Added model selection function, updated model list acquisition logic, and enhanced model information display
* Update the model service startup logic, add integrity check for the model path, and support obtaining the model path from different fields
* Service Change
* full cloud service
* feat: implement async cloud training process with job tracking and API key management
* Progress bar modification
* feat: Add Local and Cloud Training Configuration Components
- Introduced LocalTrainingConfig component for configuring local training parameters.
- Updated TrainingConfiguration component to include tabs for Local and Cloud training configurations.
- Added API functions for setting and getting cloud service API keys.
- Created useCloudProviderStore for managing cloud provider configurations.
- Enhanced event utility to include a new event for showing cloud provider modal.
* Refactor cloud provider and training configuration components
- Updated CloudProviderModal to handle cloud service API key management.
- Replaced API key handling with model configuration updates in CloudProviderModal.
- Enhanced CloudTrainingConfig to manage cloud models based on API key availability.
- Introduced new cloud service functions for listing available models and managing training jobs.
- Modified LocalTrainingConfig to ensure default model selection and synchronization.
- Updated TrainingConfiguration to manage model switching between local and cloud environments.
- Refactored useCloudProviderStore to integrate cloud service API key handling.
- Adjusted useTrainingStore to prioritize model name selection based on the active environment.
* Stream Output
* feat: Enhance training configuration and progress components
- Updated LocalTrainingConfig to improve default model handling and avoid unnecessary updates.
- Introduced LocalTrainingProgress component to manage local training progress display.
- Refactored TrainingConfiguration to support both local and cloud training types, including updated button text and actions.
- Modified TrainingProgress to conditionally render local or cloud training progress based on the selected training type.
- Added cloud service functions for starting training and managing job information.
- Adjusted training parameter interfaces to ensure consistency across local and cloud models.
* Stream response change
* feat: Enhance cloud training and inference capabilities
- Updated TrainingProgress component to handle cloud training progress data and job ID.
- Modified trainExposureModel to allow nullable path and added optional stageName.
- Enhanced useSSE hook to support cloud model inference with new parameters.
- Introduced CloudProgressData type to align cloud training progress with local training structure.
- Implemented cloud inference request handling with local knowledge retrieval in cloudService.
- Added utility functions for managing active cloud model state in cloudModelUtils.
- Updated cloud inference endpoint to support local knowledge retrieval before cloud inference.
- Refactored advanced chat service to utilize new message structure for cloud inference.
- Enhanced prompt strategies to incorporate knowledge retrieval based on user messages.
* feat: Delete the training parameter debugging information component
* Resume training at breakpoint
* Repair data redundancy
* Stop system modification
* fix error: reset training
* fix stop and reset
* Change chat reply format
* Enhance cloud and local service management with status tracking and improved progress reporting
- Implemented service status file management in cloud and local services to track active status and model information.
- Added endpoints to start and stop cloud services, including validation for existing services.
- Enhanced local service management with status checks and progress updates during document processing and chunk embedding.
- Introduced real-time progress tracking for document embedding and chunk processing, allowing for incremental updates.
- Improved error handling and logging throughout the service management processes.
- Refactored chat request handling to intelligently route between local and cloud services based on current status.
* feat:Cleaned up code comment
* translate Chinese comments to English in cloud service modules
* translate into chinese
* feat: Enhance cloud provider configuration and training management with API key handling and tab switching logic
* bug fix
* Add is_trained field modification in the cloud
* feat: Refactor training parameters management to separate local and cloud configurations
* feat: Update training parameter types to improve type safety and consistency
* feat: Add data synthesis mode to cloud training parameters and update related components
* feat: The document embedding part is restored to its original state
* refactor: optimize cloud training process with improved stop handling and file path updates
* feat: Update the default values and merging logic of cloud training parameters to ensure parameter consistency
* feat: Add API key preloading function to optimize the loading experience when the modal box is opened
* feat: Optimize CloudProviderModal component, add API key preloading and state management
* fix: Simplify cloud provider display by removing conditional rendering for Alibaba Cloud
* feat: Update .gitignore to include job_id.json and add .gitkeep for gguf directory
---------
Co-authored-by: wyx-hhhh <1360479992@qq.com>
* feat:add current file to model download progress
* git ignore yarn
* feat: show current downloading
---------
Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai>
Co-authored-by: kevinaimonster <kevinaimonster@gmail.com>
* Join AI Network -> Export your Second Me
* Default Synthesis Mode -> high
Default Epoch -> 3
* Set Thinking-Mode Default Value
* Better Display Of ReadMe
* default value of thinking mode
* Set Default value of enableL0Retrival to false
* feature: use uv to setup python environment
* TrainProcessService add singleten method: get_instance
* feat: fix code
* Added CUDA support (#228)
* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* fix: train service singlten
---------
Co-authored-by: Zachary Pitroda <30330004+zpitroda@users.noreply.github.com>
* Add CUDA support
- CUDA detection
- Memory handling
- Ollama model release after training
* Fix logging issue
added cuda support flag so log accurately reflected cuda toggle
* Update llama.cpp rebuild
Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run
* Improved vram management
Enabled memory pinning and optimizer state offload
* Fix CUDA check
rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support
* Added fast restart and fixed CUDA check command
Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild
Fixed make docker-check-cuda command to correctly reflect cuda support
* Added docker-compose.gpu.yml
Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification
* Fixed cuda toggle
Last push accidentally broke cuda toggle
* Code review fixes
Fixed errors resulting from removed code:
- Added return save_path to end of save_hf_model function
- Rolled back download_file_with_progress function
* Update Makefile
Use cuda by default when using docker-restart-backend-fast
* Minor cleanup
Removed unnecessary makefile command and fixed gpu logging
* Delete .gpu_selected
* Simplified cuda training code
- Removed dtype setting to let torch automatically handle it
- Removed vram logging
- Removed Unnecessary/old comments
* Fixed gpu/cpu selection
Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu
* Fix Ollama embedding error
Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones
* Fixed model selection & memory error
Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly
* Fix Issue #157
Add chroma_utils.py to manage chromaDB and added docs for explanation
* Add logging and debugging process
- Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist.
- Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization.
- Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization.
- Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval.
- Improved logging throughout the code to provide more context in error messages, making debugging easier.
* split train process service
* split train process service
* split train param
* save train param to file
---------
Co-authored-by: kevin-mindverse <kevin@mindverse.ai>
