* Add code for data filtering llm judge
* Ignore log file created on root (mainly for synthetic_data_generation.log)
* Fix metadata API compatibility issues by commenting out metadata tags in LLM API calls
- Commented out metadata.tags parameters in all LLM API calls across the codebase
- This fixes compatibility issues with custom LLM providers that don't support metadata
- Affects shades generation, topics generation, wiki generation, bio QA, and question generation
- Preserves the original code structure for future re-enabling if needed
* feat: add data filtering pipeline with Ollama integration
- Add MergedDataJudge class for intelligent data filtering using Ollama Gemma
- Integrate automatic Ollama CLI installation into project setup process
- Add DATA_FILTERING step to training pipeline with concurrent processing
- Include testing for MergedDataJudge in its local main() function
- Add Ollama dependency to pyproject.toml
* feat: add automatic Ollama model cleanup after data filtering
* Add logging for outputting data filtering parameters
* fix: adjust error handling for MergedDataJudge:
- Keep original merged.json unchanged when any error occurs
- Exit filtering process immediately on errors instead of continuing with defaults
- Ensure training pipeline continues safely even if data filtering fails
* Add frontend for data filtering pipeline
* resolve data filtering quality_level error by commenting out problematic fields, change TrainProcessService back to original class definition
* fix: quote unquoted shade icons to prevent JSON parsing errors
* Fixed wiki_res.json missing due to no database connection at wiki/base.py module import
* Added scoring reasoning as part of the merged data
* fix: filter ANSI escape sequences from Ollama logs in data filtering step
* fix: Add data filtering steps to cloud training to resolve KeyError
- Added 'Data Filtering' step to cloud training progress holder
- Added data filtering step execution in cloud training service
- Added data filtering parameters to cloud training routes
- Updated frontend to send data filtering parameters
- Fixed missing except clause in cloud training service
This resolves the KeyError: 'data_filtering' when switching from cloud to local training.
* feature: add China image configuration options
* feature: update Python and Poetry mirror configurations to use Tsinghua and Aliyun
* feature: improve Poetry mirror configuration to check for existing sources
* feature: rename setup-china to setup-cn and update pip mirror to Aliyun
* Add MD file
* Modify GitHub download
* Change to Chinese Mainland
---------
Co-authored-by: yanmuyuan <2216646664@qq.com>
