mirror of
https://github.com/mindverse/Second-Me.git
synced 2026-07-20 06:33:26 +00:00
f5bb0dad59
* Add code for data filtering llm judge * Ignore log file created on root (mainly for synthetic_data_generation.log) * Fix metadata API compatibility issues by commenting out metadata tags in LLM API calls - Commented out metadata.tags parameters in all LLM API calls across the codebase - This fixes compatibility issues with custom LLM providers that don't support metadata - Affects shades generation, topics generation, wiki generation, bio QA, and question generation - Preserves the original code structure for future re-enabling if needed * feat: add data filtering pipeline with Ollama integration - Add MergedDataJudge class for intelligent data filtering using Ollama Gemma - Integrate automatic Ollama CLI installation into project setup process - Add DATA_FILTERING step to training pipeline with concurrent processing - Include testing for MergedDataJudge in its local main() function - Add Ollama dependency to pyproject.toml * feat: add automatic Ollama model cleanup after data filtering * Add logging for outputting data filtering parameters * fix: adjust error handling for MergedDataJudge: - Keep original merged.json unchanged when any error occurs - Exit filtering process immediately on errors instead of continuing with defaults - Ensure training pipeline continues safely even if data filtering fails * Add frontend for data filtering pipeline * resolve data filtering quality_level error by commenting out problematic fields, change TrainProcessService back to original class definition * fix: quote unquoted shade icons to prevent JSON parsing errors * Fixed wiki_res.json missing due to no database connection at wiki/base.py module import * Added scoring reasoning as part of the merged data * fix: filter ANSI escape sequences from Ollama logs in data filtering step * fix: Add data filtering steps to cloud training to resolve KeyError - Added 'Data Filtering' step to cloud training progress holder - Added data filtering step execution in cloud training service - Added data filtering parameters to cloud training routes - Updated frontend to send data filtering parameters - Fixed missing except clause in cloud training service This resolves the KeyError: 'data_filtering' when switching from cloud to local training.
86 lines
1.8 KiB
Text
86 lines
1.8 KiB
Text
__pycache__
|
|
.DS_Store
|
|
.idea
|
|
.ipynb_checkpoints
|
|
.pytest_cache
|
|
.ruff_cache
|
|
.vscode/
|
|
.ruff_cache/
|
|
poetry.lock
|
|
.hf_cache/
|
|
.poetry-venv/
|
|
|
|
llama.cpp
|
|
*.ipynb
|
|
data/db/*
|
|
data/chroma_db/*
|
|
data/
|
|
lpm_kernel/L2/base_model/
|
|
lpm_kernel/L2/data_pipeline/output/
|
|
lpm_kernel/L2/data_pipeline/graphrag_indexing/cache/
|
|
lpm_kernel/L2/data_pipeline/raw_data/*
|
|
!lpm_kernel/L2/data_pipeline/raw_data/.gitkeep
|
|
lpm_kernel/L2/data_pipeline/tmp/
|
|
lpm_kernel/L2/output_models/
|
|
|
|
data/sqlite/*
|
|
data/uploads/*
|
|
data/progress/*
|
|
lpm_frontend/node_modules
|
|
lpm_frontend/yarn.lock
|
|
job_id.json
|
|
|
|
# L2 Model Storage
|
|
resources/model/output/merged_model/*
|
|
!resources/model/output/merged_model/.gitkeep
|
|
resources/model/output/personal_model/*
|
|
!resources/model/output/personal_model/.gitkeep
|
|
resources/model/output/*.json
|
|
resources/model/output/*.gguf
|
|
!resources/model/output/.gitkeep
|
|
|
|
|
|
resources/L1/processed_data/subjective/*
|
|
!resources/L1/processed_data/subjective/.gitkeep
|
|
resources/L1/processed_data/objective/*
|
|
!resources/L1/processed_data/objective/.gitkeep
|
|
resources/L1/graphrag_indexing_output/report/*
|
|
resources/L1/graphrag_indexing_output/subjective/*
|
|
|
|
|
|
resources/raw_content/*
|
|
!resources/raw_content/.gitkeep
|
|
|
|
# Base model storage address
|
|
resources/L2/base_model/*
|
|
!resources/L2/base_model/.gitkeep
|
|
resources/L2/data_pipeline/raw_data/*
|
|
!resources/L2/data_pipeline/raw_data/.gitkeep
|
|
|
|
|
|
resources/model/processed_data/L1/processed_data/objective/*
|
|
resources/model/processed_data/L1/processed_data/subjective/*
|
|
|
|
resources/L2/data/*
|
|
!resources/L2/data/.gitkeep
|
|
|
|
resources/model/output/gguf/*
|
|
!resources/model/output/gguf/.gitkeep
|
|
resources/L2/base_models/*
|
|
logs/*.log
|
|
logs/train/*.log
|
|
*.log
|
|
|
|
# Runtime files
|
|
run/*
|
|
!run/.gitkeep
|
|
|
|
#config
|
|
.backend.pid
|
|
.frontend.pid
|
|
logs/train/
|
|
llama_cpp_backup/llama.cpp.zip
|
|
scripts/check_cuda_status.ps1
|
|
scripts/test_cuda_detection.bat
|
|
.env
|
|
.gpu_selected
|