f5bb0dad59
* Add code for data filtering llm judge * Ignore log file created on root (mainly for synthetic_data_generation.log) * Fix metadata API compatibility issues by commenting out metadata tags in LLM API calls - Commented out metadata.tags parameters in all LLM API calls across the codebase - This fixes compatibility issues with custom LLM providers that don't support metadata - Affects shades generation, topics generation, wiki generation, bio QA, and question generation - Preserves the original code structure for future re-enabling if needed * feat: add data filtering pipeline with Ollama integration - Add MergedDataJudge class for intelligent data filtering using Ollama Gemma - Integrate automatic Ollama CLI installation into project setup process - Add DATA_FILTERING step to training pipeline with concurrent processing - Include testing for MergedDataJudge in its local main() function - Add Ollama dependency to pyproject.toml * feat: add automatic Ollama model cleanup after data filtering * Add logging for outputting data filtering parameters * fix: adjust error handling for MergedDataJudge: - Keep original merged.json unchanged when any error occurs - Exit filtering process immediately on errors instead of continuing with defaults - Ensure training pipeline continues safely even if data filtering fails * Add frontend for data filtering pipeline * resolve data filtering quality_level error by commenting out problematic fields, change TrainProcessService back to original class definition * fix: quote unquoted shade icons to prevent JSON parsing errors * Fixed wiki_res.json missing due to no database connection at wiki/base.py module import * Added scoring reasoning as part of the merged data * fix: filter ANSI escape sequences from Ollama logs in data filtering step * fix: Add data filtering steps to cloud training to resolve KeyError - Added 'Data Filtering' step to cloud training progress holder - Added data filtering step execution in cloud training service - Added data filtering parameters to cloud training routes - Updated frontend to send data filtering parameters - Fixed missing except clause in cloud training service This resolves the KeyError: 'data_filtering' when switching from cloud to local training. |
||
|---|---|---|
| .github/ISSUE_TEMPLATE | ||
| dependencies | ||
| docker | ||
| docs | ||
| images | ||
| integrate | ||
| logs | ||
| lpm_frontend | ||
| lpm_kernel | ||
| mcp | ||
| resources | ||
| run | ||
| scripts | ||
| .env | ||
| .gitattributes | ||
| .gitignore | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| docker-compose-gpu.yml | ||
| docker-compose.yml | ||
| Dockerfile.backend | ||
| Dockerfile.backend.apple | ||
| Dockerfile.backend.cuda | ||
| Dockerfile.frontend | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| README_ja.md | ||
| SECURITY.md | ||
Our Vision
Companies like OpenAI built "Super AI" that threatens human independence. We crave individuality: AI that amplifies, not erases, YOU.
We’re challenging that with "Second Me": an open-source prototype where you craft your own AI self—a new AI species that preserves you, delivers your context, and defends your interests.
It’s locally trained and hosted—your data, your control—yet globally connected, scaling your intelligence across an AI network. Beyond that, it’s your AI identity interface—a bold standard linking your AI to the world, sparks collaboration among AI selves, and builds tomorrow’s truly native AI apps.
Tech enthusiasts, AI pros, domain experts, Join us! Second Me is your launchpad to extend your mind into the digital horizon.
Key Features
Train Your AI Self with AI-Native Memory (Paper)
Start training your Second Me today with your own memories! Using Hierarchical Memory Modeling (HMM) and the Me-Alignment Algorithm, your AI self captures your identity, understands your context, and reflects you authentically.
Scale Your Intelligence on the Second Me Network
Launch your AI self from your laptop onto our decentralized network—anyone or any app can connect with your permission, sharing your context as your digital identity.
Build Tomorrow’s Apps with Second Me
Roleplay: Your AI self switches personas to represent you in different scenarios.
AI Space: Collaborate with other Second Mes to spark ideas or solve problems.
100% Privacy and Control
Unlike traditional centralized AI systems, Second Me ensures that your information and intelligence remain local and completely private.
Getting started & staying tuned with us
Star and join us, and you will receive all release notifications from GitHub without any delay!
Quick Start
📊 Model Size vs. Memory (Reference Guide)
Note: "B" in the table represents "billion parameters model". Data shown are examples only; actual supported model sizes may vary depending on system optimization, deployment environment, and other hardware/software conditions.
| Memory (GB) | Docker Deployment (Windows/Linux) | Docker Deployment (Mac) | Integrated Setup (Windows/Linux) | Integrated Setup (Mac) |
|---|---|---|---|---|
| 8 | ~0.5B | ~0.5B | ~1.5B | ~0.5B |
| 16 | 1.5B | 0.5B | ~3.0B | ~0.5B |
| 32 | ~3B | ~1.5B | ~3B | ~1.5B |
Note
: Models below 0.5B may not provide satisfactory performance for complex tasks. And we're continuously improving cross-platform support - please submit an issue for feedback or compatibility problems on different operating systems.
MLX Acceleration: Mac M-series users can use MLX to run larger models (CLI-only).
⚡ Get your Second Me running in just 3 steps:
# 1. Clone the repository
git clone https://github.com/mindverse/Second-Me.git
cd Second-Me
# 2. Start Docker containers
make docker-up
# 3. Access the web interface
# Open your browser and visit: http://localhost:3000
👉 For detailed instructions — including integrated (non-Docker) setup, model selection, memory requirements, and platform-specific tips, check the full Deployment Guide on GitBook.
❓ Got questions about setup, models, or any troubleshooting? Check our FAQ.
Tutorial and Use Cases
🛠️ Feel free to follow User tutorial to build your Second Me.
💡 Check out the links below to see how Second Me can be used in real-life scenarios:
- Felix AMA (Roleplay app)
- Brainstorming a 15-Day European City Itinerary (Network app)
- Icebreaking as a Speed Dating Match (Network app)
What's Next: May 2025
Second Me continues to evolve as the open-source identity infrastructure for AI. Here's what's on deck for May:
- 🗂️ Version Control: Smarter versioning of memory and identity states
- 🧠 Continuous Training Pipelines: Keep your AI self evolving over time, with ongoing updates based on new memory inputs.
- ⚙️ Performance & Stability Improvements: Enhancements across inference ability, model alignment, and base model upgrades
- ☁️ Cloud Solutions: Explore cloud-based solutions for both model training (fine-tuning) and model deployment, to reduce the hardware burden on users' local machines.
Contributing
We’d love for you to help shape what’s coming next — whether it’s fixing bugs, building new features, or improving docs.
- 📘 Check out our Contribution Guide to get started
- 💻 Submit ideas, issues, or PRs on GitHub
- 💬 Join the conversation and stay updated in our Discord — it’s where the community lives
Contributors
We would like to express our gratitude to all the individuals who have contributed to Second Me! If you're interested in contributing to the future of intelligence uploading, whether through code, documentation, or ideas, please feel free to submit a pull request to our repository: Second-Me.
Made with contrib.rocks.
Acknowledgements
This work leverages the power of the open-source community.
For data synthesis, we utilized GraphRAG from Microsoft.
For model deployment, we utilized llama.cpp, which provides efficient inference capabilities.
Our base models primarily come from the Qwen2.5 series.
We also want to extend our sincere gratitude to all users who have experienced Second Me. We recognize that there is significant room for optimization throughout the entire pipeline, and we are fully committed to iterative improvements to ensure everyone can enjoy the best possible experience locally.
License
Second Me is open source software licensed under the Apache License 2.0. See the LICENSE file for more details.