mirror of
https://github.com/MODSetter/SurfSense.git
synced 2025-09-01 18:19:08 +00:00
Update README.md
This commit is contained in:
Rohan Verma
GitHub
parent
7f38091d3d
commit
0e954faa29
1 changed files with 94 additions and 123 deletions
217
README.md
217
README.md
|
|
@ -1,144 +1,58 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# SurfSense
|
||||
|
||||
Well when I’m browsing the internet, I tend to save a ton of content—but remembering when and what you saved? Total brain freeze! That’s where SurfSense comes in. SurfSense is like a Knowledge Graph Brain 🧠 for anything you see (Social Media Chats, Calender Invites, Important Mails, Tutorials, Recipies and anything ) on the World Wide Web. Now, you’ll never forget any browsing session. Easily capture your web browsing session and desired webpage content using an easy-to-use cross browser extension. Then, ask your personal knowledge base anything about your saved content, and voilà—instant recall!
|
||||
Well when I’m browsing the internet, I tend to save a ton of content—but remembering when and what you saved? Total brain freeze! That’s where SurfSense comes in. SurfSense is a Personal AI Assistant for anything you see (Social Media Chats, Calender Invites, Important Mails, Tutorials, Recipies and anything ) on the World Wide Web. Now, you’ll never forget any browsing session. Easily capture your web browsing session and desired webpage content using an easy-to-use cross browser extension. Then, ask your personal knowledge base anything about your saved content, and voilà—instant recall!
|
||||
|
||||
# Video
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/37985a8b-acbd-4fff-b276-512bbf0bf6aa
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/f9c49698-f868-4a66-9601-16d375eaad64
|
||||
|
||||
|
||||
|
||||
## Key Features
|
||||
|
||||
- 💡 **Idea**: Save any content you see on the internet in your own Knowledge Graph.
|
||||
- 💡 **Idea**: Save any content you see on the internet in your own personal knowledge base.
|
||||
- ⚙️ **Cross Browser Extension**: Save content from your favourite browser.
|
||||
- 🔍 **Powerful Search**: Quickly find anything in your Web Browsing Sessions.
|
||||
- 💬 **Chat with your Web History**: Interact in Natural Language with your saved Web Browsing Sessions.
|
||||
- 🔔 **Local LLM Support**: Works Flawlessly with Ollama local LLMs.
|
||||
- 🏠 **Self Hostable**: Open source and easy to deploy locally.
|
||||
- 📊 **Use GraphRAG**: Utilize the power of GraphRAG to find meaningful relations in your saved content.
|
||||
- 🔟% **Cheap On Wallet**: Works Flawlessly with OpenAI gpt-4o-mini model.
|
||||
- 🕸️ **No WebScraping**: Extension directly reads the data from DOM.
|
||||
- 🔔 **Automatic Important Notifications**: Get critical notifications such as important meetings, invites etc.
|
||||
- 📊 **Advanced RAG Techniques**: Utilize the power of Advanced RAG Techniques.
|
||||
- 🔟% **Cheap On Wallet**: Works Flawlessly with OpenAI gpt-4o-mini model and Ollama local LLMs.
|
||||
- 🕸️ **No WebScraping**: Extension directly reads the data from DOM to get accurate data.
|
||||
|
||||
## How to get started?
|
||||
**UPDATE 20 SEPTEMBER 2024:**
|
||||
|
||||
Before we begin, we need to set up our Neo4j Graph Database. This is where SurfSense stores all your saved information. For a quick setup, I suggest getting your free Neo4j Aura DB from [https://neo4j.com/cloud/platform/aura-graph-database/](https://neo4j.com/cloud/platform/aura-graph-database/) or setting it up locally.
|
||||
- SursSense now works on Hierarchical Indices.
|
||||
- Knowledge Graph dependency is removed for now until I find some better Graph RAG solutions.
|
||||
- Added support for Local LLMs
|
||||
|
||||
After obtaining your Neo4j credentials, make sure to get your OpenAI API Key from [https://platform.openai.com/](https://platform.openai.com/).
|
||||
Until I find a good host for my backend you need to setup SurfSense locally for now.
|
||||
|
||||
**UPDATE 24 AUGUST 2024:** Extension code is now migrated to Plasmo. You can use extension in any webbrowser. All Webstore links will be updated soon.
|
||||
|
||||
1. Register Your SurfSense account at https://www.surfsense.net/signup
|
||||
2. Download SurfSense Extension from https://chromewebstore.google.com/detail/surfsense/jihmihbdpfjhppdlifphccgefjhifblf
|
||||
|
||||
Now you are ready to use SurfSense. Start by first logging into the Extension.
|
||||
|
||||
When you start the extension you should see a Login page like this
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
After logging in you will need to fill your Neo4j Credentials & OpenAPI Key.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
After Saving you should be able to use extension now.
|
||||
|
||||
|
||||
|
||||
|Options|Explanations|
|
||||
|--|--|
|
||||
| Clear Inactive History Sessions | It clears the saved content for Inactive Tab Sessions. |
|
||||
| Save Current Webpage Snapshot | Stores the current webpage session info into SurfSense history store|
|
||||
| Save to SurfSense | Processes the SurfSense History Store & Initiates a Save Job |
|
||||
|
||||
4. Now just start browsing the Internet. Whatever you want to save any content take its Snapshot and save it to SurfSense. After Save Job is completed you are ready to ask anything about it to your Knowledge Graph Brain 🧠.
|
||||
5. Critical Notifications are automatically generated. Check them out at https://www.surfsense.net/notifications .
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
6. Now go to SurfSense Chat Options at https://www.surfsense.net/chat & fill the Neo4j Credentials & OpenAPI Key if asked.
|
||||
|
||||
|
||||
|
||||
|
||||
|OPTIONS|DESCRIPTION|
|
||||
|--|--|
|
||||
| Precision Chat | Used for detailed search and chatting with your saved web sessions and their content. |
|
||||
| General Chat | Used for general questions about your content. Doesn't work well with Dates & Time.|
|
||||
|
||||
### Chat Screenshots
|
||||
---
|
||||
#### PRECISION
|
||||
|
||||
##### Search
|
||||
|
||||
|
||||
|
||||
|
||||
##### Results
|
||||
|
||||
|
||||
|
||||
|
||||
##### Multi Webpage Chat
|
||||
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
#### GENERAL
|
||||
As an example lets visit : https://myanimelist.net/anime/season (Summer 2024 Anime atm) and save it to SurfSense.
|
||||
|
||||
Now lets ask SurfSense "Give list of summer 2024 animes with images."
|
||||
|
||||
Sample Response:
|
||||
|
||||
|
||||
|
||||
Now Let's ask it more information about our related session.
|
||||
|
||||
|
||||
|
||||
Sample More Description Response:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Local Setup Guide
|
||||
|
||||
#### Backend
|
||||
### Backend
|
||||
|
||||
For authentication purposes, you’ll also need a PostgreSQL instance running on your machine.
|
||||
|
||||
Now lets setup the SurfSense BackEnd
|
||||
1. Clone this repo.
|
||||
2. Go to ./backend subdirectory.
|
||||
3. Setup Python Virtual Enviroment
|
||||
3. Setup Python Virtual Environment
|
||||
4. Run `pip install -r requirements.txt` to install all required dependencies.
|
||||
5. Update the required Environment variables in envs.py
|
||||
5. Update/Make the required Environment variables in .env
|
||||
|
||||
|ENV VARIABLE|Description |
|
||||
|--|--|
|
||||
| IS_LOCAL_SETUP | 'true' for Ollama mistral-nemo or 'false' from OpenAI gpt-4o-mini|
|
||||
| POSTGRES_DATABASE_URL | postgresql+psycopg2://user:pass@host:5432/database|
|
||||
| API_SECRET_KEY | Can be any Random String value. Make Sure to remember it for as you need to send it in request to Backend for security purposes.|
|
||||
| API_SECRET_KEY | Can be any Random String value. Make Sure to remember it for as you need to send it in user registration request to Backend for security purposes.|
|
||||
|
||||
|
||||
6. Backend is a FastAPI Backend so now just run the server on unicorn using command `uvicorn server:app --host 0.0.0.0 --port 8000`
|
||||
|
|
@ -148,25 +62,13 @@ Now lets setup the SurfSense BackEnd
|
|||
|
||||
---
|
||||
|
||||
#### Extension
|
||||
|
||||
**UPDATE:** Extension code is now migrated to Plasmo. Follow this guide to build for your target browser now : https://docs.plasmo.com/framework/workflows/build
|
||||
|
||||
env eg in `.env.local`
|
||||
|
||||
Now resister a quick user through Swagger API > Try it Out: http://127.0.0.1:8000/docs#/default/register_user_register_post
|
||||
|
||||
Make Sure in request body `"apisecretkey"` value is same value as `API_SECRET_KEY` we been assigning.
|
||||
|
||||
---
|
||||
|
||||
#### FrontEnd
|
||||
### FrontEnd
|
||||
|
||||
For local frontend setup just fill out the `.env` file of frontend.
|
||||
|
||||
|ENV VARIABLE|DESCRIPTION|
|
||||
|--|--|
|
||||
| NEXT_PUBLIC_API_SECRET_KEY | Same String value your set for Backend & Extension |
|
||||
| NEXT_PUBLIC_API_SECRET_KEY | Same String value your set for Backend |
|
||||
| NEXT_PUBLIC_BACKEND_URL | Give hosted backend url here. Eg. `http://127.0.0.1:8000`|
|
||||
| NEXT_PUBLIC_RECAPTCHA_SITE_KEY | Google Recaptcha v2 Client Key |
|
||||
| RECAPTCHA_SECRET_KEY | Google Recaptcha v2 Server Key|
|
||||
|
|
@ -175,12 +77,80 @@ and run it using `pnpm run dev`
|
|||
|
||||
You should see your Next.js frontend running at `localhost:3000`
|
||||
|
||||
**Make sure to register a account from frontend so you can login to extension.**
|
||||
|
||||
---
|
||||
|
||||
### Extension
|
||||
|
||||
Extension is in plasmo framework which is a cross browser extension framework.
|
||||
|
||||
For building extension just fill out the `.env` file of frontend.
|
||||
|
||||
|ENV VARIABLE|DESCRIPTION|
|
||||
|--|--|
|
||||
| PLASMO_PUBLIC_BACKEND_URL| SurfSense Backend URL eg. "http://127.0.0.1:8000" |
|
||||
|
||||
Build the extension for your favorite browser using this guide: https://docs.plasmo.com/framework/workflows/build#with-a-specific-target
|
||||
|
||||
When you load and start the extension you should see a Login page like this
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
After logging in you will need to fill your OpenAPI Key. Fill random value if you are using Ollama.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
After Saving you should be able to use extension now.
|
||||
|
||||
|
||||
|
||||
|
||||
|Options|Explanations|
|
||||
|--|--|
|
||||
| Search Space | Think of it like a category tag for the webpages you want to save. |
|
||||
| Clear Inactive History Sessions | It clears the saved content for Inactive Tab Sessions. |
|
||||
| Save Current Webpage Snapshot | Stores the current webpage session info into SurfSense history store|
|
||||
| Save to SurfSense | Processes the SurfSense History Store & Initiates a Save Job |
|
||||
|
||||
4. Now just start browsing the Internet. Whatever you want to save any content take its Snapshot and save it to SurfSense. After Save Job is completed you are ready to ask anything about it to SurfSense 🧠.
|
||||
|
||||
6. Now go to SurfSense Dashboard After Logging in.
|
||||
|
||||
|DASHBOARD OPTIONS|DESCRIPTION|
|
||||
|--|--|
|
||||
| Playground | See saved documents and can have chat with multiple docs. |
|
||||
| Search Space Chat | Used for questions about your content in particular search space.|
|
||||
| Saved Chats | All your saved chats.|
|
||||
| Settings | If you want to update your Open API key.|
|
||||
|
||||
## Screenshots
|
||||
|
||||
#### Playground
|
||||
|
||||
|
||||
|
||||
#### Search Spaces Chat (Ollama LLM)
|
||||
|
||||
|
||||
|
||||
|
||||
#### Multiple Document Chat (Ollama LLM)
|
||||
|
||||
|
||||
|
||||
#### Saved Chats
|
||||
|
||||
|
||||
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Extenstion** : Chrome Manifest v3
|
||||
- **Extenstion** : Manifest v3 on Plasmo
|
||||
- **BackEnd** : FastAPI with LangChain
|
||||
- **FrontEnd**: Next.js with Aceternity.
|
||||
|
||||
|
|
@ -188,14 +158,15 @@ You should see your Next.js frontend running at `localhost:3000`
|
|||
In Progress...........
|
||||
|
||||
## Future Work
|
||||
- Based on feedback, I will work on making it compatible with local models.
|
||||
- Implement Canvas.
|
||||
- Add support for file uploads QA.
|
||||
- Based on feedback, I will work on making it compatible with local models. **[Done]**
|
||||
- Cross Browser Extension **[Done]**
|
||||
- Generalize the way SurfSense uses Graphs. Will soon make an integration with FalkorDB soon.
|
||||
- Critical Notifications **[Done]**
|
||||
- Critical Notifications **[Done | PAUSED]**
|
||||
- Saving Chats **[Done]**
|
||||
- Basic keyword search page for saved sessions **[Done]**
|
||||
- Multi & Single Document Chat **[Done]**
|
||||
- Implement some tricks from GraphRAG papers to optimize current GraphRAG logic.
|
||||
|
||||
|
||||
## Contribute
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue