Merge pull request #96 from cubxxw/feat/optimize-docker-compose

refactor: remove frontend and backend services

DEPLOYMENT_GUIDE.md

@@ -0,0 +1,124 @@
+# SurfSense Deployment Guide
+
+This guide explains the different deployment options available for SurfSense using Docker Compose.
+
+## Deployment Options
+
+SurfSense uses a flexible Docker Compose configuration that allows you to easily switch between deployment modes without manually editing files. Our approach uses Docker's built-in override functionality with two configuration files:
+
+1. **docker-compose.yml**: Contains essential core services (database and pgAdmin)
+2. **docker-compose.override.yml**: Contains application services (frontend and backend)
+
+This structure provides several advantages:
+- No need to comment/uncomment services manually
+- Clear separation between core infrastructure and application services
+- Easy switching between development and production environments
+
+## Deployment Modes
+
+### Full Stack Mode (Development)
+
+This mode runs everything: frontend, backend, database, and pgAdmin. It's ideal for development environments where you need the complete application stack.
+
+```bash
+# Both files are automatically used (docker-compose.yml + docker-compose.override.yml)
+docker compose up -d
+```
+
+### Core Services Mode (Production)
+
+This mode runs only the database and pgAdmin services. It's suitable for production environments where you might want to deploy the frontend and backend separately or need to run database migrations.
+
+```bash
+# Explicitly use only the main file
+docker compose -f docker-compose.yml up -d
+```
+
+## Custom Deployment Options
+
+### Running Specific Services
+
+You can specify which services to start by naming them:
+
+```bash
+# Start only database
+docker compose up -d db
+
+# Start database and pgAdmin
+docker compose up -d db pgadmin
+
+# Start only backend (requires db to be running)
+docker compose up -d backend
+```
+
+### Using Custom Override Files
+
+You can create and use custom override files for different environments:
+
+```bash
+# Create a staging configuration
+docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d
+```
+
+## Environment Variables
+
+The deployment can be customized using environment variables:
+
+```bash
+# Change default ports
+FRONTEND_PORT=4000 BACKEND_PORT=9000 docker compose up -d
+
+# Or use a .env file
+# Create or modify .env file with your desired values
+docker compose up -d
+```
+
+## Common Deployment Workflows
+
+### Initial Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/MODSetter/SurfSense.git
+cd SurfSense
+
+# Copy example env files
+cp .env.example .env
+cp surfsense_backend/.env.example surfsense_backend/.env
+cp surfsense_web/.env.example surfsense_web/.env
+
+# Edit the .env files with your configuration
+
+# Start full stack for development
+docker compose up -d
+```
+
+### Database-Only Mode (for migrations or maintenance)
+
+```bash
+# Start just the database
+docker compose -f docker-compose.yml up -d db
+
+# Run migrations or maintenance tasks
+docker compose exec db psql -U postgres -d surfsense
+```
+
+### Scaling in Production
+
+For production deployments, you might want to:
+
+1. Run core services with Docker Compose
+2. Deploy frontend/backend with specialized services like Vercel, Netlify, or dedicated application servers
+
+This separation allows for better scaling and resource utilization in production environments.
+
+## Troubleshooting
+
+If you encounter issues with the deployment:
+
+- Check container logs: `docker compose logs -f [service_name]`
+- Ensure all required environment variables are set
+- Verify network connectivity between containers
+- Check that required ports are available and not blocked by firewalls
+
+For more detailed setup instructions, refer to [DOCKER_SETUP.md](DOCKER_SETUP.md). 

DOCKER_SETUP.md

@@ -36,6 +36,20 @@ PGADMIN_DEFAULT_EMAIL=admin@surfsense.com
 PGADMIN_DEFAULT_PASSWORD=surfsense
 ```
 
+## Deployment Options
+
+SurfSense uses a flexible Docker Compose setup that allows you to choose between different deployment modes:
+
+### Option 1: Full-Stack Deployment (Development Mode)
+Includes frontend, backend, database, and pgAdmin. This is the default when running `docker compose up`.
+
+### Option 2: Core Services Only (Production Mode)
+Includes only database and pgAdmin, suitable for production environments where you might deploy frontend/backend separately.
+
+Our setup uses two files:
+- `docker-compose.yml`: Contains core services (database and pgAdmin)
+- `docker-compose.override.yml`: Contains application services (frontend and backend)
+
 ## Setup
 
 1. Make sure you have all the necessary environment variables set up:
@@ -43,53 +57,85 @@ PGADMIN_DEFAULT_PASSWORD=surfsense
    - Copy `surfsense_web/.env.example` to `surfsense_web/.env` and fill in the required values
    - Optionally: Copy `.env.example` to `.env` in the project root to customize Docker settings
 
-2. Build and start the containers:
+2. Deploy based on your needs:
+
+   **Full Stack (Development Mode)**:
    ```bash
-   docker-compose up --build
+   # Both files are automatically used
+   docker compose up --build
+   ```
+
+   **Core Services Only (Production Mode)**:
+   ```bash
+   # Explicitly use only the main file
+   docker compose -f docker-compose.yml up --build
    ```
 
 3. To run in detached mode (in the background):
    ```bash
-   docker-compose up -d
+   # Full stack
+   docker compose up -d
+   
+   # Core services only
+   docker compose -f docker-compose.yml up -d
    ```
 
 4. Access the applications:
-   - Frontend: http://localhost:3000
+   - Frontend: http://localhost:3000 (when using full stack)
-   - Backend API: http://localhost:8000
+   - Backend API: http://localhost:8000 (when using full stack)
-   - API Documentation: http://localhost:8000/docs
+   - API Documentation: http://localhost:8000/docs (when using full stack)
    - pgAdmin: http://localhost:5050
 
+## Customizing the Deployment
+
+If you need to make temporary changes to either full stack or core services deployment, you can:
+
+1. **Temporarily disable override file**:
+   ```bash
+   docker compose -f docker-compose.yml up -d
+   ```
+
+2. **Use a custom override file**:
+   ```bash
+   docker compose -f docker-compose.yml -f custom-override.yml up -d
+   ```
+
+3. **Temporarily modify which services start**:
+   ```bash
+   docker compose up -d db pgadmin
+   ```
+
 ## Useful Commands
 
 - Stop the containers:
   ```bash
-  docker-compose down
+  docker compose down
   ```
 
 - View logs:
   ```bash
   # All services
-  docker-compose logs -f
+  docker compose logs -f
   
   # Specific service
-  docker-compose logs -f backend
+  docker compose logs -f backend
-  docker-compose logs -f frontend
+  docker compose logs -f frontend
-  docker-compose logs -f db
+  docker compose logs -f db
-  docker-compose logs -f pgadmin
+  docker compose logs -f pgadmin
   ```
 
 - Restart a specific service:
   ```bash
-  docker-compose restart backend
+  docker compose restart backend
   ```
 
 - Execute commands in a running container:
   ```bash
   # Backend
-  docker-compose exec backend python -m pytest
+  docker compose exec backend python -m pytest
   
   # Frontend
-  docker-compose exec frontend pnpm lint
+  docker compose exec frontend pnpm lint
   ```
 
 ## Database
@@ -129,5 +175,18 @@ pgAdmin is a web-based administration tool for PostgreSQL. It is included in the
 - If you encounter permission errors, you may need to run the docker commands with `sudo`.
 - If ports are already in use, modify the port mappings in the `.env` file or directly in the `docker-compose.yml` file.
 - For backend dependency issues, you may need to modify the `Dockerfile` in the backend directory.
-- For frontend dependency issues, you may need to modify the `Dockerfile` in the frontend directory.
+- If you encounter frontend dependency errors, adjust the frontend's `Dockerfile` accordingly.
 - If pgAdmin doesn't connect to the database, ensure you're using `db` as the hostname, not `localhost`, as that's the Docker network name. 
+- If you need only specific services, you can explicitly name them: `docker compose up db pgadmin`
+
+## Understanding Docker Compose File Structure
+
+The project uses Docker's default override mechanism:
+
+1. **docker-compose.yml**: Contains essential services (database and pgAdmin)
+2. **docker-compose.override.yml**: Contains development services (frontend and backend)
+
+When you run `docker compose up` without additional flags, Docker automatically merges both files.
+When you run `docker compose -f docker-compose.yml up`, only the specified file is used.
+
+This approach lets you maintain a cleaner codebase without manually commenting/uncommenting services in your configuration files. 

README.md

@@ -119,7 +119,10 @@ SurfSense provides two installation methods:
 1. **[Docker Installation](https://www.surfsense.net/docs/docker-installation)** - The easiest way to get SurfSense up and running with all dependencies containerized.
    - Includes pgAdmin for database management through a web UI
    - Supports environment variable customization via `.env` file
+   - Flexible deployment options (full stack or core services only)
+   - No need to manually edit configuration files between environments
    - See [Docker Setup Guide](DOCKER_SETUP.md) for detailed instructions
+   - For deployment scenarios and options, see [Deployment Guide](DEPLOYMENT_GUIDE.md)
 
 2. **[Manual Installation (Recommended)](https://www.surfsense.net/docs/manual-installation)** - For users who prefer more control over their setup or need to customize their deployment.
 

docker-compose.override.yml

@@ -0,0 +1,34 @@
+version: '3.8'
+
+services:
+  frontend:
+    build:
+      context: ./surfsense_web
+      dockerfile: Dockerfile
+    ports:
+      - "${FRONTEND_PORT:-3000}:3000"
+    volumes:
+      - ./surfsense_web:/app
+      - /app/node_modules
+    depends_on:
+      - backend
+    environment:
+      - NEXT_PUBLIC_API_URL=${NEXT_PUBLIC_API_URL:-http://backend:8000}
+
+  backend:
+    build:
+      context: ./surfsense_backend
+      dockerfile: Dockerfile
+    ports:
+      - "${BACKEND_PORT:-8000}:8000"
+    volumes:
+      - ./surfsense_backend:/app
+    depends_on:
+      - db
+    env_file:
+      - ./surfsense_backend/.env
+    environment:
+      - DATABASE_URL=postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-surfsense}
+      - PYTHONPATH=/app
+      - UVICORN_LOOP=asyncio
+      - UNSTRUCTURED_HAS_PATCHED_LOOP=1

docker-compose.yml

@@ -1,38 +1,6 @@
 version: '3.8'
 
 services:
-  frontend:
-    build:
-      context: ./surfsense_web
-      dockerfile: Dockerfile
-    ports:
-      - "${FRONTEND_PORT:-3000}:3000"
-    volumes:
-      - ./surfsense_web:/app
-      - /app/node_modules
-    depends_on:
-      - backend
-    environment:
-      - NEXT_PUBLIC_API_URL=${NEXT_PUBLIC_API_URL:-http://backend:8000}
-
-  backend:
-    build:
-      context: ./surfsense_backend
-      dockerfile: Dockerfile
-    ports:
-      - "${BACKEND_PORT:-8000}:8000"
-    volumes:
-      - ./surfsense_backend:/app
-    depends_on:
-      - db
-    env_file:
-      - ./surfsense_backend/.env
-    environment:
-      - DATABASE_URL=postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-surfsense}
-      - PYTHONPATH=/app
-      - UVICORN_LOOP=asyncio
-      - UNSTRUCTURED_HAS_PATCHED_LOOP=1
-
   db:
     image: ankane/pgvector:latest
     ports:

surfsense_web/content/docs/docker-installation.mdx

@@ -133,7 +133,7 @@ For other LLM providers, refer to the [LiteLLM documentation](https://docs.litel
    **Linux/macOS/Windows:**
 
    ```bash
-   docker-compose up --build
+   docker compose up --build
    ```
 
    To run in detached mode (in the background):
@@ -141,10 +141,10 @@ For other LLM providers, refer to the [LiteLLM documentation](https://docs.litel
    **Linux/macOS/Windows:**
 
    ```bash
-   docker-compose up -d
+   docker compose up -d
    ```
 
-   **Note for Windows users:** If you're using older Docker Desktop versions, you might need to use `docker compose` (with a space) instead of `docker-compose`.
+   **Note for Windows users:** If you're using older Docker Desktop versions, you might need to use `docker compose` (with a space) instead of `docker compose`.
 
 3. **Access the Applications**
 
@@ -180,7 +180,7 @@ pgAdmin is included in the Docker setup to help manage your PostgreSQL database.
   **Linux/macOS/Windows:**
 
   ```bash
-  docker-compose down
+  docker compose down
   ```
 
 - **View logs:**
@@ -189,12 +189,12 @@ pgAdmin is included in the Docker setup to help manage your PostgreSQL database.
 
   ```bash
   # All services
-  docker-compose logs -f
+  docker compose logs -f
 
   # Specific service
-  docker-compose logs -f backend
+  docker compose logs -f backend
-  docker-compose logs -f frontend
+  docker compose logs -f frontend
-  docker-compose logs -f db
+  docker compose logs -f db
   ```
 
 - **Restart a specific service:**
@@ -202,7 +202,7 @@ pgAdmin is included in the Docker setup to help manage your PostgreSQL database.
   **Linux/macOS/Windows:**
 
   ```bash
-  docker-compose restart backend
+  docker compose restart backend
   ```
 
 - **Execute commands in a running container:**
@@ -211,10 +211,10 @@ pgAdmin is included in the Docker setup to help manage your PostgreSQL database.
 
   ```bash
   # Backend
-  docker-compose exec backend python -m pytest
+  docker compose exec backend python -m pytest
 
   # Frontend
-  docker-compose exec frontend pnpm lint
+  docker compose exec frontend pnpm lint
   ```
 
 ## Troubleshooting