vrr/openwebui-oikb
2
0
Fork 0
mirror of https://github.com/open-webui/oikb.git synced 2026-07-09 16:00:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

doc: readme

Browse source
This commit is contained in:
Timothy Jaeryang Baek 2026-05-20 13:50:01 +04:00
parent 24ee149a4a
commit 08a1bd1859
1 changed files with 52 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

211
README.md
View file

@ -1,179 +1,91 @@
# 📚 oikb
A CLI tool that syncs content to [Open WebUI](https://github.com/open-webui/open-webui) Knowledge Bases. 30+ connectors for local files, Git repos, cloud storage, wikis, ticketing, messaging, CRM, and more.
Keep your [Open WebUI](https://github.com/open-webui/open-webui) Knowledge Bases in sync. Point it at a local directory, a GitHub repo, a Confluence space, an S3 bucket, or any of 30+ supported sources. Only new and modified files are uploaded via incremental SHA-256 diffing.
> [!IMPORTANT]
> Requires **Open WebUI 0.9.6+** (uses the Knowledge Base sync API).
> Requires **Open WebUI 0.9.6+**
## Why oikb?
Manually uploading files to Knowledge Bases is tedious. oikb automates it with incremental SHA-256 diffing, so only new and modified files are uploaded. Directory structure is mirrored automatically. Run it once, run it in CI, or leave it watching for changes.
## Connectors
oikb connects to all your tools. One CLI, incremental sync, automatic directory mirroring.
### Code Repositories
| Source | Syntax | Auth |
|---|---|---|
| **GitHub** | `github:owner/repo` | `GITHUB_TOKEN` |
| **GitLab** | `gitlab:owner/repo` | `GITLAB_TOKEN` |
| **Bitbucket** | `bitbucket:owner/repo` | `BITBUCKET_USER` + `BITBUCKET_APP_PASSWORD` |
### Cloud Storage
| Source | Syntax | Auth |
|---|---|---|
| **Local directory** | `./docs` | - |
| **Amazon S3** | `s3://bucket/prefix` | AWS credentials |
| **Google Cloud Storage** | `gs://bucket/prefix` | `GOOGLE_APPLICATION_CREDENTIALS` |
| **Azure Blob Storage** | `az://container/prefix` | `AZURE_STORAGE_CONNECTION_STRING` |
| **Dropbox** | `dropbox:/path` | `DROPBOX_TOKEN` |
| **Cloudflare R2** | `r2://bucket/prefix` | `R2_ACCOUNT_ID` + R2 keys |
### Knowledge Bases & Wikis
| Source | Syntax | Auth |
|---|---|---|
| **Confluence** | `confluence:SPACEKEY` | `CONFLUENCE_URL` + `CONFLUENCE_TOKEN` |
| **Notion** | `notion:<database-id>` | `NOTION_TOKEN` |
| **BookStack** | `bookstack:` | `BOOKSTACK_URL` + `BOOKSTACK_TOKEN_*` |
| **Discourse** | `discourse:category` | `DISCOURSE_URL` + `DISCOURSE_API_KEY` |
| **GitBook** | `gitbook:<space-id>` | `GITBOOK_TOKEN` |
| **Guru** | `guru:` | `GURU_USER` + `GURU_TOKEN` |
### Ticketing & Task Management
| Source | Syntax | Auth |
|---|---|---|
| **Jira** | `jira:PROJECT` | `JIRA_URL` + `JIRA_TOKEN` |
| **Linear** | `linear:<team-id>` | `LINEAR_TOKEN` |
| **Asana** | `asana:<project-id>` | `ASANA_TOKEN` |
| **ClickUp** | `clickup:<space-id>` | `CLICKUP_TOKEN` |
| **Zendesk** | `zendesk:<subdomain>` | `ZENDESK_SUBDOMAIN` + `ZENDESK_TOKEN` |
| **Freshdesk** | `freshdesk:<domain>` | `FRESHDESK_DOMAIN` + `FRESHDESK_TOKEN` |
| **Airtable** | `airtable:<base-id>` | `AIRTABLE_TOKEN` |
### Messaging
| Source | Syntax | Auth |
|---|---|---|
| **Slack** | `slack:<channel-id>` | `SLACK_TOKEN` |
| **Discord** | `discord:<channel-id>` | `DISCORD_TOKEN` |
| **Microsoft Teams** | `teams:<team-id>/<channel-id>` | `TEAMS_*` credentials |
| **Gmail** | `gmail:user@domain.com` | `GOOGLE_APPLICATION_CREDENTIALS` |
### Sales & CRM
| Source | Syntax | Auth |
|---|---|---|
| **Salesforce** | `salesforce:` | `SALESFORCE_URL` + `SALESFORCE_TOKEN` |
| **HubSpot** | `hubspot:` | `HUBSPOT_TOKEN` |
### Other
| Source | Syntax | Auth |
|---|---|---|
| **Website / Sitemap** | `web:https://docs.example.com` | - |
| **Google Drive** | `gdrive:<folder-id>` | `GOOGLE_APPLICATION_CREDENTIALS` |
| **SharePoint** | `sharepoint:site/library` | `SHAREPOINT_*` credentials |
Most connectors use `httpx` (included). Some require an extra:
## Quick Start
```bash
pip install oikb[gdrive] # Google Drive
pip install oikb[gmail] # Gmail
pip install oikb[s3] # Amazon S3, Cloudflare R2
pip install oikb[gcs] # Google Cloud Storage
pip install oikb[azure] # Azure Blob Storage
pip install oikb[dropbox] # Dropbox
pip install oikb[web] # Web crawler
pip install oikb[all] # Everything
pip install oikb
export OPEN_WEBUI_URL=http://localhost:3000
export OPEN_WEBUI_API_KEY=sk-your-api-key
# Sync a directory to a Knowledge Base
oikb sync ./docs --kb-id your-kb-id
# Or watch for changes and auto-sync continuously
oikb watch ./docs --kb-id your-kb-id
```
## Getting Started
### Docker (recommended)
Or with Docker:
```bash
docker run --rm \
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
-e OPEN_WEBUI_API_KEY=sk-your-key \
-v ./docs:/data \
ghcr.io/open-webui/oikb sync /data --kb-id your-kb-id
ghcr.io/open-webui/oikb watch /data --kb-id your-kb-id
```
### pip
```bash
pip install oikb
```
```bash
export OPEN_WEBUI_URL=http://localhost:3000
export OPEN_WEBUI_API_KEY=sk-your-api-key
oikb sync ./docs --kb-id your-kb-id
```
> [!TIP]
> No config file needed. Env vars are enough. You can also use `oikb config set url ...` and `oikb config set token ...` to save them to `~/.config/oikb/config.yaml`.
## Commands
| Command | Description |
|---|---|
| `oikb sync <source>` | Incremental sync to a Knowledge Base |
| `oikb watch <source>` | Watch for changes and auto-sync |
| `oikb diff <source>` | Preview what a sync would do |
| `oikb ls` | List files in a Knowledge Base |
| `oikb status` | Show KB info and file count |
| `oikb reset` | Delete all files in a Knowledge Base |
| `oikb config` | Manage saved URL and API key |
## 30+ Connectors
Beyond local directories, oikb can sync from remote sources using the same `oikb sync <source>` interface.
| Category | Sources |
|---|---|
| **Code Repos** | GitHub, GitLab, Bitbucket |
| **Cloud Storage** | S3, GCS, Azure Blob, Dropbox, R2, Google Drive, SharePoint |
| **Wikis & KBs** | Confluence, Notion, BookStack, Discourse, GitBook, Guru |
| **Ticketing** | Jira, Linear, Zendesk, Freshdesk, Asana, ClickUp, Airtable |
| **Messaging** | Slack, Discord, Microsoft Teams, Gmail |
| **Sales & CRM** | Salesforce, HubSpot |
| **Web** | Website / Sitemap crawler |
```bash
# Sync from any connector
oikb sync ./docs --kb-id your-kb-id
oikb sync github:owner/repo --kb-id your-kb-id
oikb sync confluence:ENG --kb-id your-kb-id
oikb sync notion:abc123 --kb-id your-kb-id
oikb sync slack:C0123ABC --kb-id your-kb-id
oikb sync jira:PROJ --kb-id your-kb-id
oikb sync s3://bucket/prefix --kb-id your-kb-id
oikb sync web:https://docs.example.com --kb-id your-kb-id
# Preview changes without uploading
oikb diff ./docs --kb-id your-kb-id
# Watch for changes and auto-sync
oikb watch ./docs --kb-id your-kb-id
# List files / show info / reset
oikb ls --kb-id your-kb-id
oikb status --kb-id your-kb-id
oikb reset --kb-id your-kb-id
oikb sync slack:C0123ABC --kb-id your-kb-id
```
Some connectors need an optional extra: `pip install oikb[gdrive]`, `pip install oikb[s3]`, or `pip install oikb[all]` for everything.
## Configuration
Settings are resolved in this order (highest priority wins):
Resolved in order (highest priority wins):
1. **CLI flags** (`--url`, `--token`)
2. **Environment variables** (`OPEN_WEBUI_URL`, `OPEN_WEBUI_API_KEY`)
3. **Config file** (`~/.config/oikb/config.yaml`)
## Declarative Config (`.oikb.yaml`)
### Multi-source (`.oikb.yaml`)
For recurring multi-source syncs:
Define multiple sources in a single config file:
```yaml
sync:
- source: ./docs
kb: project-docs
- source: github:owner/wiki
kb: team-wiki
branch: main
- source: confluence:ENG
kb: eng-handbook
- source: slack:C0123ABC
kb: team-updates
- source: jira:PROJ
kb: project-issues
```
```bash
@ -181,27 +93,9 @@ oikb sync # Sync all entries
oikb sync --name docs # Sync a specific entry
```
## Docker
## Docker Compose
Published at `ghcr.io/open-webui/oikb`.
```bash
# One-shot sync
docker run --rm \
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
-e OPEN_WEBUI_API_KEY=sk-your-key \
-v ./docs:/data \
ghcr.io/open-webui/oikb sync /data --kb-id your-kb-id
# Watch mode
docker run --rm -d \
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
-e OPEN_WEBUI_API_KEY=sk-your-key \
-v ./docs:/data \
ghcr.io/open-webui/oikb watch /data --kb-id your-kb-id
```
### Docker Compose
Run as a sidecar alongside Open WebUI:
```yaml
services:
@ -223,7 +117,7 @@ services:
restart: unless-stopped
```
### GitHub Actions
## GitHub Actions
```yaml
- name: Sync docs to Open WebUI
@ -237,11 +131,10 @@ services:
## How It Works
1. Scan source (local dir, GitHub, Confluence, S3, etc.), compute checksums
2. Send manifest to Open WebUI's `/sync/diff` endpoint
3. Server diffs against stored file hashes
4. Delete stale files, create missing directories
5. Upload only new and modified files
1. Scan source, compute checksums
2. Send manifest to Open WebUI `/sync/diff`
3. Delete stale files, create missing directories
4. Upload only new and modified files
## License