diff --git a/AGENTS.md b/AGENTS.md index 532355c..76f66ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,55 @@ # Agents -Read [`AGENT_GUIDE.md`](./AGENT_GUIDE.md) first. It is the source of truth for repo conventions, architecture, editor model, mobile shell, settings, debugging workflow, and coding rules. +Joplock is a standalone web client for a stock, unmodified Joplin Server: Node (no framework), server-rendered HTML with htmx fragment swaps, and one large client file (`public/app.js`). It reads the shared Postgres directly and writes through Joplin Server's own API, so the same account keeps working from Joplin desktop, mobile and CLI. + +This file is loaded on every task, so it is short on purpose. It gives the rules and says where the **reasons** live. It does not say where code lives: use the code index below (it cannot go stale). The topic files in `docs/agents/` hold the hard-won "why" and "do not regress" rules. **Read the one that matches what you are about to change, before you change it.** + +## Before you touch ... read + +| You are changing | Read first | +|---|---| +| the editor: CodeMirror/TinyMCE, mode switching, autosave, markdown round-trip, a phantom "Edited" | `docs/agents/editor.md` | +| any save path, vaults/encryption, note history/restore, anything reading the editor DOM across a note switch | `docs/agents/vaults-and-save-safety.md` | +| how notes/notebooks/resources are serialized or written (anything another Joplin client can see) | `docs/agents/joplin-fidelity.md` | +| the notebook tree, drag and drop, sort, pins, sharing, `user_items` | `docs/agents/notebooks-and-sharing.md` | +| upload, paste or drop of images/attachments | `docs/agents/editor-uploads.md` | +| `public/cm-extras.js` or the CodeMirror bundle | `docs/agents/editor-markdown-extras.md` | +| mobile layout, PWA, service worker, viewport/keyboard, static serving, anything iOS | `docs/agents/pwa-mobile.md` | +| text expander, AI autocomplete, `/ask` | `docs/agents/ai-expander-ask.md` | +| settings, themes, admin, backup/recovery, MFA, debug logging | `docs/agents/settings-and-admin.md` | +| server shape, htmx/fragment conventions, routes, data model, or you are unsure where something belongs | `docs/agents/architecture.md` | +| tests, Playwright, the dev stack, "my change does not show up" | `docs/agents/testing-and-dev.md` | + +## Rules that always apply + +1. Do not modify Joplin Server source for Joplock features unless explicitly approved. +2. The server is authoritative and the browser is ephemeral: no browser-local authoritative storage, and logout must clear client-visible state (shared machines). +3. Stay sync-compatible with normal Joplin clients. A Joplock write changes **only what the user edited** (`docs/agents/joplin-fidelity.md`). +4. Keep the sidecar API app-oriented; do not expose the raw sync/storage model to the frontend. +5. Keep changes minimal. Fix the cause, not the symptom, and add a test that fails without the fix. When you fix mobile, check desktop (and the reverse): they share the editor fragment. + +## Workflow + +- `npm test` before every commit (CI runs it on Node 20; this machine may run a newer Node). +- **Pushing to `master` publishes the Docker image**: GitHub Actions runs the tests, then builds and pushes `ghcr.io/abort-retry-ignore/joplock:latest`. Do not commit to `master` or push unless the user asks. +- Source edits need `./scripts/rebuild-dev.sh` (the image copies `app/`, `public/` and `server.js` at build time; a container restart is not enough). Browser tests (`npm run test:ui`) need that dev stack and `JOPLOCK_ADMIN_*` in the environment; see `docs/agents/testing-and-dev.md`. +- `plans/` is gitignored local history, **not documentation**; `plans/README.md` lists what is stale. Trust the code and these docs over any plan. +- Asset versions and the service worker cache name are computed (`app/assetVersion.js`): never hand-edit or "bump" them. +- After large changes run `graft build` so the code index matches. + +## Traps that have each broken real data or a real phone + +One line each; the topic file has the detail. + +- **Inline `onclick="foo()"` only sees `window.foo`.** A function works in a unit test and is "not defined" in the browser unless `public/app.js` assigns it to `window` (`tests/inlineHandlersExposed.test.js`). +- **Never write one note's text into another.** Debounced saves, async fetches and TinyMCE are shared across note switches; every save path has an identity guard (`vaults-and-save-safety.md`). +- **The markdown <-> TinyMCE round trip must be idempotent**, or notes show a phantom "Edited" on open and get re-saved (`editor.md`). +- **Do not decode `items.content` for every item in SQL**: it also holds binary resources and one attachment aborts the statement (`notebooks-and-sharing.md`). +- **iOS home-screen apps**: never set `apple-mobile-web-app-status-bar-style: black-translucent`; never guess a keyboard from `innerHeight - visualViewport.height`; decide mobile vs desktop from the physical screen, never the window height (`pwa-mobile.md`). Chromium cannot reproduce any of this: measure on a device with `/diag`. +- **DOM ids and classes are a contract with inline JS and tests.** Rename them deliberately. +- **Floating menus need an opaque background and edge clamping**; pure `:hover` styles go inside `@media (hover: hover)` (`architecture.md`, `pwa-mobile.md`). + +## Code index ## Graft — repo context graph diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md deleted file mode 100644 index 597197b..0000000 --- a/AGENT_GUIDE.md +++ /dev/null @@ -1,1070 +0,0 @@ -# Joplock Agent Guide - - - -## Purpose - -This repo owns Joplock, standalone thin-client sidecar web UI for stock Joplin Server. - -Use this guide when working in this repository. - -## Product Direction - -- Joplin Server stays unmodified -- Joplock stays separate project and separate repo -- Reuses existing Joplin Server auth/session/user model through sidecar logic -- Keeps compatibility with desktop/mobile/CLI clients on same server and same DB -- Browser stays thin and untrusted -- Shared-browser safety matters: logout should clear client-visible state/cache as much as platform allows -- Installable PWA shell, no offline notes/editing -- Uses same Postgres database as Joplin Server, no separate app DB - -## Architecture Overview - -### Stack - -- **Server**: Node.js HTTP server, no framework -- **Client**: SSR HTML + htmx fragment swaps + shared browser logic in `public/app.js` -- **Editor**: Dual-mode. Markdown mode = CodeMirror 6 (mounted into `#cm-host`); rendered mode = TinyMCE 8. The `#note-body` textarea is the hidden form/sync target for both. -- **Code blocks**: Full-screen code modal with a CM6 editor and language picker. Highlighting differs by mode: **markdown mode (CM6)** uses CM6's own `syntaxHighlighting(joplockHighlight)` with the language parsers bundled from `cm-build/`; **rendered mode (TinyMCE)** uses the native `codesample` plugin (PrismJS `.token` spans). highlight.js (`public/hljs.min.js`) is loaded on the page but is only reachable from the dead `#note-preview` path — it highlights nothing live (see the dead-code note below). Rendered mode points TinyMCE's codesample plugin at full `window.Prism` bundle (`public/prism.min.js`) so every language offered by modal has grammar support. Prism token colors are injected into the TinyMCE iframe via `content_style` in `_tinyMCEContentFontStyle()` (the oxide dark content skin ships no `.token` CSS). `codemirror.min.js` and `prism.min.js` are loaded on page before TinyMCE/app.js. -- **Autosave**: htmx delayed PUT after typing pause (deferred while modals are open) -- **Markdown**: server-side `renderMarkdown()`, client-side Turndown `htmlToMarkdown()` -- **Auth**: reuses Joplin Server `sessionId` cookie -- **DB access**: reads direct from shared Postgres; writes go through stock Joplin Server API -- **Exports**: `POST /api/export/{docx,pdf,html}` in `app/routes/api.js` — DOCX and PDF are rendered server-side with `pandoc` (PDF via `weasyprint` plus pandoc's print CSS), HTML is a single self-contained file with inlined theme CSS, base64 images and base64 attachment links. `public/html-docx.js` is the legacy client-side DOCX path. - - Exports work in BOTH editor modes. Rendered mode posts TinyMCE HTML (`format:'html'`); markdown mode posts the markdown (`format:'markdown'`, via `_exportSource()`) and all three endpoints render it with `renderMarkdown()` before the shared strip-links / inline-images pipeline, so images and attachments come out the same in either mode. DOCX with no `format` keeps the legacy pandoc-native markdown input. Pinned by `tests/exportMarkdownMode.test.js` (pandoc stubbed) and the real-container download test in `playwright-tests/md-extras.spec.js`. -- **Ops surface**: admin user management (`/admin/users`), `pg_dump` backup/restore (`/admin/backups`, `/admin/restore`), break-glass recovery mode (`/recovery`), and login rate limiting — all served by the same sidecar process. -- **Script load order** (`app/templates/pages.js`): `htmx` → `turndown` → `codemirror` → `cm-extras` → `prism` → `tinymce` → `hljs` → `html-docx` → inline config → `app.js` (`defer`). `viewport.js` sits between the inline config and `app.js` (it reads `_joplockConfig.debug`). Every local script and stylesheet carries `?v=` (`app/assetVersion.js`, computed from the shipped files; `tests/pwaHead.test.js` enforces it), which is what lets the server serve them immutable. - -### Runtime Shape - -- Initial page load is full SSR HTML from `layoutPage()` in `app/templates/pages.js` -- After load, most interactions are fragment-driven via htmx -- The browser is intentionally thin: most state is DOM state, form state, or small client-only UI state in `public/app.js` -- There is no frontend router and no SPA store -- Desktop and mobile share the same server routes and most of the same editor code; mobile is a different screen shell around the same editor fragment - -### Request Flow - -1. Browser hits Joplock -2. Joplock validates `sessionId` against Joplin session/user tables -3. Fragment endpoints return HTML chunks; htmx swaps DOM -4. Writes serialize note/folder/resource and send upstream to stock Joplin Server API - -### Main UI Flow - -1. `GET /` renders the full shell -2. Navigation / notes / editor content is loaded from fragment endpoints -3. Selecting a folder swaps the notes list or nav tree fragment -4. Selecting a note swaps in `editorFragment()` -5. Autosave sends `PUT /fragments/editor/:id` with the current form state -6. Preview rendering uses `POST /fragments/preview` - -### Fragment Conventions - -- `app/templates/*.js` returns raw HTML strings, not JSX/templates/components -- htmx targets are mostly `#nav-panel`, `#notelist-panel`, `#editor-panel`, and mobile-specific targets like `#mobile-editor-body` -- Out-of-band swaps are used sparingly; note metadata is one example -- Client logic often relies on stable IDs, so be careful renaming DOM IDs used by inline JS - -## Sharing Ownership Model - -- One authoritative tree owned by sharer (`owner_id` never transferred). -- Recipients gain access via Joplin `user_items` + accepted `share_users` (auto-accept on invite). -- **Vault notebooks cannot be shared**: `POST /api/web/shares` rejects them with 400 `Vault notebooks cannot be shared` (`app/routes/shares.js`). -- `share_users.can_write` controls whether recipients can edit shared notes (default `1` — editable). Only the owner can move, delete, or stop-share. -- Move into a shared notebook sets `share_id`; move out clears it and drops recipient access to that item. -- Revoke/stop sharing removes recipient access; owner keeps folders/notes in place. -- Recipients can leave a shared notebook via the share dialog ("Leave notebook" button) → removes their `share_users` row + `user_items` entries. -- Only the share owner sees invite/remove/stop controls in the share dialog. Recipients see only their own access status and the Leave button. - -### Shared-note guard layers - -- **Route layer** (`app/routes/fragments.js`, `app/routes/api.js`): `resolveItemShareAccess` → `assertCanWrite`/`assertOwnerForDestructive` on create/update/delete/restore/move. `canWrite` respects `share_users.can_write` for recipients. Owner-only for move/delete regardless of `can_write`. Recipient creates in shared folder blocked with 403. -- **Proxy layer** (`app/proxy/shareProxyGuard.js`): inspects PUT and DELETE sync-proxy requests. PUT checks `resolveItemShareAccess` → `canWrite`. DELETE is owner-only. Inherits `noteIdFromItemPath`/`bufferRequest` from vault proxy guard. -- **Editor UI** (`app/templates/fragments.js`, `public/app.js`): `editorFragment` accepts `canWrite` param from route handler (queries `share_users.can_write`). When `canWrite=false` renders read-only banner, disables folder select, hides delete button, sets `contenteditable="false"` on title. When `canWrite=true` the editor is fully interactive. -- **Share-id propagation** (`app/items/shareAccess.js` for the field derivation, `app/routes/shares.js` for the write): `deriveShareFieldsForMove(targetFolder)` returns `{ shareId, isShared }` from the target folder, so a move sets/clears `share_id`. `ensureShareIdsOnNotebook()` writes `share_id` directly to the items DB content JSON. `createNote`/`updateNote` serialize these fields into Joplin note metadata. -- **Share dialog API** (`app/routes/shares.js`): uses Joplin Server's `/api/shares/:id/users` endpoints (not deprecated `/api/share_users`). PATCH/ACCEPT/REJECT/DELETE operations use DB-only writes since Joplin Server's newer API doesn't support mutations on individual share_users. `can_write` column managed via direct `share_users` table updates. The module also exports helpers reused elsewhere: `autoAcceptShareUser`, `populateUserItems`, `ensureShareIdsOnNotebook`, `STATUS_ACCEPTED`/`STATUS_WAITING`. -- **Never decode `items.content` for every item in share SQL** (`app/routes/shares.js`): `items` also holds binary resource blobs (`jop_type 0`, every uploaded image/PDF). `convert_from(content,'UTF8')` raises on them and aborts the whole statement — and the routes swallow DB errors, so invites silently stopped fanning out `user_items` (shared notebook never appeared) and revoke/stop-sharing silently left access behind as soon as the owner had one attachment. Read `share_id` only through `shareIdOf()` (a `CASE WHEN jop_type IN (1, 2)` guard; SQL gives no AND/OR evaluation-order guarantee). Pinned by `tests/sharesBinaryItems.test.js`. - -### File map - -| Layer | File | -|-------|------| -| Access helpers | `app/items/shareAccess.js` | -| Share dialog API routes | `app/routes/shares.js` | -| Proxy write guard | `app/proxy/shareProxyGuard.js` | -| Fragment write gates | `app/routes/fragments.js` | -| API write gates | `app/routes/api.js` | -| Share dialog template | `app/templates/shares.js` | -| Share dialog client | `public/app.js` (openShareDialog, inviteToShare, toggleShareWrite, leaveShareNotebook, etc.) | -| Read-only editor | `app/templates/fragments.js` (editorFragment) | -| Unit tests | `tests/shareAccess.test.js`, `tests/shareProxyGuard.test.js`, `tests/shareWriteGuards.test.js` | -| Playwright tests | `playwright-tests/share-modal.spec.js`, `playwright-tests/share-access.spec.js`, `playwright-tests/share-revoke-move.spec.js` | - -## Nested Notebooks - -Joplin notebooks nest via the folder's `parent_id`. Joplock renders, creates, moves and deletes them on desktop and mobile. There is **no schema change** and no sidecar table: the tree is just `jop_parent_id` on the shared `items` rows, so Joplin desktop/mobile/CLI see exactly the same hierarchy. - -### Where the logic lives - -| Concern | File | -|---|---| -| Pure tree helpers (build/flatten/descendants/ancestors/roll-up counts/`canNestUnder`/option labels) | `app/items/folderTree.js` | -| Create / move / delete / share-eligibility rules, shared by fragment + JSON routes | `app/items/folderOps.js` (`ctx.folderOps`, built in `createServer.js`) | -| Subtree share stamping, recipient grant/revoke | `app/routes/shares.js` (`collectSubtreeItems`, `setShareOnItems`, `createShareSync`) | -| Sync-proxy guard (no notebooks under a vault) | `app/proxy/vaultProxyGuard.js` (`enforceFolderWrite`) | -| Desktop tree markup | `navigationFragment` in `app/templates/fragments.js` | -| Mobile tree markup | `mobileFoldersFragment` in `app/templates/mobile.js` | -| Pickers | `notebookOptionsHtml` (editor select), `folderPickerOptions` + `GET /fragments/folder-options` (move / new-notebook modals) | -| Tests | `tests/folderTree.test.js`, `tests/folderOps.test.js`, `tests/folderRoutes.test.js`, `tests/nestedFoldersUi.test.js`, `playwright-tests/nested-folders.spec.js`, `playwright-tests/nested-share.spec.js` | - -### Tree semantics (keep them) - -- **Siblings keep input order.** `foldersByUserId` already orders by case-folded title; `folderTree` never re-sorts, so flat lists and the virtual "All Notes"/"Trash" rows come out unchanged. -- **Defensive like Joplin's `buildTree`:** a folder whose parent is not visible (not shared with this user, deleted) is shown at the top level; a parent cycle never loops (first folder of the cycle in list order becomes a root); depth is capped at `MAX_DEPTH`. Every folder always appears. -- **Counts:** the nav and mobile list show **rolled-up** counts (a collapsed parent is honest). `counts` from `folderNoteCountsByUserId` stays the **direct** count and still drives note pagination (`/fragments/folder-notes`), `data-note-count`, and "does this notebook have notes to fetch". Selecting a parent lists only its own notes; "All Notes" is the aggregate. -- Sub-notebooks sort/render **before** the notebook's own notes. - -### Desktop DOM contract (easy to break) - -``` -.nav-folder[data-folder-id][data-parent-id][data-depth][style=--nav-depth:N] - > .nav-folder-row - > .nav-folder-children (only when it has sub-notebooks; contains more .nav-folder) - > .nav-folder-notes[data-folder-id] (this notebook's OWN notes, lazy loaded) -``` - -- **Never** use `el.querySelector('.nav-folder-notes…')` or descendant CSS (`.nav-folder.collapsed .x`) on a notebook: sub-notebooks are inside it and would match first / leak collapse state. Use `navFolderNotesDiv(el)` (direct child) and the child combinator in CSS (`.nav-folder.collapsed > .nav-folder-notes`). -- **Multi-expand, no accordion.** Any number of notebooks can be open. Opening one also opens the notebooks above it (`toggleNavFolder`). `initNavPanel` trusts saved state; with no saved state the selected notebook **and its ancestors** open. Notes are lazy-loaded only for notebooks that are open **and visible** (`navFolderVisible`); expanding a parent loads already-open descendants (`navLoadOpenDescendants`). A notebook with 0 direct notes never requests a notes page. -- Saved state is `localStorage['joplock-nav-folders']` (notes-list open). - -### Drag and drop (desktop) - -Drag a notebook onto another to nest it; drop it on the **"Drop here to move to the top level"** strip (only shown while dragging a *nested* notebook) to un-nest it. Hovering a closed notebook that has sub-notebooks for 700 ms opens it so deeper targets are reachable. - -- The template decides who takes part: `navigationFragment` puts `draggable="true" data-dnd="1"` on the row of ordinary notebooks only (not All Notes, Trash, vaults or Joplin-encrypted notebooks). The script reads those attributes; the **server re-validates every move** (`PUT /fragments/folders/:id`), and a refusal is shown with `alert()` (e.g. "Stop sharing this notebook before moving it into another notebook"). -- Handlers (`navDragStart/Over/Leave/Drop/End`, `navDropAllowed`, `navMoveFolder` in `public/app.js`) are **delegated on `document`**, so they survive htmx re-renders. `dragenter` and `dragover` share one handler. `dragleave` must be judged by position when `relatedTarget` is null (Chromium fires it, with a null target, when the pointer crosses a row's own child elements); clearing the highlight on every `dragleave` made it flicker off. Visual changes happen a tick after `dragstart` because changing layout synchronously there cancels the drag in some browsers. -- Desktop only: touch has no HTML5 drag-and-drop, mobile uses the "Move notebook..." sheet. Real-mouse coverage: `playwright-tests/nested-folders.spec.js` ("drag and drop"); pure rules and lifecycle: `tests/nestedFoldersUi.test.js`. - -### Notebook order (A-Z / most recently updated) - -Per-user setting `folderSort` (`alpha` default | `recent`; Settings -> Appearance, plus a quick switch in the sidebar header `#nav-sort-btn` and the mobile folders header `#mobile-sort-btn`, both calling `toggleFolderSort()`; `` picks which label shows). - -- **What "recent" means** (`sortFoldersByRecent` in `app/items/folderTree.js`, data from `itemService.folderActivityByUserId`): a notebook's rank is the newest note activity **anywhere in its subtree** (so a parent rises with its busiest child, like Joplin's own "updated date" notebook sort), using the note's `user_updated_time` (falls back to the row time) and the notebook's creation time when nothing is in it. Trashed and conflict notes do not count. Because `user_updated_time` only moves on a real title/body edit (see Joplin Field Fidelity), **moving or trashing a note does not reorder notebooks**. -- **Parents always show.** The order is applied to the FLAT list before the tree is built, so reordering can never separate a notebook from its parent; siblings simply sort by their own rolled-up recency, ties keep alphabetical order. -- **Where it applies:** the sidebar (`navData` -> `orderFoldersForUser`) and the mobile folders screen. **Pickers are always alphabetical** (editor notebook ``'s option text, so the tree indentation carries over. -- Both expansion keys are cleared on logout (`pages.js`). - -### Rules enforced server side (`folderOps`) - -- **Structure:** not into itself or a descendant (mirrors Joplin's `canNestUnder`). Same-parent move is a no-op. -- **Delete = promote.** Sub-notebooks move up to the deleted notebook's parent; its notes move to that parent (top level: the existing top-level "General", created if missing). Deleting never deletes notes or sub-notebooks. (Old behaviour moved every note to General and orphaned sub-notebooks.) -- **Vaults are top-level leaves.** Cannot contain notebooks, cannot be moved, cannot be created nested (`POST /api/web/vaults` refuses a nested notebook or one with children), cannot end up in a share. The new-notebook modal disables the parent picker when "vault" is ticked. `vaultProxyGuard` also rejects a notebook PUT whose parent is a vault, **but see "Sync-proxy guards do not apply to stock Joplin clients" below: that guard is currently inert for real clients.** -- **Sharing:** - - Only the owner creates/moves/deletes inside a share (recipients get 403). - - A **share root stays top level** (Joplin's rule): moving one under anything is refused ("Stop sharing…"); sharing a nested notebook **lifts it to the top level first** (`prepareForShare`; the share dialog says so); a notebook already inside a share cannot be shared again; a subtree containing a vault cannot be shared. - - `ensureShareIdsOnNotebook` now stamps the **whole subtree** (every sub-notebook and the notes directly in each), writing `share_id`/`is_shared` into the item JSON **and** the `jop_share_id` column (falls back to JSON only if a server lacks the column). Creating a sub-notebook in a share inherits it; moving a subtree into/out of a share re-stamps it and grants/revokes recipient `user_items` (`createShareSync.setSubtreeShare`). - - **Subtree share changes go through the API.** `createShareSync.setSubtreeShare` re-saves every notebook and note of the subtree with the new share fields (`itemWriteService`, lossless thanks to pass-through) so Joplin Server emits change events. Silent database writes are invisible to stock clients: a subtree moved **out** of a share used to stay in the recipient's Joplin app. Items the API cannot rewrite (Joplin-encrypted) get the database sweep only. Verified with a real client recipient in `integration-tests/joplin-sharing.test.js`. - - **Stopping a share clears the dead `share_id` from the owner's whole tree** (notebook, sub-notebooks, notes; JSON and `jop_share_id`), like Joplin's own `unshareFolder`. Left in place, Joplin Server answers any later write of those items with `share not found` (422 — the note could no longer be edited or trashed) and **silently refuses to delete a root folder that still carries a share id** (returns 200, does nothing). Pinned by `playwright-tests/nested-share.spec.js` step 6. - - Recipient access keeps the existing Joplock convention: `user_items.item_id` holds the **`jop_id`** (Joplin Server's own fan-out uses `items.id`, which `itemAccessExpression` does not read). -- `updateFolder` now preserves `created_time`, `user_created_time`, `icon` and `master_key_id` (a rename/move used to reset them). - -### Endpoints - -- `PUT /fragments/folders/:id` and `PUT /api/web/folders/:id` accept `title` and/or `parentId` (`''` = top level; both in one write). `POST /fragments/folders` and `POST /api/web/folders` accept `parentId`. `DELETE` promotes. -- `GET /fragments/folder-options?exclude=&selected=` → `