diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index 4b97db3..e0f87f7 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -150,11 +150,12 @@ Joplin notebooks nest via the folder's `parent_id`. Joplock renders, creates, mo - **Structure:** not into itself or a descendant (mirrors Joplin's `canNestUnder`). Same-parent move is a no-op. - **Delete = promote.** Sub-notebooks move up to the deleted notebook's parent; its notes move to that parent (top level: the existing top-level "General", created if missing). Deleting never deletes notes or sub-notebooks. (Old behaviour moved every note to General and orphaned sub-notebooks.) -- **Vaults are top-level leaves.** Cannot contain notebooks, cannot be moved, cannot be created nested (`POST /api/web/vaults` refuses a nested notebook or one with children), cannot end up in a share. The new-notebook modal disables the parent picker when "vault" is ticked. The sync proxy rejects (403) a notebook PUT whose parent is a vault, so an external Joplin client cannot hang plaintext notebooks under a vault. +- **Vaults are top-level leaves.** Cannot contain notebooks, cannot be moved, cannot be created nested (`POST /api/web/vaults` refuses a nested notebook or one with children), cannot end up in a share. The new-notebook modal disables the parent picker when "vault" is ticked. `vaultProxyGuard` also rejects a notebook PUT whose parent is a vault, **but see "Sync-proxy guards do not apply to stock Joplin clients" below: that guard is currently inert for real clients.** - **Sharing:** - Only the owner creates/moves/deletes inside a share (recipients get 403). - A **share root stays top level** (Joplin's rule): moving one under anything is refused ("Stop sharing…"); sharing a nested notebook **lifts it to the top level first** (`prepareForShare`; the share dialog says so); a notebook already inside a share cannot be shared again; a subtree containing a vault cannot be shared. - `ensureShareIdsOnNotebook` now stamps the **whole subtree** (every sub-notebook and the notes directly in each), writing `share_id`/`is_shared` into the item JSON **and** the `jop_share_id` column (falls back to JSON only if a server lacks the column). Creating a sub-notebook in a share inherits it; moving a subtree into/out of a share re-stamps it and grants/revokes recipient `user_items` (`createShareSync.setSubtreeShare`). + - **Subtree share changes go through the API.** `createShareSync.setSubtreeShare` re-saves every notebook and note of the subtree with the new share fields (`itemWriteService`, lossless thanks to pass-through) so Joplin Server emits change events. Silent database writes are invisible to stock clients: a subtree moved **out** of a share used to stay in the recipient's Joplin app. Items the API cannot rewrite (Joplin-encrypted) get the database sweep only. Verified with a real client recipient in `integration-tests/joplin-sharing.test.js`. - **Stopping a share clears the dead `share_id` from the owner's whole tree** (notebook, sub-notebooks, notes; JSON and `jop_share_id`), like Joplin's own `unshareFolder`. Left in place, Joplin Server answers any later write of those items with `share not found` (422 — the note could no longer be edited or trashed) and **silently refuses to delete a root folder that still carries a share id** (returns 200, does nothing). Pinned by `playwright-tests/nested-share.spec.js` step 6. - Recipient access keeps the existing Joplock convention: `user_items.item_id` holds the **`jop_id`** (Joplin Server's own fan-out uses `items.id`, which `itemAccessExpression` does not read). - `updateFolder` now preserves `created_time`, `user_created_time`, `icon` and `master_key_id` (a rename/move used to reset them). @@ -169,6 +170,38 @@ Joplin notebooks nest via the folder's `parent_id`. Joplock renders, creates, mo Verified live while building this: a **note created in a shared notebook after the share** is not visible to a Joplock recipient, and a **note moved out of a share stays visible** to the recipient (the note write paths never call grant/revoke; only folder operations do). Fixing means calling `createShareSync.setItemsShare` from the note create/move paths. +## Joplin Field Fidelity (do not regress) + +Joplock shares one account with stock Joplin clients, so **a Joplock write must change only what the user edited**. Originally `serializeNote`/`serializeFolder` rebuilt every item from hard-coded values: a single body edit turned a to-do into a plain note, wiped author/source URL/location, flipped an HTML note to markdown, reset `order` and overwrote `source_application`. Measured with the real Joplin CLI synced through Joplock's proxy, then fixed. + +### How it works + +- **Pass-through serializer** (`app/items/joplinItem.js`): reproduces Joplin's `serialize` / `serialize_format` (canonical key order, ISO times, `\n` escaping, no empty body block). Values come from the item's **stored fields** and only the fields Joplock owns are overridden. +- **Items carry their stored fields.** `mapNoteRow` / `mapFolderRow` attach `fields` (the parsed `items.content` JSON). Every write path passes `existing`, so `serializeNote({ fields: existing.fields, ... })` writes everything else back untouched, including fields Joplock has never heard of (`is_locked`, `extracted_resource_ids`, `user_data`, `application_data`, ...). Defaults (`order = now`, zero geo strings, `markup_language 1`, `source_application net.cozic.joplock-web`) apply **only to brand-new items**; edited items keep their original `source_application`. +- **Joplock-owned fields:** title, body, parent, deleted time, `is_shared`/`share_id`, `updated_time` (always now, sync needs it). `user_updated_time` moves **only on a real title/body edit**; move, trash, restore and share changes preserve it (Joplin's `moveToFolder` semantics). +- **`fields` is internal.** `publicItem()` / `publicItems()` (`routes/_helpers.js`) strip it from JSON API responses. Keep using them on any new endpoint that returns a note or notebook. +- **The server only stores fields its own Joplin library knows** (it drops the rest on upload and when serving), so "unknown to Joplock" fields that can persist are ones a current Joplin writes. Joplin clients themselves also drop unknown fields on read. +- **Share id lives in a column.** Joplin Server moves `share_id` out of the item JSON into `items.jop_share_id` on the first API save. Always read it via `shareIdOfRow()` (column first, JSON fallback) / `shareIdOf()` in SQL. Reading only the JSON made a shared note look unshared after its first edit, so the *second* Joplock edit wrote `share_id: ''` and silently took the note away from every recipient. + +### Items Joplock must treat carefully + +| Item | Behaviour | +|---|---| +| `markup_language = 2` (HTML note) | Shown **rendered and read-only** (`app/htmlNoteRenderer.js`, allowlist sanitizer on untrusted HTML + TinyMCE's own sanitizing). Form is `data-html-note="1"`: no markdown/rendered switch, no toolbar, title not editable, `tinyMCESyncToTA` is a no-op. The server refuses any body change (403) but still allows move/trash/restore. Never convert it to markdown. | +| Joplin end-to-end encrypted (`jop_encryption_applied = 1`) | **Locked placeholder only.** `e2ee: true`, title/body blanked, no editor form at all. `updateNote`/`updateFolder` throw 403 (`assertNotE2ee`); a live one cannot be hard-deleted; a notebook holding any cannot be deleted (409, checked **before** any write so a delete never half-completes). | +| `is_conflict = 1` (conflict copy) | **Hidden** from lists, counts, headers and search (`conflictFilterSql`), like Joplin's "Conflicts" notebook. Stays untouched, and travels with its notebook when that is deleted (`includeConflicts`). | +| Attachments | `serializeResource` writes Joplin 3.x's OCR fields; `ocr_driver_id` is `1` (Joplin's default; `0` is a legacy quirk). | + +### Real-client integration tests (opt-in) + +`npm run setup:joplin-cli` installs the real Joplin terminal client under `~/.cache` (not in the repo). `npm run test:joplin` (dev stack up, admin env vars set) drives it headless through Joplock's `/joplin` proxy using a dedicated `compat-client@joplock.test` account (`integration-tests/joplin/harness.js`): `joplin-fidelity`, `joplin-guards`, `joplin-nested`, `joplin-sharing`. They create items in the client, sync, edit/move/trash in Joplock, sync back and compare **field by field** from the client's own SQLite. Raw uploads authenticate with `X-API-AUTH`, like a real client. Joplin Server applies share changes on a 10 s task, so the sharing tests wait ~13 s per step. One test is a tracked `todo` (below). + +### Known issues found with the real client (not fixed) + +- **Sync-proxy guards do not apply to stock Joplin clients.** `vaultProxyGuard` and `shareProxyGuard` identify the user with `authenticatedUser()`, which reads only Joplock's `sessionId` **cookie**. Joplin apps authenticate with an `X-API-AUTH` header and send no cookie, so both guards "stream through". Verified: a real client wrote a **plaintext note into a vault notebook and created a notebook under a vault**, and nothing stopped it. The vault/share/"no notebook under a vault" proxy protections described elsewhere in this guide therefore only hold for browser-origin traffic. Making them live needs `authenticatedUser` to accept `X-API-AUTH` **and** a fix for a second bug: when a request does authenticate, both guards call `bufferRequest()` on the same stream, so the second read never completes and the request hangs (seen as a hung cookie-authenticated PUT to `/joplin/...`). Behaviour will change for existing clients (403s on vault writes), so treat it as its own change. +- **Collaborator content is invisible to the share owner in Joplock.** Access checks use Joplock's own `user_items` rows keyed by `jop_id`; Joplin Server keys them by `items.id`. A note a collaborator creates in a Joplin app never shows for the owner in Joplock (it does in the owner's Joplin apps). Tracked as the `todo` test in `joplin-sharing.test.js`. Fixing it means changing the access model together with Joplock's own removal paths (otherwise removed users keep access). +- Notes created in a share after sharing do not reach a **Joplock** recipient, and a note moved out of a share stays visible to a Joplock recipient (the note write paths never grant/revoke Joplock's `jop_id` rows; only folder operations do). Stock-client recipients are handled correctly by the server. + ## Core Rules 1. Do not modify Joplin Server source for Joplock features unless explicitly approved. @@ -207,7 +240,7 @@ Server (`app/routes/_helpers.js`) rules: - `assertVaultNoteBodyEncrypted(vaultService, userId, existingParentId, targetParentId, body, noteId, opts)` parses the ciphertext blob and rejects the write when `meta.vault` mismatches the target folder's vault, or `meta.noteId` (if present) mismatches the target note. - **Vault boundary enforcement** (three layers): 1. **ParentId immutability / no conflict copies**: if a note currently lives in a vault folder, the server rejects any PUT that changes its `parentId` (400: `Vault notes cannot be moved to a different folder`). Vault notes cannot change folder — the folder select is disabled client-side and the server enforces this on every write path. Conflict `createCopy` is also rejected for vault notes: ciphertext is bound to source note id, while copying unlocked DOM content would make a plaintext duplicate. - 2. **Ciphertext required inside vaults**: saving to a vault folder always requires an encrypted body. Same-folder saves and sync-proxy writes are both covered. `enforceExistingVault=true` (sync-proxy only) extends this to prevent external Joplin clients from stripping ciphertext by re-parenting. + 2. **Ciphertext required inside vaults**: saving to a vault folder always requires an encrypted body. Same-folder saves are covered; sync-proxy writes are covered **only for cookie-authenticated (browser) traffic, not stock Joplin clients** (see "Known issues found with the real client"). `enforceExistingVault=true` (sync-proxy only) extends this to prevent external Joplin clients from stripping ciphertext by re-parenting. 3. **Metadata integrity**: encrypted blobs carry `vault` (folder id) and `noteId` (bound target); both are validated against the destination to prevent cross-vault or cross-note ciphertext smuggling. - **Client-side**: the folder select (`#editor-folder-select`) is rendered `disabled` for vault-protected notes and stays disabled after unlock — vault notes cannot change parent folder through the UI. The only reachable folder-change path is plain→vault (encrypt on move). - Every note-write path passes the target `noteId`: `app/routes/api.js` (PUT), `app/routes/fragments.js` (autosave PUT), `app/routes/history.js` (restore), `app/proxy/vaultProxyGuard.js` (sync proxy). @@ -373,6 +406,7 @@ the service that owns them (e.g. `app/settingsService.js` also carries the ### Tests - `tests/*.test.js` — unit/integration tests (`node:test`; several suites extract real functions out of `public/app.js` and run them under a JSDOM harness) - Run: `npm test` (i.e. `node --test tests/*.test.js`) +- Real Joplin client compatibility (opt-in, needs the dev stack): `npm run setup:joplin-cli` once, then `npm run test:joplin` (`integration-tests/*.test.js`) - `playwright-tests/*.spec.js` — browser E2E against a live dev stack; run with `npm run test:ui` ### Deployment @@ -966,6 +1000,7 @@ Recommended inner loop: ## Recently Completed Work +- **Joplin field fidelity + compatibility harness**: pass-through serializer, stored `fields` on every item, HTML notes rendered read-only, Joplin-E2EE items as locked placeholders, conflict copies hidden, share id read from `jop_share_id`, OCR fields on uploads; opt-in real-client tests (`npm run test:joplin`). Also fixed along the way: **restoring a note from the trash returned 404 for every trashed note** (the ownership lookup ignored trashed notes), and **editing a shared note twice in Joplock silently unshared it**. See "Joplin Field Fidelity". - **Nested notebooks**: Joplin's `parent_id` hierarchy now renders as a tree on desktop (multi-expand, rolled-up counts, ancestors open) and mobile (inline expandable tree), with create-under-parent, move (tree picker), delete-promotes, whole-subtree sharing, and vaults as top-level leaves. `app/items/folderTree.js` + `app/items/folderOps.js`. `updateFolder` also stopped resetting `created_time`/`icon`. - **Rich-mode uploads now insert through the live editor**: uploading from the modal or the older `#file-upload` picker in rendered mode used to target the hidden `#note-body` textarea, which TinyMCE's own lazy sync overwrote before the debounced save fired — the attachment silently vanished. `_captureUploadInsertTarget()` now captures a TinyMCE caret range (`{mode:'tinymce', rng}`) when the host is visible and the editor is not read-only, and `_insertResourceIntoTinyMCEFromMarkdown()` rebuilds the ``/`` from the saved `![](:/id)` markdown, inserts it with the same `_tinyMCEBlockAttachmentHtml()` padding as drag-drop, and syncs `#note-body` immediately. All three rich-mode insert paths call `_endTinyMCEPostLoadWindow()` so the insert is treated as a real edit instead of post-load echo (otherwise the reconcile re-baselines the hash and the save sees "unchanged"). Read-only (mobile rendered) keeps the textarea/CM fallback. See "Upload behavior" above. - **flushSave uses `keepalive: true`**: the unload-time flush (visibilitychange, tab close, note-switch navigation) was being aborted mid-flight — Chromium sent a truncated request with no `Cookie` header (401) and the pending edits were lost. Pinned by `tests/saveIdentityGuard.test.js`. diff --git a/app/createServer.js b/app/createServer.js index 3d7a106..cb5034b 100644 --- a/app/createServer.js +++ b/app/createServer.js @@ -135,7 +135,7 @@ const createServer = options => { itemService, itemWriteService, vaultService, - shareSync: database ? routeShares.createShareSync({ itemService, database }) : null, + shareSync: database ? routeShares.createShareSync({ itemService, itemWriteService, database }) : null, }); const upstreamRequestContext = _request => ({ diff --git a/app/items/folderOps.js b/app/items/folderOps.js index e215d5f..a1a33cd 100644 --- a/app/items/folderOps.js +++ b/app/items/folderOps.js @@ -105,6 +105,7 @@ const createFolderOps = ({ itemService, itemWriteService, vaultService, shareSyn if (shareFields.shareId && shareSync) { await shareSync.setSubtreeShare({ ownerId: user.id, folderId: created.id, shareId: shareFields.shareId, previousShareId: '', + sessionId: user.sessionId, requestContext, }); } return { id: created.id, parentId: parent, ...shareFields }; @@ -168,6 +169,7 @@ const createFolderOps = ({ itemService, itemWriteService, vaultService, shareSyn if (shareChanges && shareSync) { await shareSync.setSubtreeShare({ ownerId: user.id, folderId, shareId: newShareId, previousShareId: oldShareId, + sessionId: user.sessionId, requestContext, }); } return { changed: true, folder: state.folder, shareChanged: shareChanges }; @@ -218,6 +220,7 @@ const createFolderOps = ({ itemService, itemWriteService, vaultService, shareSyn if (childShareId !== newShareId && shareSync) { await shareSync.setSubtreeShare({ ownerId: user.id, folderId: child.id, shareId: newShareId, previousShareId: childShareId, + sessionId: user.sessionId, requestContext, }); } } diff --git a/app/items/itemService.js b/app/items/itemService.js index 0f26908..93bced3 100644 --- a/app/items/itemService.js +++ b/app/items/itemService.js @@ -12,6 +12,14 @@ const decodeItemContent = content => { return JSON.parse(raw); }; +// Joplin Server keeps an item's share id in the `jop_share_id` COLUMN and drops it +// from the stored JSON as soon as the item is saved through the sync API (any save, +// by Joplock or a Joplin app). The JSON copy only exists on items Joplock stamped +// itself and goes stale. Reading only the JSON made a shared item look unshared after +// its first edit, so the next Joplock edit wrote `share_id: ''` and silently took the +// note away from every recipient. +const shareIdOfRow = (row, content) => `${row.jop_share_id || content.share_id || ''}`; + // Items encrypted by Joplin's own end-to-end encryption (not Joplock vaults). // The stored content is ciphertext: title/body are empty and rewriting the item // from here would replace the ciphertext with plaintext. They are shown as a @@ -40,8 +48,8 @@ const mapFolderRow = row => { masterKeyId: content.master_key_id || '', updatedTime: Number(row.jop_updated_time || content.updated_time || 0), ownerId: row.owner_id || '', - shareId: content.share_id || '', - isShared: !!(Number(content.is_shared || 0)), + shareId: shareIdOfRow(row, content), + isShared: !!(Number(content.is_shared || 0)) || !!shareIdOfRow(row, content), }; }; @@ -93,8 +101,8 @@ const mapNoteRow = row => { createdTime: Number(content.created_time || row.created_time || 0), updatedTime: Number(row.jop_updated_time || content.updated_time || 0), ownerId: row.owner_id || '', - shareId: content.share_id || '', - isShared: !!(Number(content.is_shared || 0)), + shareId: shareIdOfRow(row, content), + isShared: !!(Number(content.is_shared || 0)) || !!shareIdOfRow(row, content), }; }; @@ -110,8 +118,8 @@ const mapNoteHeaderRow = row => { deletedTime: Number(row.deleted_time || 0), updatedTime: Number(row.jop_updated_time || 0), ownerId: row.owner_id || '', - shareId: row.share_id || '', - isShared: !!Number(row.is_shared || 0), + shareId: `${row.jop_share_id || row.share_id || ''}`, + isShared: !!Number(row.is_shared || 0) || !!(row.jop_share_id || row.share_id), }; }; @@ -176,7 +184,7 @@ const createItemService = database => { return { async foldersByUserId(userId) { const result = await database.query(` - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id FROM items WHERE jop_type = $2${deletedFilterSql('exclude')} AND ${itemAccessExpression()} ORDER BY LOWER(COALESCE(convert_from(content, 'UTF8')::json->>'title', '')) ASC, created_time ASC @@ -187,7 +195,7 @@ const createItemService = database => { async folderByUserIdAndJopId(userId, folderId) { const result = await database.query(` - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id FROM items WHERE jop_type = $2 AND jop_id = $3 AND ${itemAccessExpression()} LIMIT 1 @@ -210,7 +218,7 @@ const createItemService = database => { } const result = await database.query(` - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id FROM items ${where} ORDER BY jop_updated_time DESC, created_time DESC @@ -228,6 +236,7 @@ const createItemService = database => { jop_updated_time, owner_id, jop_encryption_applied, + jop_share_id, COALESCE(convert_from(content, 'UTF8')::json->>'title', '') AS title, COALESCE((convert_from(content, 'UTF8')::json->>'deleted_time')::bigint, 0) AS deleted_time, COALESCE(convert_from(content, 'UTF8')::json->>'share_id', '') AS share_id, @@ -296,6 +305,7 @@ const createItemService = database => { jop_updated_time, owner_id, jop_encryption_applied, + jop_share_id, COALESCE(convert_from(content, 'UTF8')::json->>'title', '') AS title, COALESCE((convert_from(content, 'UTF8')::json->>'deleted_time')::bigint, 0) AS deleted_time, COALESCE(convert_from(content, 'UTF8')::json->>'share_id', '') AS share_id, @@ -315,9 +325,9 @@ const createItemService = database => { if (!termSql) return []; const limitIdx = 2 + termParams.length + 1; const result = await database.query(` - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id FROM ( - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id, ${safeJsonExpression("convert_from(content, 'UTF8')")} AS parsed FROM items WHERE jop_type = $2 AND ${itemAccessExpression()} @@ -335,7 +345,7 @@ const createItemService = database => { async noteByUserIdAndJopId(userId, noteId, options = {}) { const deleted = options.deleted || 'exclude'; const result = await database.query(` - SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied + SELECT id, jop_id, jop_parent_id, jop_updated_time, created_time, content, owner_id, jop_encryption_applied, jop_share_id FROM items WHERE jop_type = $2 AND jop_id = $3${deletedFilterSql(deleted)} AND ${itemAccessExpression()} LIMIT 1 diff --git a/app/items/itemWriteService.js b/app/items/itemWriteService.js index ca0d11d..c098808 100644 --- a/app/items/itemWriteService.js +++ b/app/items/itemWriteService.js @@ -150,7 +150,7 @@ const serializeResource = resource => { ocr_details: '', ocr_status: 0, ocr_error: '', - ocr_driver_id: 0, + ocr_driver_id: 1, // Joplin's default (PrintedText); 0 is a legacy quirk it only tolerates }, overrides, }), diff --git a/app/routes/shares.js b/app/routes/shares.js index 6ee1272..f5aaeea 100644 --- a/app/routes/shares.js +++ b/app/routes/shares.js @@ -48,16 +48,21 @@ const inviteesList = data => { const shareFolderId = share => share && (share.folder_id || share.notebook_id || share.folderId || share.notebookId || ''); -// `share_id` of an item, decoded from its JSON content. +// `share_id` of an item. +// +// The authoritative value is the `jop_share_id` COLUMN: Joplin Server moves share_id +// out of the item JSON on every API save, so the JSON copy is absent or stale for any +// item that has been edited. The JSON is only a fallback for items stamped directly in +// the database. // // `items` also holds binary resource blobs (jop_type 0, e.g. every uploaded // image/PDF). Those are not valid UTF-8, so convert_from() raises -// "invalid byte sequence" and aborts the WHOLE statement — and SQL gives no +// "invalid byte sequence" and aborts the WHOLE statement - and SQL gives no // guarantee that a sibling `jop_type = ...` predicate runs first. The CASE // guarantees only note/folder rows are ever decoded. (With an unguarded // expression every share fan-out / revoke silently did nothing once the // account owned a single uploaded attachment: the callers swallow errors.) -const shareIdOf = (alias = '') => `(CASE WHEN ${alias}jop_type IN (1, 2) THEN convert_from(${alias}content, 'UTF8')::json->>'share_id' END)`; +const shareIdOf = (alias = '') => `(COALESCE(NULLIF(${alias}jop_share_id, ''), CASE WHEN ${alias}jop_type IN (1, 2) THEN convert_from(${alias}content, 'UTF8')::json->>'share_id' END))`; const newId = () => randomBytes(16).toString('hex'); @@ -201,24 +206,59 @@ const revokeRecipientAccess = async (database, shareId, itemIds) => { // a database (unit tests with fakes). // shareId the share the items now belong to ('' = none) // previousShareId the share they belonged to before ('' = none) -const createShareSync = ({ itemService, database }) => ({ - // A notebook and everything beneath it changes share. - async setSubtreeShare({ ownerId, folderId, shareId = '', previousShareId = '' }) { - if (!database) return; - const { folderIds, noteIds } = await collectSubtreeItems(itemService, database, ownerId, folderId); - const ids = folderIds.concat(noteIds); - await setShareOnItems(database, ownerId, ids, shareId); - if (previousShareId && previousShareId !== shareId) await revokeRecipientAccess(database, previousShareId, ids); - if (shareId) await grantRecipientAccess(database, ownerId, shareId, folderId); - }, - // Individual items (e.g. notes moved out of a deleted notebook) change share. - async setItemsShare({ ownerId, itemIds, shareId = '', previousShareId = '', rootFolderId = '' }) { - if (!database || !itemIds || !itemIds.length) return; - await setShareOnItems(database, ownerId, itemIds, shareId); - if (previousShareId && previousShareId !== shareId) await revokeRecipientAccess(database, previousShareId, itemIds); - if (shareId) await grantRecipientAccess(database, ownerId, shareId, rootFolderId); - }, -}); +const createShareSync = ({ itemService, itemWriteService, database }) => { + // Re-save every notebook and note of the subtree THROUGH THE API with the new share + // fields. This is what makes stock Joplin clients follow along: Joplin Server only + // adds/removes a recipient's copy (and tells their client to delete it) when it + // sees a change event with previous_share_id, and only API writes create those. + // Direct database writes are silent, so a subtree moved OUT of a share used to + // stay in the recipient's Joplin app. Items the API cannot rewrite (Joplin + // end-to-end encrypted ones) are left to the database stamping below. + const stampThroughApi = async ({ ownerId, folderIds, noteIds, shareId, sessionId, requestContext }) => { + if (!itemWriteService || !sessionId) return; + const fields = { shareId, isShared: !!shareId }; + const skipped = []; + for (const id of folderIds) { + const folder = await itemService.folderByUserIdAndJopId(ownerId, id).catch(() => null); + if (!folder || folder.e2ee || (folder.shareId || '') === shareId) continue; + try { await itemWriteService.updateFolder(sessionId, folder, fields, requestContext); } catch (error) { skipped.push(error); } + } + for (const id of noteIds) { + const note = await itemService.noteByUserIdAndJopId(ownerId, id, { deleted: 'all' }).catch(() => null); + if (!note || note.e2ee || (note.shareId || '') === shareId) continue; + try { await itemWriteService.updateNote(sessionId, note, fields, requestContext); } catch (error) { skipped.push(error); } + } + if (skipped.length) throw skipped[0]; + }; + + return { + // A notebook and everything beneath it changes share. + async setSubtreeShare({ ownerId, folderId, shareId = '', previousShareId = '', sessionId, requestContext }) { + if (!database) return; + const { folderIds, noteIds } = await collectSubtreeItems(itemService, database, ownerId, folderId); + const ids = folderIds.concat(noteIds); + // API first (it compares against the current share), then the database sweep + // for anything the API could not write, then Joplock's own recipient rows. + let apiError = null; + try { + await stampThroughApi({ ownerId, folderIds, noteIds, shareId, sessionId, requestContext }); + } catch (error) { apiError = error; } + await setShareOnItems(database, ownerId, ids, shareId); + if (previousShareId && previousShareId !== shareId) await revokeRecipientAccess(database, previousShareId, ids); + if (shareId) await grantRecipientAccess(database, ownerId, shareId, folderId); + if (apiError) throw apiError; + }, + // Individual items (e.g. notes moved out of a deleted notebook) change share. The + // caller has already written them through the API; this keeps Joplock's own + // recipient rows and the stored JSON in step. + async setItemsShare({ ownerId, itemIds, shareId = '', previousShareId = '', rootFolderId = '' }) { + if (!database || !itemIds || !itemIds.length) return; + await setShareOnItems(database, ownerId, itemIds, shareId); + if (previousShareId && previousShareId !== shareId) await revokeRecipientAccess(database, previousShareId, itemIds); + if (shareId) await grantRecipientAccess(database, ownerId, shareId, rootFolderId); + }, + }; +}; const createShareUpstream = async (ctx, sessionId, notebookId) => { // Joplin Server has used both folder_id and notebook_id historically. diff --git a/app/templates/pages.js b/app/templates/pages.js index d1f0560..241fcdc 100644 --- a/app/templates/pages.js +++ b/app/templates/pages.js @@ -10,7 +10,7 @@ const { } = require('./shared'); const { noteMetaFragment } = require('./fragments'); -const ASSET_VERSION = '20261008nestedfolders'; +const ASSET_VERSION = '20261009joplinfidelity'; const noteFontFamilyCSS = (settings) => { const f = settings.noteFontFamily || 'sans'; diff --git a/integration-tests/joplin-guards.test.js b/integration-tests/joplin-guards.test.js new file mode 100644 index 0000000..e02809c --- /dev/null +++ b/integration-tests/joplin-guards.test.js @@ -0,0 +1,217 @@ +'use strict'; + +// Items Joplock must never mangle in a Joplin account: HTML notes, end-to-end +// encrypted items and conflict copies - plus attachments, which the client must +// be able to download like any other resource. + +const { describe, it, before, after } = require('node:test'); +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const path = require('node:path'); +const os = require('node:os'); +const { skipReason, sleep, JoplinClient, connectAccount } = require('./joplin/harness'); + +const skip = skipReason(); +const PREFIX = `jc-grd-${Date.now().toString(36)}`; +const id32 = () => crypto.randomBytes(16).toString('hex'); +const TS = '2026-01-02T03:04:05.678Z'; + +// A plain note as a stock client would upload it. +const rawNote = (id, parent, title, body, extra = []) => [ + title, '', body, '', + `id: ${id}`, `parent_id: ${parent}`, `created_time: ${TS}`, `updated_time: ${TS}`, + 'is_conflict: 0', 'latitude: 0.00000000', 'longitude: 0.00000000', 'altitude: 0.0000', 'author: ', 'source_url: ', + 'is_todo: 0', 'todo_due: 0', 'todo_completed: 0', 'source: joplin-desktop', 'source_application: net.cozic.joplin-desktop', + 'application_data: ', 'order: 1767323045678', `user_created_time: ${TS}`, `user_updated_time: ${TS}`, + 'encryption_cipher_text: ', 'encryption_applied: 0', 'markup_language: 1', 'is_shared: 0', 'share_id: ', + 'conflict_original_id: ', 'master_key_id: ', 'user_data: ', 'deleted_time: 0', ...extra, 'type_: 1', +].join('\n'); + +describe('HTML notes', { skip, concurrency: false }, () => { + let jp; let client; + before(async () => { jp = await connectAccount(); client = await new JoplinClient('html').init(); }); + after(async () => { if (client) client.close(); if (jp) await jp.cleanup(PREFIX); }); + + it('are rendered sanitized and read-only; content edits are refused, moves are not', async () => { + const book = `${PREFIX}-html`; + const title = `${PREFIX}-clip`; + const html = '

Clipped page

hello world

bad'; + await client.run('mkbook', book); + await client.run('use', book); + await client.run('mknote', title); + await client.run('set', title, 'markup_language', '2'); + await client.run('set', title, 'body', html); + await client.sync(); + const n = await client.note(title); + assert.equal(n.markup_language, 2); + + // Joplock shows it rendered and read-only, with the dangerous bits gone. + const ed = await jp.ok('GET', `/fragments/editor/${n.id}`); + assert.match(ed.text, /data-html-note="1"/); + assert.match(ed.text, /HTML note/); + const slot = /data-rendered-body="([^"]*)"/.exec(ed.text); + assert.ok(slot, 'rendered body present'); + const rendered = slot[1].replace(/"/g, '"').replace(/</g, '<').replace(/>/g, '>').replace(/&/g, '&').replace(/'/g, "'"); + assert.match(rendered, /

Clipped page<\/h2>/); + assert.match(rendered, /world<\/b>/); + assert.doesNotMatch(rendered, /script|onerror|javascript:|alert/i); + + // A content edit is refused and the stored HTML is untouched. + const before = await jp.rawItem(n.id); + const refused = await jp.updateNote(n.id, { title, body: '

overwritten

' }); + assert.equal(refused.status, 403, refused.text); + assert.equal(await jp.rawItem(n.id), before, 'item on the server is byte-identical'); + + // The real autosave path (form PUT) for a MOVE of an HTML note: allowed, body kept. + const dest = await jp.createFolder(`${PREFIX}-html-dest`); + const moved = await jp.request('PUT', `/fragments/editor/${n.id}`, { + form: { title, body: n.body, parentId: dest, currentFolderId: dest }, + }); + assert.equal(moved.status, 200, moved.text); + await sleep(300); + await client.sync(); + const after = await client.note(title); + assert.equal(after.parent_id, dest); + assert.equal(after.markup_language, 2); + assert.equal(after.body, html, 'HTML body byte-for-byte'); + }); +}); + +describe('Joplin end-to-end encrypted items', { skip, concurrency: false }, () => { + let jp; + before(async () => { jp = await connectAccount(); }); + after(async () => { if (jp) await jp.cleanup(PREFIX); }); + + // What a client uploads for an encrypted item: ids, parent and the ciphertext only. + const encryptedItem = (type, id, parent) => [ + `id: ${id}`, ...(type === 1 ? [`parent_id: ${parent}`] : []), + 'encryption_cipher_text: JED01000007fakeciphertextfakeciphertextfakeciphertext', + 'encryption_applied: 1', + `type_: ${type}`, + ].join('\n'); + + it('notes show as a locked placeholder and can never be changed from Joplock', async () => { + const folder = await jp.createFolder(`${PREFIX}-e2ee`); + const id = id32(); + await jp.putRawItem(id, encryptedItem(1, id, folder)); + const before = await jp.rawItem(id); + assert.match(before, /encryption_applied: 1/); + + const headers = await jp.noteHeaders(); + const h = headers.find(x => x.id === id); + assert.ok(h, 'listed (not silently dropped)'); + assert.match(h.title, /Encrypted note/); + + const ed = await jp.ok('GET', `/fragments/editor/${id}`); + assert.match(ed.text, /editor-e2ee/); + assert.doesNotMatch(ed.text, / { + const parent = await jp.createFolder(`${PREFIX}-e2ee-parent`); + const encFolder = id32(); + await jp.putRawItem(encFolder, [ + `id: ${encFolder}`, `parent_id: ${parent}`, 'encryption_cipher_text: JED01000007fakefolderciphertext', 'encryption_applied: 1', 'type_: 2', + ].join('\n')); + const folders = await jp.folders(); + const f = folders.find(x => x.id === encFolder); + assert.ok(f, 'listed'); + assert.match(f.title, /Encrypted notebook/); + const before = await jp.rawItem(encFolder); + + assert.equal((await jp.request('PUT', `/api/web/folders/${encFolder}`, { json: { title: 'x' } })).status, 403); + assert.equal((await jp.request('PUT', `/api/web/folders/${encFolder}`, { json: { parentId: '' } })).status, 403); + assert.equal((await jp.request('DELETE', `/api/web/folders/${encFolder}`)).status, 403); + const del = await jp.request('DELETE', `/api/web/folders/${parent}`); + assert.equal(del.status, 409, del.text); + assert.equal(await jp.rawItem(encFolder), before, 'untouched'); + assert.ok((await jp.folders()).some(x => x.id === parent), 'the parent was not half-deleted'); + }); +}); + +describe('Conflict copies', { skip, concurrency: false }, () => { + let jp; let client; + before(async () => { jp = await connectAccount(); client = await new JoplinClient('cf').init(); }); + after(async () => { if (client) client.close(); if (jp) await jp.cleanup(PREFIX); }); + + it('are hidden from Joplock lists like in Joplin, and travel with a deleted notebook', async () => { + const parent = await jp.createFolder(`${PREFIX}-conf`); + const normal = id32(); + const conflict = id32(); + await jp.putRawItem(normal, rawNote(normal, parent, `${PREFIX}-normal`, 'normal body')); + await jp.putRawItem(conflict, rawNote(conflict, parent, `${PREFIX}-conflict`, 'conflict body', ['is_conflict: 1']).replace('is_conflict: 0\n', '').replace('conflict_original_id: ', `conflict_original_id: ${normal}`)); + + const headers = await jp.noteHeaders(); + assert.ok(headers.some(h => h.id === normal), 'normal note listed'); + assert.ok(!headers.some(h => h.id === conflict), 'conflict note hidden from the note list'); + const inFolder = (await jp.ok('GET', `/api/web/notes?folderId=${parent}`)).data.items; + assert.ok(!inFolder.some(n => n.id === conflict)); + + // the real client still has it (as a conflict) + await client.sync(); + const theirs = await client.noteById(conflict); + assert.ok(theirs, 'the client has the conflict copy'); + assert.equal(theirs.is_conflict, 1); + + // deleting the notebook moves BOTH notes (the hidden one must not be left pointing at nothing) + assert.equal((await jp.request('DELETE', `/api/web/folders/${parent}`)).status, 204); + await sleep(300); + await client.sync(); + const normalAfter = await client.noteById(normal); + const conflictAfter = await client.noteById(conflict); + assert.notEqual(normalAfter.parent_id, parent, 'normal note moved out'); + assert.notEqual(conflictAfter.parent_id, parent, 'conflict copy moved out too'); + assert.equal(conflictAfter.is_conflict, 1, 'and is still a conflict copy'); + }); +}); + +describe('Attachments uploaded in Joplock', { skip, concurrency: false }, () => { + let jp; let client; let tmp; + before(async () => { jp = await connectAccount(); client = await new JoplinClient('res').init(); tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'jc-res-')); }); + after(async () => { if (client) client.close(); if (jp) await jp.cleanup(PREFIX); if (tmp) fs.rmSync(tmp, { recursive: true, force: true }); }); + + // 1x1 PNG + const PNG = Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==', 'base64'); + + it('are downloaded by the client and look exactly like client-made attachments', async () => { + // reference: an attachment made by the real client + const book = `${PREFIX}-res`; + await client.run('mkbook', book); + await client.run('use', book); + await client.run('mknote', `${PREFIX}-res-note`); + const file = path.join(tmp, 'ref.png'); + fs.writeFileSync(file, PNG); + await client.run('attach', `${PREFIX}-res-note`, file); + await client.sync(); + const [ref] = await client.all('SELECT * FROM resources WHERE filename = ? OR title = ?', ['ref.png', 'ref.png']); + assert.ok(ref, 'client-made resource exists'); + + // the same image uploaded through Joplock's UI endpoint, linked from a note + const resId = await jp.uploadResource('upl.png', 'image/png', PNG); + const folder = (await jp.folders()).find(f => f.title === book); + await jp.createNote(`${PREFIX}-res-note2`, `![upl.png](:/${resId})`, folder.id); + await sleep(400); + await client.sync(); + const mine = (await client.all('SELECT * FROM resources WHERE id = ?', [resId]))[0]; + assert.ok(mine, 'the client received the resource record'); + assert.equal(mine.mime, 'image/png'); + assert.equal(mine.size, PNG.length); + assert.equal(mine.file_extension, 'png'); + for (const field of ['encryption_applied', 'encryption_blob_encrypted', 'ocr_status', 'ocr_driver_id', 'ocr_error', 'ocr_text', 'is_shared']) { + assert.equal(mine[field], ref[field], `${field} matches what Joplin itself writes`); + } + const blob = path.join(client.profile, 'resources', `${resId}.png`); + assert.ok(fs.existsSync(blob), 'the client downloaded the attachment'); + assert.deepEqual(fs.readFileSync(blob), PNG, 'byte-identical'); + }); +}); diff --git a/integration-tests/joplin-nested.test.js b/integration-tests/joplin-nested.test.js new file mode 100644 index 0000000..3bb0804 --- /dev/null +++ b/integration-tests/joplin-nested.test.js @@ -0,0 +1,139 @@ +'use strict'; + +// Nested notebooks between Joplock and the real Joplin client, both directions. + +const { describe, it, before, after } = require('node:test'); +const assert = require('node:assert/strict'); +const { skipReason, sleep, JoplinClient, connectAccount } = require('./joplin/harness'); + +const skip = skipReason(); +const PREFIX = `jc-nest-${Date.now().toString(36)}`; + +describe('Nested notebooks between Joplock and a Joplin client', { skip, concurrency: false }, () => { + let jp; let client; + before(async () => { jp = await connectAccount(); client = await new JoplinClient('nest').init(); }); + after(async () => { if (client) client.close(); if (jp) await jp.cleanup(PREFIX); }); + const pull = async () => { await sleep(300); await client.sync(); }; + const parentOf = async id => (await jp.folders()).find(f => f.id === id)?.parentId; + + it('a tree built in the Joplin client appears nested in Joplock', async () => { + const [a, b, c] = ['a', 'b', 'c'].map(x => `${PREFIX}-t1${x}`); + await client.run('mkbook', a); + await client.run('mkbook', '-p', a, b); + await client.run('mkbook', '-p', b, c); + await client.run('use', c); + await client.run('mknote', `${PREFIX}-t1-leaf`); + await client.sync(); + + const [fa, fb, fc] = await Promise.all([a, b, c].map(t => client.folder(t))); + assert.equal(fb.parent_id, fa.id, 'the client itself nested them'); + const folders = await jp.folders(); + const byId = Object.fromEntries(folders.map(f => [f.id, f])); + assert.equal(byId[fa.id].parentId, ''); + assert.equal(byId[fb.id].parentId, fa.id); + assert.equal(byId[fc.id].parentId, fb.id); + + // and the nav really renders them nested + const nav = (await jp.ok('GET', '/fragments/nav')).text; + assert.match(nav, new RegExp(`data-folder-id="${fc.id}"[^>]*data-parent-id="${fb.id}"[^>]*data-depth="2"`)); + const leaf = (await jp.noteHeaders()).find(n => n.title === `${PREFIX}-t1-leaf`); + assert.equal(leaf.parentId, fc.id); + }); + + it('a tree built in Joplock arrives nested in the Joplin client', async () => { + const a = await jp.createFolder(`${PREFIX}-t2a`); + const b = await jp.createFolder(`${PREFIX}-t2b`, a); + const c = await jp.createFolder(`${PREFIX}-t2c`, b); + await jp.createNote(`${PREFIX}-t2-leaf`, 'leaf body', c); + await pull(); + assert.equal((await client.folderById(a)).parent_id, ''); + assert.equal((await client.folderById(b)).parent_id, a); + assert.equal((await client.folderById(c)).parent_id, b); + assert.equal((await client.note(`${PREFIX}-t2-leaf`)).parent_id, c); + }); + + it('moves made in Joplock reach the client, and moves made in the client reach Joplock', async () => { + const a = await jp.createFolder(`${PREFIX}-t3a`); + const b = await jp.createFolder(`${PREFIX}-t3b`); + const c = await jp.createFolder(`${PREFIX}-t3c`, a); + await pull(); + + // Joplock: c -> under b + assert.equal((await jp.request('PUT', `/api/web/folders/${c}`, { json: { parentId: b } })).status, 200); + await pull(); + assert.equal((await client.folderById(c)).parent_id, b); + + // client: c -> top level, then under a again + await client.run('mv', c, 'root'); + await client.sync(); + await sleep(300); + assert.equal(await parentOf(c), '', 'client moved it to the top level'); + await client.run('mv', c, a); + await client.sync(); + await sleep(300); + assert.equal(await parentOf(c), a); + }); + + it('both refuse to nest a notebook under its own descendant', async () => { + const a = await jp.createFolder(`${PREFIX}-t4a`); + const b = await jp.createFolder(`${PREFIX}-t4b`, a); + await pull(); + await assert.rejects(client.run('mv', a, b), /Cannot move/i, 'Joplin refuses'); + const r = await jp.request('PUT', `/api/web/folders/${a}`, { json: { parentId: b } }); + assert.equal(r.status, 400, 'Joplock refuses too'); + assert.equal(await parentOf(a), ''); + }); + + it('renames keep the nesting and travel both ways', async () => { + const a = await jp.createFolder(`${PREFIX}-t5a`); + const b = await jp.createFolder(`${PREFIX}-t5b`, a); + await pull(); + assert.equal((await jp.request('PUT', `/api/web/folders/${b}`, { json: { title: `${PREFIX}-t5b-renamed` } })).status, 200); + await pull(); + const rb = await client.folderById(b); + assert.equal(rb.title, `${PREFIX}-t5b-renamed`); + assert.equal(rb.parent_id, a, 'still nested after a Joplock rename'); + + await client.run('ren', b, `${PREFIX}-t5b-again`); + await client.sync(); + await sleep(300); + const jb = (await jp.folders()).find(f => f.id === b); + assert.equal(jb.title, `${PREFIX}-t5b-again`); + assert.equal(jb.parentId, a); + }); + + it('deleting a middle notebook in Joplock promotes its children and notes; the client converges', async () => { + const a = await jp.createFolder(`${PREFIX}-t6a`); + const b = await jp.createFolder(`${PREFIX}-t6b`, a); + const c = await jp.createFolder(`${PREFIX}-t6c`, b); + const nb = await jp.createNote(`${PREFIX}-t6-note-b`, 'in the middle', b); + const nc = await jp.createNote(`${PREFIX}-t6-note-c`, 'in the leaf', c); + await pull(); + assert.equal((await jp.request('DELETE', `/api/web/folders/${b}`)).status, 204); + await pull(); + assert.equal(await client.folderById(b), undefined, 'the deleted notebook is gone from the client'); + assert.equal((await client.folderById(c)).parent_id, a, 'its child was promoted'); + assert.equal((await client.noteById(nb)).parent_id, a, 'its note moved to the parent'); + assert.equal((await client.noteById(nc)).parent_id, c, 'the child keeps its own note'); + }); + + it('Joplin 3 trashing a notebook with sub-notebooks removes the whole subtree from Joplock', async () => { + const a = `${PREFIX}-t7a`; + await client.run('mkbook', a); + await client.run('mkbook', '-p', a, `${PREFIX}-t7b`); + await client.run('use', `${PREFIX}-t7b`); + await client.run('mknote', `${PREFIX}-t7-note`); + await client.sync(); + const fa = await client.folder(a); + const fb = await client.folder(`${PREFIX}-t7b`); + await sleep(300); + assert.ok((await jp.folders()).some(f => f.id === fb.id), 'visible first'); + + await client.run('rmbook', '-f', a); + await client.sync(); + await sleep(400); + const ids = (await jp.folders()).map(f => f.id); + assert.ok(!ids.includes(fa.id) && !ids.includes(fb.id), 'neither level is listed any more'); + assert.ok(!(await jp.noteHeaders()).some(n => n.title === `${PREFIX}-t7-note` && !n.deletedTime), 'the note is not in a live notebook'); + }); +}); diff --git a/integration-tests/joplin-sharing.test.js b/integration-tests/joplin-sharing.test.js new file mode 100644 index 0000000..2d2d63c --- /dev/null +++ b/integration-tests/joplin-sharing.test.js @@ -0,0 +1,136 @@ +'use strict'; + +// Shared NESTED notebooks: owner works in Joplock, the recipient is a stock Joplin +// client. Joplin Server applies share changes on a 10 second schedule, so every +// step waits for it exactly like a real session would. +// +// What this guards: a stock client only follows share changes that come with a +// sync change event (an API write). Updating descendants with silent database +// writes used to leave a subtree that was moved OUT of a share sitting in the +// recipient's Joplin app. + +const { describe, it, before, after } = require('node:test'); +const assert = require('node:assert/strict'); +const { skipReason, sleep, Joplock, JoplinClient, ACCOUNT } = require('./joplin/harness'); + +const skip = skipReason(); +const PREFIX = `jc-shr-${Date.now().toString(36)}`; +const SHARE_TASK_MS = 13000; // the server task runs every 10s + +describe('Shared nested notebooks reach a stock Joplin client', { skip, concurrency: false }, () => { + let owner; let client; let shareId; + const ids = {}; + + before(async () => { + owner = await Joplock.login(process.env.PLAYWRIGHT_ADMIN_EMAIL || process.env.JOPLOCK_ADMIN_EMAIL, process.env.PLAYWRIGHT_ADMIN_PASSWORD || process.env.JOPLOCK_ADMIN_PASSWORD); + client = await new JoplinClient('share').init(); + }); + after(async () => { + if (client) client.close(); + if (owner) { + if (shareId) await owner.request('DELETE', `/api/web/shares/${shareId}`); + await owner.cleanup(PREFIX); + } + }); + + // what the recipient's Joplin app holds for this run + const view = async () => { + await sleep(SHARE_TASK_MS); + await client.sync(); + await sleep(400); + await client.sync(); + const folders = (await client.all('SELECT id, title, parent_id, share_id FROM folders WHERE title LIKE ?', [`${PREFIX}%`])); + const notes = (await client.all('SELECT id, title, parent_id FROM notes WHERE title LIKE ?', [`${PREFIX}%`])); + return { folders, notes, folderTitles: folders.map(f => f.title.replace(`${PREFIX}-`, '')).sort(), noteTitles: notes.map(n => n.title.replace(`${PREFIX}-`, '')).sort() }; + }; + + it('shares the whole subtree: notebooks keep their nesting and share id, notes come along', async () => { + ids.root = await owner.createFolder(`${PREFIX}-root`); + ids.sub = await owner.createFolder(`${PREFIX}-sub`, ids.root); + ids.leaf = await owner.createFolder(`${PREFIX}-leaf`, ids.sub); + await owner.createNote(`${PREFIX}-note-root`, 'r', ids.root); + ids.noteLeaf = await owner.createNote(`${PREFIX}-note-leaf`, 'l', ids.leaf); + const share = await owner.ok('POST', '/api/web/shares', { json: { notebookId: ids.root } }); + shareId = share.data.id; + await owner.ok('POST', `/api/web/shares/${shareId}/invites`, { json: { email: ACCOUNT.email, can_write: 1 } }); + + const v = await view(); + assert.deepEqual(v.folderTitles, ['leaf', 'root', 'sub']); + assert.deepEqual(v.noteTitles, ['note-leaf', 'note-root']); + const byId = Object.fromEntries(v.folders.map(f => [f.id, f])); + assert.equal(byId[ids.root].parent_id, ''); + assert.equal(byId[ids.sub].parent_id, ids.root); + assert.equal(byId[ids.leaf].parent_id, ids.sub); + for (const f of v.folders) assert.equal(f.share_id, shareId, `${f.title} carries the share id`); + }); + + it('a notebook (with sub-notebook and notes) moved INTO the share arrives; moved back OUT it disappears with all of it', async () => { + ids.priv = await owner.createFolder(`${PREFIX}-priv`); + ids.privKid = await owner.createFolder(`${PREFIX}-privkid`, ids.priv); + await owner.createNote(`${PREFIX}-note-priv`, 'p', ids.priv); + await owner.createNote(`${PREFIX}-note-kid`, 'k', ids.privKid); + + assert.equal((await owner.request('PUT', `/api/web/folders/${ids.priv}`, { json: { parentId: ids.sub } })).status, 200); + let v = await view(); + for (const t of ['priv', 'privkid']) assert.ok(v.folderTitles.includes(t), `${t} arrived`); + for (const t of ['note-priv', 'note-kid']) assert.ok(v.noteTitles.includes(t), `${t} arrived`); + + assert.equal((await owner.request('PUT', `/api/web/folders/${ids.priv}`, { json: { parentId: '' } })).status, 200); + v = await view(); + for (const t of ['priv', 'privkid']) assert.ok(!v.folderTitles.includes(t), `${t} is gone from the recipient's Joplin app`); + for (const t of ['note-priv', 'note-kid']) assert.ok(!v.noteTitles.includes(t), `${t} is gone from the recipient's Joplin app`); + assert.ok(v.folderTitles.includes('root') && v.folderTitles.includes('leaf'), 'the rest of the share is untouched'); + }); + + it('notebooks and notes created inside the share reach the recipient; edits made in the Joplin app survive an owner edit in Joplock', async () => { + ids.fresh = await owner.createFolder(`${PREFIX}-fresh`, ids.leaf); + await owner.createNote(`${PREFIX}-note-fresh`, 'f', ids.fresh); + let v = await view(); + assert.ok(v.folderTitles.includes('fresh') && v.noteTitles.includes('note-fresh')); + + // the recipient edits the owner's note in a nested shared notebook, setting fields Joplock does not model + const target = `${PREFIX}-note-leaf`; + await client.run('set', ids.noteLeaf, 'body', 'edited in the recipient joplin app'); + await client.run('set', ids.noteLeaf, 'author', 'Recipient Author'); + await client.sync(); + await sleep(SHARE_TASK_MS); + assert.equal((await owner.note(ids.noteLeaf)).body, 'edited in the recipient joplin app', "the owner sees the recipient's edit"); + + // the owner edits it in Joplock: the recipient's fields must survive + const r = await owner.updateNote(ids.noteLeaf, { title: target, body: 'edited again by the owner in joplock', parentId: ids.leaf }); + assert.equal(r.status, 200, r.text); + await view(); + const back = await client.noteById(ids.noteLeaf); + assert.equal(back.body, 'edited again by the owner in joplock'); + assert.equal(back.author, 'Recipient Author', "the recipient's author field survives an owner edit in Joplock"); + }); + + // KNOWN GAP (predates nested notebooks): Joplock decides who may see an item from its + // own `user_items` rows keyed by jop_id, while Joplin Server keys them by items.id. A note + // a collaborator creates in a Joplin app therefore never appears for the share OWNER in + // Joplock (it does appear in the owner's Joplin apps). Fixing it means changing the share + // access model together with Joplock's own removal paths, otherwise removed users would + // keep access. Tracked here as a todo so it stays visible without failing the suite. + it('notes a collaborator creates in the share are visible to the owner in Joplock', { todo: 'share access model: jop_id vs items.id user_items rows' }, async () => { + await client.run('use', `${PREFIX}-leaf`); + await client.run('mknote', `${PREFIX}-note-from-recipient`); + await client.sync(); + await sleep(SHARE_TASK_MS); + const theirs = (await owner.noteHeaders()).find(h => h.title === `${PREFIX}-note-from-recipient`); + const recipient = await Joplock.login(ACCOUNT.email, ACCOUNT.password); + await recipient.cleanup(`${PREFIX}-note-from-recipient`); + assert.ok(theirs, "the owner sees the collaborator's note in Joplock"); + }); + + it('stopping the share takes every level away from the recipient and leaves the owner able to edit', async () => { + const stopped = await owner.request('DELETE', `/api/web/shares/${shareId}`); + assert.ok(stopped.status < 300, stopped.text); + shareId = null; + const v = await view(); + for (const t of ['root', 'sub', 'leaf', 'fresh']) assert.ok(!v.folderTitles.includes(t), `${t} removed from the recipient`); + const ownerFolders = await owner.folders(); + for (const id of [ids.root, ids.sub, ids.leaf]) assert.equal(ownerFolders.find(f => f.id === id).shareId || '', '', 'owner side share id cleared'); + const rename = await owner.request('PUT', `/api/web/folders/${ids.sub}`, { json: { title: `${PREFIX}-sub-renamed` } }); + assert.equal(rename.status, 200, rename.text); + }); +}); diff --git a/integration-tests/joplin/harness.js b/integration-tests/joplin/harness.js index 392e128..a9e4e0e 100644 --- a/integration-tests/joplin/harness.js +++ b/integration-tests/joplin/harness.js @@ -100,6 +100,19 @@ class Joplock { if (r.status >= 300) throw new Error(`put raw item ${id}: HTTP ${r.status} ${r.text.slice(0, 200)}`); } + // Upload an attachment the way the browser UI does (POST /fragments/upload). + async uploadResource(filename, mime, data) { + const boundary = `----jcupload${Date.now()}`; + const head = Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${filename}"\r\nContent-Type: ${mime}\r\n\r\n`); + const tail = Buffer.from(`\r\n--${boundary}--\r\n`); + const r = await this.request('POST', '/fragments/upload', { + headers: { 'Content-Type': `multipart/form-data; boundary=${boundary}` }, + raw: Buffer.concat([head, data, tail]), + }); + if (r.status !== 200) throw new Error(`upload failed: ${r.status} ${r.text.slice(0, 200)}`); + return r.data.resourceId; + } + // Remove everything whose title starts with the given prefix (best effort). async cleanup(prefix) { try { diff --git a/package.json b/package.json index ca8350a..07cda72 100644 --- a/package.json +++ b/package.json @@ -1,43 +1,43 @@ { - "name": "joplock", - "private": true, - "version": "0.1.0", - "description": "Thin-client sidecar web UI for stock Joplin Server", - "main": "server.js", - "scripts": { - "start": "node server.js", - "test": "node --test tests/*.test.js", - "build:cm": "cd cm-build && npm install && npm run build", - "build:hljs": "cd hljs-build && npm install && npm run build", - "build:prism": "cd prism-build && npm install && npm run build", - "test:ui": "playwright test", - "test:ui:headed": "playwright test --headed", - "generate:pwa-assets": "node ./scripts/generatePwaAssets.mjs", - "docker:build": "docker build -t joplock .", - "docker:up": "docker compose up -d", - "docker:down": "docker compose down", - "docker:up:build": "docker compose -f docker-compose.example-full-build.yml up -d --build", - "docker:down:build": "docker compose -f docker-compose.example-full-build.yml down", - "docker:up:dev": "docker compose -f docker-compose.dev.yml up -d --build", - "docker:down:dev": "docker compose -f docker-compose.dev.yml down", - "docker:up:full": "docker compose -f docker-compose.example-full.yml up -d", - "docker:down:full": "docker compose -f docker-compose.example-full.yml down", - "setup:joplin-cli": "bash scripts/setup-joplin-cli.sh", - "test:joplin": "node --test --test-concurrency=1 integration-tests/*.test.js" - }, - "dependencies": { - "@adobe/css-tools": "4.4.4", - "@mixmark-io/domino": "2.2.0", - "bcryptjs": "^3.0.3", - "html-entities": "1.4.0", - "markdown-it": "^14.1.1", - "pg": "8.16.3", - "qr-image": "^3.2.0", - "tinymce": "^8.6.0" - }, - "devDependencies": { - "@playwright/test": "^1.59.1", - "jsdom": "26.1.0", - "sharp": "^0.34.5" - } + "name": "joplock", + "private": true, + "version": "0.1.0", + "description": "Thin-client sidecar web UI for stock Joplin Server", + "main": "server.js", + "scripts": { + "start": "node server.js", + "test": "node --test tests/*.test.js", + "build:cm": "cd cm-build && npm install && npm run build", + "build:hljs": "cd hljs-build && npm install && npm run build", + "build:prism": "cd prism-build && npm install && npm run build", + "test:ui": "playwright test", + "test:ui:headed": "playwright test --headed", + "generate:pwa-assets": "node ./scripts/generatePwaAssets.mjs", + "docker:build": "docker build -t joplock .", + "docker:up": "docker compose up -d", + "docker:down": "docker compose down", + "docker:up:build": "docker compose -f docker-compose.example-full-build.yml up -d --build", + "docker:down:build": "docker compose -f docker-compose.example-full-build.yml down", + "docker:up:dev": "docker compose -f docker-compose.dev.yml up -d --build", + "docker:down:dev": "docker compose -f docker-compose.dev.yml down", + "docker:up:full": "docker compose -f docker-compose.example-full.yml up -d", + "docker:down:full": "docker compose -f docker-compose.example-full.yml down", + "setup:joplin-cli": "bash scripts/setup-joplin-cli.sh", + "test:joplin": "node --test --test-concurrency=1 integration-tests/*.test.js" + }, + "dependencies": { + "@adobe/css-tools": "4.4.4", + "@mixmark-io/domino": "2.2.0", + "bcryptjs": "^3.0.3", + "html-entities": "1.4.0", + "markdown-it": "^14.1.1", + "pg": "8.16.3", + "qr-image": "^3.2.0", + "tinymce": "^8.6.0" + }, + "devDependencies": { + "@playwright/test": "^1.59.1", + "jsdom": "26.1.0", + "sharp": "^0.34.5" + } } diff --git a/playwright-tests/joplin-items.spec.js b/playwright-tests/joplin-items.spec.js new file mode 100644 index 0000000..bf25873 --- /dev/null +++ b/playwright-tests/joplin-items.spec.js @@ -0,0 +1,186 @@ +'use strict'; + +// Joplin items Joplock must treat carefully, in the real UI: +// - HTML notes (markup_language 2): shown rendered and read-only, never rewritten +// - end-to-end encrypted notes: locked placeholder, no editor at all +// The items are uploaded the way a stock Joplin client does (X-API-AUTH, sync API). + +const crypto = require('node:crypto'); +const { test, expect } = require('@playwright/test'); +const { acceptDialogs, login, slug, teardownTestData } = require('./helpers'); + +const id32 = () => crypto.randomBytes(16).toString('hex'); +const TS = '2026-01-02T03:04:05.678Z'; + +const sessionId = async page => (await page.context().cookies()).find(c => c.name === 'sessionId').value; + +// Raw sync-API calls use the standalone `request` fixture (no browser cookies) with only +// X-API-AUTH, like a stock Joplin client. page.request would also send Joplock's +// session cookie, which is not what a Joplin app does. +const putRaw = async (page, request, id, text) => { + const boundary = `----pw${Date.now()}`; + const body = `--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="item.md"\r\nContent-Type: text/markdown\r\n\r\n${text}\r\n--${boundary}--\r\n`; + const res = await request.put(`/joplin/api/items/root:/${id}.md:/content`, { + headers: { 'X-API-AUTH': await sessionId(page), 'Content-Type': `multipart/form-data; boundary=${boundary}` }, + data: Buffer.from(body), + }); + expect(res.status(), await res.text()).toBeLessThan(300); +}; +const getRaw = async (page, request, id) => (await request.get(`/joplin/api/items/root:/${id}.md:/content`, { headers: { 'X-API-AUTH': await sessionId(page) } })).text(); +const delRaw = async (page, request, id) => request.delete(`/joplin/api/items/root:/${id}.md:`, { headers: { 'X-API-AUTH': await sessionId(page) } }).catch(() => null); + +const api = (page, method, url, body) => page.evaluate(async ([m, u, b]) => { + const res = await fetch(u, { method: m, credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: b === undefined ? undefined : JSON.stringify(b) }); + const text = await res.text(); + let data = null; try { data = JSON.parse(text); } catch { /* html */ } + return { status: res.status, data, text }; +}, [method, url, body]); + +const noteProps = (id, parent, extra = []) => [ + `id: ${id}`, `parent_id: ${parent}`, `created_time: ${TS}`, `updated_time: ${TS}`, + 'is_conflict: 0', 'latitude: 0.00000000', 'longitude: 0.00000000', 'altitude: 0.0000', 'author: ', 'source_url: ', + 'is_todo: 0', 'todo_due: 0', 'todo_completed: 0', 'source: joplin-desktop', 'source_application: net.cozic.joplin-desktop', + 'application_data: ', 'order: 1767323045678', `user_created_time: ${TS}`, `user_updated_time: ${TS}`, + 'encryption_cipher_text: ', 'encryption_applied: 0', ...extra, 'is_shared: 0', 'share_id: ', + 'conflict_original_id: ', 'master_key_id: ', 'user_data: ', 'deleted_time: 0', 'type_: 1', +].join('\n'); + +const openNote = async (page, folderTitle, noteTitle) => { + const folder = page.locator(`#nav-panel .nav-folder[data-folder-title="${folderTitle}"]`).first(); + await expect(folder).toBeVisible({ timeout: 15000 }); + await folder.locator(':scope > .nav-folder-row .nav-folder-toggle').click(); + await folder.locator('.notelist-item', { hasText: noteTitle }).first().click(); +}; + +test.describe('HTML notes', () => { + test('desktop: rendered, sanitized, read-only, and never rewritten', async ({ page, request }, testInfo) => { + test.skip(testInfo.project.name !== 'desktop'); + acceptDialogs(page); + const folder = slug('pw-joplinitems'); + const noteId = id32(); + const title = 'Clipped web page'; + const html = '

Clipped heading

Plain bold text

bad'; + await login(page); + try { + const folderId = (await api(page, 'POST', '/api/web/folders', { title: folder, parentId: '' })).data.item.id; + await putRaw(page, request, noteId, `${title}\n\n${html}\n\n${noteProps(noteId, folderId, ['markup_language: 2'])}`); + const before = await getRaw(page, request, noteId); + await page.reload(); + await openNote(page, folder, title); + + const form = page.locator('#editor-panel #note-editor-form[data-html-note="1"]'); + await expect(form).toBeVisible({ timeout: 15000 }); + await expect(page.locator('#editor-panel .html-note-banner')).toContainText('HTML note'); + await expect(page.locator('#editor-panel #markdown-toggle')).toBeHidden(); + await expect(page.locator('#editor-panel #preview-toggle')).toBeHidden(); + await expect(page.locator('#editor-panel #editor-toolbar')).toBeHidden(); + await expect(page.locator('#editor-panel .editor-title')).toHaveAttribute('contenteditable', 'false'); + + // rendered in TinyMCE, with the hostile parts gone + const frame = page.frameLocator('iframe.tox-edit-area__iframe'); + await expect(frame.locator('body')).toContainText('Clipped heading', { timeout: 15000 }); + await expect(frame.locator('h2')).toHaveText('Clipped heading'); + await expect(frame.locator('script, [onerror]')).toHaveCount(0); + await expect(frame.locator('a[href^="javascript"]')).toHaveCount(0); + expect(await page.evaluate(() => window.__pwx)).toBeUndefined(); + + // read-only: typing does nothing and nothing gets saved + await frame.locator('body').click({ force: true }); + await page.keyboard.type('SHOULD NOT APPEAR'); + await expect(frame.locator('body')).not.toContainText('SHOULD NOT APPEAR'); + await page.keyboard.press('Control+b'); + await page.waitForTimeout(3500); // longer than the 2s autosave debounce + expect(await getRaw(page, request, noteId)).toBe(before); + + // the stored HTML is still HTML, byte for byte + expect(before).toMatch(/markup_language: 2/); + expect(await getRaw(page, request, noteId)).toContain(''); + } finally { + await delRaw(page, request, noteId); + await teardownTestData(page, { folders: [folder] }); + } + }); + + test('mobile: the HTML note opens rendered with no mode switch', async ({ page, request }, testInfo) => { + test.skip(testInfo.project.name !== 'mobile'); + acceptDialogs(page); + const folder = slug('pw-joplinitems'); + const noteId = id32(); + const title = 'Clipped mobile page'; + await login(page); + try { + const folderId = (await api(page, 'POST', '/api/web/folders', { title: folder, parentId: '' })).data.item.id; + await putRaw(page, request, noteId, `${title}\n\n

Mobile heading

body

\n\n${noteProps(noteId, folderId, ['markup_language: 2'])}`); + await page.reload(); + await expect(page.locator('#mobile-app[aria-hidden="false"]')).toBeVisible(); + await page.locator('#mobile-folders-body .mobile-folder-row', { hasText: folder }).first().click(); + await page.locator('#mobile-notes-body .mobile-note-row', { hasText: title }).first().click(); + await expect(page.locator('#mobile-editor-screen.mobile-screen-active')).toBeVisible(); + await expect(page.locator('#mobile-editor-body .html-note-banner')).toBeVisible({ timeout: 15000 }); + await expect(page.locator('#mobile-md-toggle')).toBeHidden(); + await expect(page.locator('#mobile-preview-toggle')).toBeHidden(); + await expect(page.frameLocator('iframe.tox-edit-area__iframe').locator('h3')).toHaveText('Mobile heading', { timeout: 15000 }); + } finally { + await delRaw(page, request, noteId); + await teardownTestData(page, { folders: [folder] }); + } + }); + + test('switching from an HTML note to a markdown note restores the normal editor', async ({ page, request }, testInfo) => { + test.skip(testInfo.project.name !== 'desktop'); + acceptDialogs(page); + const folder = slug('pw-joplinitems'); + const htmlId = id32(); + await login(page); + try { + const folderId = (await api(page, 'POST', '/api/web/folders', { title: folder, parentId: '' })).data.item.id; + await putRaw(page, request, htmlId, `An html note\n\n

html

\n\n${noteProps(htmlId, folderId, ['markup_language: 2'])}`); + await api(page, 'POST', '/api/web/notes', { title: 'A markdown note', body: '# md body', parentId: folderId }); + await page.reload(); + await openNote(page, folder, 'An html note'); + await expect(page.locator('#editor-panel #note-editor-form[data-html-note="1"]')).toBeVisible({ timeout: 15000 }); + await page.locator('#nav-panel .notelist-item', { hasText: 'A markdown note' }).first().click(); + const md = page.locator('#editor-panel #note-editor-form:not([data-html-note])'); + await expect(md).toBeVisible({ timeout: 15000 }); + await expect(page.locator('#editor-panel .html-note-banner')).toHaveCount(0); + await expect(page.locator('#editor-panel #markdown-toggle')).toBeVisible(); + await expect(page.locator('#editor-panel .editor-title')).toHaveAttribute('contenteditable', 'true'); + } finally { + await delRaw(page, request, htmlId); + await teardownTestData(page, { folders: [folder] }); + } + }); +}); + +test.describe('Joplin end-to-end encrypted notes', () => { + test('desktop: locked placeholder in the list and in the editor, nothing to edit', async ({ page, request }, testInfo) => { + test.skip(testInfo.project.name !== 'desktop'); + acceptDialogs(page); + const folder = slug('pw-joplinitems'); + const noteId = id32(); + await login(page); + try { + const folderId = (await api(page, 'POST', '/api/web/folders', { title: folder, parentId: '' })).data.item.id; + await putRaw(page, request, noteId, [`id: ${noteId}`, `parent_id: ${folderId}`, 'encryption_cipher_text: JED01000007fakeciphertextfakeciphertext', 'encryption_applied: 1', 'type_: 1'].join('\n')); + const before = await getRaw(page, request, noteId); + await page.reload(); + const f = page.locator(`#nav-panel .nav-folder[data-folder-title="${folder}"]`).first(); + await expect(f).toBeVisible({ timeout: 15000 }); + await f.locator(':scope > .nav-folder-row .nav-folder-toggle').click(); + const item = f.locator('.notelist-item[data-e2ee="1"]').first(); + await expect(item).toBeVisible(); + await expect(item).toContainText('Encrypted note'); + await expect(item.locator('.note-lock-icon')).toBeVisible(); + await item.click(); + await expect(page.locator('#editor-panel #editor-e2ee')).toBeVisible({ timeout: 15000 }); + await expect(page.locator('#editor-panel')).toContainText('end-to-end encrypted by Joplin'); + await expect(page.locator('#editor-panel #note-editor-form')).toHaveCount(0); + await expect(page.locator('#editor-panel textarea')).toHaveCount(0); + await page.waitForTimeout(2500); + expect(await getRaw(page, request, noteId)).toBe(before); + } finally { + await delRaw(page, request, noteId); + await teardownTestData(page, { folders: [folder] }); + } + }); +}); diff --git a/public/service-worker.js b/public/service-worker.js index 175708e..d3ebbe3 100644 --- a/public/service-worker.js +++ b/public/service-worker.js @@ -1,4 +1,4 @@ -const CACHE_NAME = 'joplock-shell-v80-20261008nestedfolders'; +const CACHE_NAME = 'joplock-shell-v81-20261009joplinfidelity'; const STATIC_ASSETS = ['/styles.css', '/htmx.min.js', '/turndown.min.js', '/codemirror.min.js', '/cm-extras.js', '/prism.min.js', '/hljs.min.js', '/manifest.webmanifest', '/icon.svg', '/icon-192.png', '/icon-512.png', '/maskable-icon-192.png', '/maskable-icon-512.png', '/apple-touch-icon.png', '/fonts/CascadiaMono-Regular.woff2', '/fonts/CascadiaMono-Bold.woff2']; self.addEventListener('install', event => { diff --git a/tests/folderOps.test.js b/tests/folderOps.test.js index cbaf52d..367c01f 100644 --- a/tests/folderOps.test.js +++ b/tests/folderOps.test.js @@ -121,14 +121,14 @@ test('moving a notebook INTO a share stamps the subtree and grants recipients', const r = await w.ops.moveFolder({ user: USER, folderId: 'pkid', targetParentId: 'sub', requestContext: ctx }); assert.equal(r.shareChanged, true); assert.deepEqual(w.calls[0], ['updateFolder', 'pkid', { parentId: 'sub', shareId: 'S1', isShared: true }]); - assert.deepEqual(w.sync, [['subtree', { ownerId: 'u1', folderId: 'pkid', shareId: 'S1', previousShareId: '' }]]); + assert.deepEqual(w.sync.map(x => [x[0], x[1].ownerId, x[1].folderId, x[1].shareId, x[1].previousShareId, x[1].sessionId]), [['subtree', 'u1', 'pkid', 'S1', '', 's1']], 'the session is passed so descendants can be re-saved through the API'); }); test('moving a notebook OUT of a share clears the subtree and revokes recipients', async () => { const w = world({ folders: sharedTree() }); await w.ops.moveFolder({ user: USER, folderId: 'sub', targetParentId: 'private', requestContext: ctx }); assert.deepEqual(w.calls[0], ['updateFolder', 'sub', { parentId: 'private', shareId: '', isShared: false }]); - assert.deepEqual(w.sync, [['subtree', { ownerId: 'u1', folderId: 'sub', shareId: '', previousShareId: 'S1' }]]); + assert.deepEqual(w.sync.map(x => [x[0], x[1].ownerId, x[1].folderId, x[1].shareId, x[1].previousShareId, x[1].sessionId]), [['subtree', 'u1', 'sub', '', 'S1', 's1']]); }); test('moving within the same share changes the parent only (no share churn)', async () => { diff --git a/tests/itemService.test.js b/tests/itemService.test.js index e8e4994..39cdf21 100644 --- a/tests/itemService.test.js +++ b/tests/itemService.test.js @@ -173,3 +173,41 @@ test('item reads select the encryption column so E2EE items can be recognised', await svc.noteHeadersByFolder('u', 'f1'); for (const sql of sqls) assert.match(sql, /jop_encryption_applied/); }); + +// ── share id: the column is authoritative ── +test('share id comes from the jop_share_id column; the JSON copy is only a fallback (it vanishes after the first API save)', () => { + const afterApiSave = mapNote(row({ title: 'T', body: 'b' /* no share_id: the server stripped it */ }, { jop_share_id: 'SHARE1' })); + assert.equal(afterApiSave.shareId, 'SHARE1'); + assert.equal(afterApiSave.isShared, true); + const dbStamped = mapNote(row({ title: 'T', share_id: 'SHARE2', is_shared: 1 }, { jop_share_id: '' })); + assert.equal(dbStamped.shareId, 'SHARE2', 'JSON fallback for items stamped directly in the database'); + const both = mapNote(row({ title: 'T', share_id: 'STALE' }, { jop_share_id: 'CURRENT' })); + assert.equal(both.shareId, 'CURRENT', 'the column wins over a stale JSON value'); + const none = mapNote(row({ title: 'T' })); + assert.equal(none.shareId, ''); + assert.equal(none.isShared, false); + assert.equal(mapFolder(row({ title: 'F' }, { jop_share_id: 'S' })).shareId, 'S'); + assert.equal(mapHeader({ jop_id: 'n', jop_parent_id: 'f', title: 't', jop_share_id: 'S' }).shareId, 'S'); +}); + +test('an edit round-trip keeps a shared note shared (the regression: 2nd Joplock edit used to write share_id empty)', () => { + const { serializeNote } = require('../app/items/itemWriteService'); + const shared = mapNote(row({ title: 'T', body: 'b' }, { jop_share_id: 'SHARE1' })); + const text = serializeNote({ id: shared.id, title: shared.title, body: 'edited', parentId: shared.parentId, createdTime: 1, isShared: shared.isShared, shareId: shared.shareId, fields: shared.fields }).body; + assert.match(text, /^share_id: SHARE1$/m); + assert.match(text, /^is_shared: 1$/m); +}); + +test('item reads select the share column', async () => { + const { createItemService } = require('../app/items/itemService'); + const sqls = []; + const svc = createItemService({ query: async sql => { sqls.push(sql); return { rows: [] }; } }); + await svc.foldersByUserId('u'); + await svc.folderByUserIdAndJopId('u', 'f'); + await svc.notesByUserId('u'); + await svc.noteByUserIdAndJopId('u', 'n'); + await svc.noteHeadersByUserId('u'); + await svc.noteHeadersByFolder('u', 'f1'); + await svc.searchNotes('u', 'x'); + for (const sql of sqls) assert.match(sql, /jop_share_id/, sql.slice(0, 90)); +}); diff --git a/tests/joplinItem.test.js b/tests/joplinItem.test.js index db03369..cc64b5b 100644 --- a/tests/joplinItem.test.js +++ b/tests/joplinItem.test.js @@ -150,6 +150,7 @@ test('serializeResource writes the Joplin 3.x OCR fields', () => { const parsed = unserialize(serializeResource({ id: 'r'.repeat(32), title: 'a.png', mime: 'image/png', filename: 'a.png', fileExtension: 'png', size: 10 }).body); for (const key of ['ocr_text', 'ocr_details', 'ocr_status', 'ocr_error', 'ocr_driver_id']) assert.ok(key in parsed, key); assert.equal(parsed.ocr_status, '0'); + assert.equal(parsed.ocr_driver_id, '1', 'Joplin default (PrintedText), not the legacy 0'); assert.equal(parsed.mime, 'image/png'); assert.equal(parsed.type_, '4'); }); diff --git a/tests/shareSync.test.js b/tests/shareSync.test.js new file mode 100644 index 0000000..f96907f --- /dev/null +++ b/tests/shareSync.test.js @@ -0,0 +1,81 @@ +// Share changes on a notebook subtree must reach stock Joplin clients, which only +// react to change events (i.e. API writes), never to silent database updates. +const test = require('node:test'); +const assert = require('node:assert/strict'); +const { createShareSync } = require('../app/routes/shares'); + +const world = ({ folders, notes, e2ee = [] }) => { + const apiWrites = []; + const sql = []; + const byFolder = Object.fromEntries(folders.map(f => [f.id, { id: f.id, parentId: f.parentId || '', shareId: f.shareId || '', e2ee: e2ee.includes(f.id) }])); + const byNote = Object.fromEntries(notes.map(n => [n.id, { id: n.id, parentId: n.parentId, shareId: n.shareId || '', e2ee: e2ee.includes(n.id) }])); + const itemService = { + foldersByUserId: async () => Object.values(byFolder), + folderByUserIdAndJopId: async (u, id) => byFolder[id] || null, + noteByUserIdAndJopId: async (u, id) => byNote[id] || null, + }; + const itemWriteService = { + updateFolder: async (sid, existing, fields) => { if (existing.e2ee) throw Object.assign(new Error('e2ee'), { statusCode: 403 }); apiWrites.push(['folder', existing.id, fields]); }, + updateNote: async (sid, existing, fields) => { if (existing.e2ee) throw Object.assign(new Error('e2ee'), { statusCode: 403 }); apiWrites.push(['note', existing.id, fields]); }, + }; + const database = { + query: async (text, params) => { + sql.push(text); + if (/SELECT jop_id FROM items WHERE owner_id/.test(text)) return { rows: notes.filter(n => params[1].includes(n.parentId)).map(n => ({ jop_id: n.id })) }; + return { rows: [] }; + }, + }; + return { sync: createShareSync({ itemService, itemWriteService, database }), apiWrites, sql }; +}; + +const TREE = [{ id: 'root' }, { id: 'sub', parentId: 'root' }, { id: 'leaf', parentId: 'sub' }, { id: 'other' }]; +const NOTES = [{ id: 'n1', parentId: 'root' }, { id: 'n2', parentId: 'leaf' }, { id: 'n3', parentId: 'other' }]; + +test('moving a subtree INTO a share re-saves every notebook and note below it through the API', async () => { + const w = world({ folders: TREE, notes: NOTES }); + await w.sync.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: 'S1', previousShareId: '', sessionId: 'sess', requestContext: {} }); + assert.deepEqual(w.apiWrites.map(x => [x[0], x[1]]).sort(), [['folder', 'leaf'], ['folder', 'root'], ['folder', 'sub'], ['note', 'n1'], ['note', 'n2']]); + for (const [, , fields] of w.apiWrites) assert.deepEqual(fields, { shareId: 'S1', isShared: true }); + assert.ok(!w.apiWrites.some(x => x[1] === 'other' || x[1] === 'n3'), 'unrelated items are not touched'); +}); + +test('moving a subtree OUT of a share clears the share id through the API too (this is what makes Joplin clients drop it)', async () => { + const folders = TREE.map(f => (f.id === 'other' ? f : { ...f, shareId: 'S1' })); + const notes = NOTES.map(n => (n.id === 'n3' ? n : { ...n, shareId: 'S1' })); + const w = world({ folders, notes }); + await w.sync.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: '', previousShareId: 'S1', sessionId: 'sess', requestContext: {} }); + assert.equal(w.apiWrites.length, 5); + for (const [, , fields] of w.apiWrites) assert.deepEqual(fields, { shareId: '', isShared: false }); + assert.ok(w.sql.some(t => /DELETE FROM user_items/.test(t)), "Joplock's own recipient rows are revoked as well"); +}); + +test('items already in the target share are not rewritten', async () => { + const w = world({ folders: [{ id: 'root', shareId: 'S1' }, { id: 'sub', parentId: 'root' }], notes: [{ id: 'n1', parentId: 'root', shareId: 'S1' }] }); + await w.sync.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: 'S1', sessionId: 'sess', requestContext: {} }); + assert.deepEqual(w.apiWrites.map(x => x[1]), ['sub']); +}); + +test('end-to-end encrypted items cannot be re-saved: they are skipped by the API pass and swept by the database', async () => { + const w = world({ folders: [{ id: 'root' }], notes: [{ id: 'n1', parentId: 'root' }, { id: 'enc', parentId: 'root' }], e2ee: ['enc'] }); + await w.sync.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: 'S1', sessionId: 'sess', requestContext: {} }); + assert.ok(!w.apiWrites.some(x => x[1] === 'enc')); + assert.ok(w.sql.some(t => /UPDATE items SET/.test(t)), 'database sweep still runs'); +}); + +test('without a session the database sweep still runs (no API available)', async () => { + const w = world({ folders: [{ id: 'root' }], notes: [{ id: 'n1', parentId: 'root' }] }); + await w.sync.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: 'S1' }); + assert.equal(w.apiWrites.length, 0); + assert.ok(w.sql.some(t => /UPDATE items SET/.test(t))); +}); + +test('an API failure is reported, but only after the database sweep and recipient rows are done', async () => { + const w = world({ folders: [{ id: 'root' }], notes: [{ id: 'n1', parentId: 'root' }] }); + const failing = createShareSync({ + itemService: { foldersByUserId: async () => [{ id: 'root', parentId: '' }], folderByUserIdAndJopId: async () => ({ id: 'root', shareId: '' }), noteByUserIdAndJopId: async () => null }, + itemWriteService: { updateFolder: async () => { throw new Error('upstream down'); } }, + database: { query: async (t, p) => { w.sql.push(t); return { rows: [] }; } }, + }); + await assert.rejects(failing.setSubtreeShare({ ownerId: 'u', folderId: 'root', shareId: 'S1', sessionId: 's', requestContext: {} }), /upstream down/); + assert.ok(w.sql.some(t => /UPDATE items SET/.test(t)), 'the sweep happened before the error surfaced'); +}); diff --git a/tests/sharesBinaryItems.test.js b/tests/sharesBinaryItems.test.js index 56571bd..6076b88 100644 --- a/tests/sharesBinaryItems.test.js +++ b/tests/sharesBinaryItems.test.js @@ -29,6 +29,8 @@ test('shareIdOf() only decodes notes and folders and honours a table alias', () const m = /const shareIdOf = \(alias = ''\) => `([^`]+)`;/.exec(src); assert.ok(m, 'shareIdOf helper must exist'); assert.match(m[1], /CASE WHEN \$\{alias\}jop_type IN \(1, 2\) THEN convert_from\(\$\{alias\}content, 'UTF8'\)::json->>'share_id' END/); + // the authoritative column comes first; the JSON decode stays behind the type guard + assert.match(m[1], /^\(COALESCE\(NULLIF\(\$\{alias\}jop_share_id, ''\), CASE WHEN/); }); test('every share fan-out / revoke query goes through shareIdOf()', () => {