* Add per-project language setting (#754)
An optional BCP-47 language on ProjectData, picked from a searchable
list of all platform locales in project settings. New projects default
to the device locale; the Alice example project is en-US.
Spell check is gated per project: when the project language does not
leniently match the dictionary locale, the dictionary is withheld
(ProjectSpellCheckRepository) and project settings explain why.
The public story page emits <html lang> and JSON-LD inLanguage from the
declared language, and EPUB export prefers it over the device locale.
The hasher contributes zero bytes when unset so existing sync hashes
stay stable.
* Fix review findings in the project-language feature
createProject now only seeds the default language for genuinely new
projects (seedDefaultLanguage), so account sync materializes server
projects with the never-synced baseline intact, and the seed is
language-only so it cannot gate spell check against a same-language
dictionary. The hasher's language block gets a -1 marker plus length
prefix so it can never collide with a tags block, and the initial
write goes through the shared saveStoredProjectData path.
The Locale type now retains the script subtag, keeping zh-Hans/zh-Hant
style locales distinct in the picker. The picker's clear row is pinned
above the list so it survives an empty search, watchSpellCheckAllowed
delivers on the main dispatcher, and the public story page hashes the
stored project-data hash into its validator instead of parsing the
blob per request, applying the language override after withDefaults so
chrome links keep the viewer's locale.
* Enforce single-owner persisted formats
The tags write in PromoteIdeaUseCase rewrote project_data.toml from
scratch, erasing the language seed createProject had just written: the
exact hazard of a second inline writer. It now read-modify-writes
through the datasource's scope-less helpers, and ProjectsListComponent's
hand-rolled reader delegates to a new blocking readStoredProjectData.
The rule is written down (ARCHITECTURE.md hard constraint 7, CLAUDE.md)
and enforced by PersistedFormatOwnershipTest, which fails the build when
raw TOML I/O appears outside a Datasource file. Migrators are exempt by
role; the two remaining legacy offenders are allowlisted as a burn-down
that can only shrink.
* Burn down the last raw TOML I/O outside datasources
ProjectStatisticsCacheReader now delegates to a scope-less
readProjectStatistics helper in StatisticsDatasource, and the example
project's fabricated activity log goes through writeDeviceLog in
WritingActivityDatasource, which also becomes the single owner of the
.activity path convention.
With no offenders left, PersistedFormatOwnershipTest drops its
burn-down allowlist entirely: only Datasource files and migrators may
touch persisted TOML formats from here on.
* Pass seedDefaultLanguage in the Android instrumented-test harness
* Pass seedDefaultLanguage in the round-trip sync HeadlessClient
The class is backed by embedded Postgres, not SQLite, and the old name gave
no hint that the instance is a handle onto one process-wide database shared
by every test in the JVM. Document that its truncate is the only isolation
boundary, so anything that can write on its own schedule must be shut down
before a test ends.
* Serialize access to the shared test FakeFileSystem
The fake is shared by the test thread, the server's Jetty threads and the
client's dispatcher threads, and it is not thread-safe: it tracks open files
in a plain ArrayList. An autosave write on the client's dispatcher thread
while a sync read on the JUnit thread iterated that list threw
ConcurrentModificationException, failing the sync and with it
StaleAllocatorAfterSyncTest.
Wrap it in a SynchronizedFileSystem that locks every call, including the
returned Source/Sink/FileHandle — the fake mutates its open-file list when
those close, not only when they open.
* Bind e2e test server to an ephemeral port to fix flakiness
EndToEndTest bound every test's server to a fixed port (54321). Because
the whole suite shares one JVM and boots/tears down a server per test, a
stopped server's socket lingering in shutdown could collide with the next
test's bind, intermittently failing an unrelated test (e.g. ReviewPageTest's
locale-fallback case) with a non-200.
Bind port 0 so the OS assigns a free port, then read it back via
resolvedConnectors() so the client targets the right port.
* Expose resolved port to integrationTests after ephemeral-port switch
RoundTripTestBase (and its doc/comment neighbors) referenced the removed
TEST_PORT constant to build the client's server URL. Expose the OS-assigned
port as a protected read-only serverPort on EndToEndTest and point the client
settings at it. Refreshed stale 54321 comments; integrationTests keeps
maxParallelForks=1 for shared JVM-global state, not the port.
Quick-capture story ideas as tagged markdown blobs in a new Project
Selection tab, stored one file per idea in .ideas/ and promotable into
a project. Offline-first; syncs as a phase inside the account sync
session (shape-agnostic server storage, hash-baseline conflicts,
tombstone/outbox deletion, ideasStateHash skip for unchanged sets).
Unifies idea + project tag suggestions behind AccountTagService.
A never-synced project (null lastSyncedHash) with default project_data
was treated as a local edit and pushed, colliding with the server's
real settings. Baseline a null lastSyncedHash against the default-data
hash so the server copy is adopted instead, matching isProjectDataDirty.
* Heal server scenes with null timestamps instead of re-downloading
A scene uploaded before scenes tracked created/lastEdited arrives from the
server with null timestamps. createScene stamps locally-meaningful values and
the old merge kept them, but those fields are hashed, so the local copy never
matched the server's null-timestamp hash — the scene re-downloaded on every
sync, forever.
On download, backfill the null timestamps with the project's creation time,
then at the download chokepoint detect that the stored copy now hashes
differently from the server's and upload the enriched copy (baselined on the
server's just-recorded hash, so a lone client heals conflict-free). This
generalizes: any future field that round-trips lossily self-heals instead of
looping.
Adds an integration test that watches the wire and asserts a second sync of
unchanged server entities pulls nothing, plus a reusable HttpClient wire tap.
* Add server-originated resync stability matrix
Sweeps every entity type across the field combinations most likely to
round-trip lossily (null/empty optional fields, populated fields, images,
archive state, scene groups, draft-to-scene references) and asserts that a
second sync of unchanged server-originated entities is silent over the wire —
no entity pulled, nothing pushed back. This is the invariant the null-timestamp
re-download bug violated.
Extends the wire tap to also track uploads.
* Add upload-direction resync stability matrix
Mirror of the server-originated matrix for the upload direction: a client
creates entities across each type's hashed fields, syncs them up, and a second
sync must be silent over the wire (nothing re-uploaded, nothing pulled). Closes
the symmetric gap so divergence is caught whether an entity originates on the
client or the server.
* Build out sync test matrix: oracle helpers, two-device harness, mixed fuzzer
Adds the convergence + stability oracle (assertConverged / assertResyncSilent
on top of the wire tap) and organizes the e2e sync tests into three regimes:
first-time, no-change, and mixed.
- HeadlessClient/secondDeviceFor: a second real device can join an existing
server project with its own local dir, enabling true two-actor tests.
- TwoDeviceSyncTest: a second device adopts a project; independent edits on two
devices converge.
- MixedSyncFuzzTest: seeded property test driving random create/edit/delete/
rename across every entity type, asserting convergence + a silent resync.
- README documents the regimes, the model-free oracle, and the coverage map.
* Refresh IdAllocator when a sync completes
findNextId ran only at project-open and sync-start, so a sync that downloaded
entities with higher ids left the allocator's next-id snapshot stale. A create
before the next sync then minted an id that collided with a just-downloaded
entity. Re-derive the allocator in FinalizeSyncOperation after entity transfer.
Adds StaleAllocatorAfterSyncTest (a device creates right after adopting a
project) and drops the now-unnecessary initializeSceneEditor() workaround from
the two-device merge test.
On a legacy server.json carrying inline tokens, GlobalSettingsStore.init
triggered loadServerSettings, which migrated the tokens with a guarded write
through ContainedFileSystem. The containment check resolved managedStorageRoots
by re-entering the still-constructing store, which Koin rebuilt, looping forever
and hanging startup on every platform with a pre-v3.5.0 synced account.
Make store construction perform only pure reads: loadServerSettings and
loadSettings no longer write during load (inline-token rewrite and corrupt-file
self-delete removed from the load paths). The inline-token migration moves to a
new idempotent MigrateInlineAuthTokens global migration, run by DataMigrator
after the store is built (after MigrateInstallIdToGlobal, which must read the
legacy installId before the tokenless rewrite drops it).
Proves the sync mechanics the editorial-review feature relies on:
server-minted drafts download to clean clients, offline ID collisions
re-ID via the existing client machinery, server-side scene rewrites
auto-download or surface the standard conflict.
* Lock the sync baseline to the server-confirmed hash
Editing a scene through the editor stamps `lastEdited` (an autosave side
effect), and the sync conflict baseline was being re-derived from local
state at mark-dirty time. Because `lastEdited` had already moved, the
recorded baseline disagreed with what the server stored — so a single
client editing its own scene and resyncing raised a phantom conflict on
every sync, with no other device involved.
Stop re-deriving the baseline. The conflict baseline (`originalHash`) is
now the hash the server last confirmed it holds: recorded only on a
successful transfer (the exact bytes uploaded or downloaded) and read
back when an entity next goes dirty. A field that mutates locally after
a sync can no longer taint the baseline.
- ProjectSynchronizationData gains `syncedHashes` (per-entity confirmed
hash). EntityOriginalState.originalHash is nullable: null means the
server never confirmed a hash (new entity, or first edit after upgrade)
so the server skips the conflict check and the baseline self-heals on
that upload.
- SyncJournal.markEntityAsDirty reads the baseline from syncedHashes
rather than taking a recomputed hash; recordSyncedHash sets it.
- EntitySynchronizer.uploadEntity reports the accepted hash via a
callback; the transfer records it on upload and download.
- The upload sources `originalHash` from syncedHashes at transfer time,
not the dirty entry's frozen copy — so a partial sync (some entities
uploaded, a later one failed) can't leave a stale baseline that forges
a phantom conflict on retry.
- FinalizeSync preserves syncedHashes written mid-transfer and prunes
deleted ids (including server-driven deletions) from the map.
- Repositories no longer recompute an entity hash to mark for sync.
Adds an integration regression test: a single client editing a scene
through the editor and resyncing must not raise a conflict (red before
this change, green after).
* Add e2e sync resync scenarios and stabilize the integration harness
High-value round-trip coverage for the sync baseline beyond the
content-edit regression, plus the test-harness fixes the new tests
needed to be reliable.
Tests:
- SyncHashStabilityTest: after a sync, the client's hash for a scene
equals the hash the server stored (baseline and server agree).
- ResyncBaselineScenariosTest: a single client must never raise a
phantom conflict on resync regardless of how the baseline was set —
metadata-only edit, rename, and an edit after a download establishes
the baseline. Each asserts the server's stored hash actually changed,
so a resync that uploaded nothing can't pass vacuously.
Harness (RoundTripTestBase):
- Bind the client IO/Default dispatchers to the same single-threaded
dispatcher as Main. FakeFileSystem is not thread-safe (its open-files
list is a plain MutableList), so the real multi-threaded dispatchers
let concurrent opens across the repos' background scopes corrupt it,
throwing intermittently from findOpenFile — the root of the suite's
order-dependent flakiness.
- newClient() registers each client for teardown-close, so a test that
throws before its own cleanup can't leak a project scope (and its
background coroutine scopes) into the next test.
- Shared syncNoConflict()/serverEntityHash() helpers replace per-file
copies.
- Close the per-test HttpClient in EndToEndTest.tearDown; it was created
every test and never closed, leaking engine threads.
* Add per-entity-type sync resync matrix
The lock-the-baseline fix lives in the shared sync layer, so it must
hold for every entity type. EntityTypeResyncMatrixTest exercises
create → sync → edit → resync (assert no phantom conflict) for notes,
timeline events, and encyclopedia entries; a create → sync → resync for
immutable scene drafts; and a create → sync → delete → resync that
confirms the entity is removed from the server (covering the deletion
path and the synced-hash pruning).
* Fix timeline store race on sync finalize
TimeLineRepository.storeTimeline() read timelineFlow.replayCache.first(),
which throws NoSuchElementException when the timeline was never loaded.
finalizeSync runs it on every sync — including syncs of a project that
never touched the timeline — so whether the async timeline load had
emitted yet was a race that could fail the whole sync. Guard with
firstOrNull (nothing loaded means nothing to store), matching the
pattern correctEventOrder already uses. Surfaced as order-dependent
flakiness once the e2e suite grew.
* Add sync fuzz/property test
A single client driving a seeded random interleaving of editor content
edits, metadata edits, renames, and syncs must never raise a conflict —
there is no other device, so any conflict is a phantom. Catches ordering
edge cases the hand-written scenarios miss (a sync landing between a
content edit and its debounced autosave, a rename between edits). Seeds
are fixed and printed on failure so a failing sequence can be replayed.
* Backfill sync baselines on the first post-upgrade sync
A client upgrading to the lock-the-baseline version has an empty
syncedHashes map — the field did not exist in the old sync.json. With no
baseline a dirty entity uploads with originalHash = null, and the server
skips the conflict check (ServerEntitySynchronizer only conflicts when
originalHash != null), silently overwriting a concurrent edit from
another device.
FetchLocalData now establishes a baseline for every entity that still
lacks one, before any upload reads it: the current local hash for an
in-sync entity (it equals the server's hash for an agreed entity), or
the frozen pre-edit hash carried over from the old dirty list for an
entity that was already dirty at upgrade time. The local-hash source is
safe — if local has silently diverged from the server, the backfilled
baseline is the old hash, so the next edit's upload is correctly flagged
as a conflict rather than a silent overwrite. Existing recorded baselines
are never touched, so it is a one-time, idempotent migration.
Residual: an entity edited for the very first time post-upgrade, before
any sync has run the backfill, still has no recoverable baseline.
* Remove dead SceneDraftRepository.markForSynchronization
Drafts are immutable after creation, so this was never called (its own
comment said so). Removing it leaves syncJournal unused in the repo, so
drop that field and its import too.
* Persist + assert scene content in SceneTimestampsTest
The test set content via onContentChanged + an immediate storeSceneBuffer.
onContentChanged registers the editor buffer asynchronously, so the
storeSceneBuffer found no buffer ("no buffer present"), no-op'd, and the
scene reached the server empty. The test only asserted timestamps, so it
passed while never validating that content survives the round-trip.
Persist via the synchronous storeSceneMarkdownRaw path, and assert the
server-stored entity's content equals the uploaded content so the blind
spot can't silently return.
* Rename IdRepository to IdAllocator
It is a special foundational primative
* Rename SyncDataRepository to SyncJournal
It is a special foundational primative
* Rename GlobalSettingsRepository to GlobalSettingsStore
It is a special foundational primitive
* docs: add Foundation primitives tier to the architecture doc
Document IdAllocator, SyncJournal, and GlobalSettingsStore as a fixed set of
stateful, cross-cutting primitives that the whole data layer may depend on —
acyclic leaves (GlobalSettingsStore <- SyncJournal <- IdAllocator) that named
the dependency reality instead of treating it as a no-sibling violation.
Split the monolithic SceneEditorRepository into single-responsibility pieces:
- SceneRepository — scene tree, structure, ordering, paths, on-disk layout
- SceneContentRepository — in-memory buffers, autosave, dirty tracking
- SceneMetadataRepository — per-scene and project metadata
- SceneEditorService — the component-facing facade that orchestrates the three
and applies the cross-cutting side-effects (statistics, writing activity,
reference index) the repositories deliberately don't reach up to perform
Components and synchronizers now talk to SceneEditorService; the underlying
repositories hold no sibling-repo dependencies. Tests are reorganized to match
the new boundaries, and the scene-editing domain API is documented in
docs/DESIGN_PATTERNS.md.
The previous comment claimed the range assertion proved the edit
"round-tripped instead of being stamped with NOW() server-side." It
doesn't — server-side NOW() during sync falls in the same window.
Replace with an honest description: storeAllBuffers inside prepareForSync
bumps lastEdited right before upload, and the debounced contentFlow can
fire again post-sync, so client-bumped-during-sync and server-stamped-
on-receipt are observationally identical without instrumenting the sync
pipeline. The range check still rules out drops, epoch-0 resets, and
clearly-wrong stamps.
Two failures came up running :integrationTests:jvmTest after the
SQLite-to-Postgres migration.
1. RoundTripTestBase seeded server_config with a raw epoch integer:
INSERT INTO server_config VALUES ('whitelist_enabled', 'false', 1704067200);
server_config.updated_at is now TIMESTAMPTZ; Postgres refused to
coerce the integer. Wrapped the value in to_timestamp(...).
2. SceneTimestampsTest's upload test asserted exact equality between a
pre-sync local lastEdited snapshot and the server's stored value.
SceneEditorRepository runs a debounced (~500ms) auto-save that calls
recordSceneActivity → bumps lastEdited on every Editor-source store,
and the storeAllBuffers pass inside sync's prepareForSync triggers
another. The exact-equality assertion happened to pass against fast
in-memory SQLite because the whole test finished inside the debounce
window; with the slower embedded Postgres it doesn't.
The test's intent is "edit timestamps round-trip to the server, not
stamped with NOW() server-side." Reframed the assertion as "server's
created and lastEdited both fall inside [beforeEdit, afterSync]" —
which is what the test actually proves.
Adds `created` and `lastEdited` timestamps to SceneMetadata, stamped
on every autosave tick in SceneEditorRepository.recordSceneActivity.
StatisticsService rolls them up into a new "Last Edited" tile on the
project home dashboard, and the Android widget reads the same field
from the stats cache.
The timestamps ride the sync wire on ApiProjectEntity.SceneEntity and
participate in EntityHasher.hashScene. On download, local values are
preserved when the server didn't ship them. No conflict resolution —
last write wins.
EntityHasher hash functions had their defaults removed so call sites
can't silently miss a newly added field; this caught a latent bug in
the second markForSynchronization hash call that was relying on
defaults for archived/confirmedReferences/dismissedReferences.
Spins up a real Jetty server and a real headless client in the same JVM,
runs the actual sync protocol over HTTP, and asserts against the shared
FakeFileSystem. Covers seven scenarios: smoke (handshake), client upload,
server download, independent edits, conflict resolved to server, conflict
resolved to client, and client delete.
- New :integrationTests Gradle module with the RoundTripTestBase harness,
HeadlessClient driver, and seven scenario tests.
- :server enables java-test-fixtures so EndToEndTest, E2eTestData, and
SqliteTestDatabase are shared with the new module instead of duplicated.
- CI: build.yml runs the suite on every PR/develop push; prepare-release.yml
adds an integration-tests job that gates all seven package jobs, so a
broken sync protocol blocks the release before any artifact is built.
