navidrome/persistence
Deluan Quintão 63a5954e4f
Some checks are pending
Pipeline: Test, Lint, Build / Check Docker configuration (push) Waiting to run
Pipeline: Test, Lint, Build / Lint i18n files (push) Waiting to run
Pipeline: Test, Lint, Build / Test JS code (push) Waiting to run
Pipeline: Test, Lint, Build / Get version info (push) Waiting to run
Pipeline: Test, Lint, Build / Lint Go code (push) Waiting to run
Pipeline: Test, Lint, Build / Test Go code (push) Waiting to run
Pipeline: Test, Lint, Build / Test Go code (Windows) (push) Waiting to run
Pipeline: Test, Lint, Build / Build (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-1 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-2 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-3 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Cleanup digest artifacts (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-4 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-5 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-6 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-7 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build Windows installers (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-8 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-9 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-10 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Push to GHCR (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Push to Docker Hub (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Package/Release (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Upload Linux PKG (push) Blocked by required conditions
perf(smartplaylist): use annotation index for playcount/rating/loved filters (#5662)
* fix(smartplaylist): use annotation index for playcount/rating/loved filters

Annotation-field criteria wrapped the column in COALESCE(col, default) so
missing annotation rows behave as 0/false. COALESCE prevents SQLite from
using the column index, forcing a full media_file scan during smart playlist
materialization - multi-second loads on large libraries, independent of rule
complexity.

Store the raw column plus its default and drop COALESCE when the compared
value cannot match the default; fall back to 'col <op> ? OR col IS NULL' when
the default would match, so never-annotated tracks are still preserved.
Sorting keeps COALESCE to retain deterministic NULL ordering. Result set is
unchanged; the materialize query now seeks the annotation index.

Signed-off-by: Deluan <deluan@deluan.com>

* fix(smartplaylist): keep COALESCE for list-valued annotation comparisons

Hardening from final review: a list value (IN (...)) can't drive the index
and a default-inclusive list has per-element NULL semantics, so route slice
values through COALESCE(col, default) to stay exactly equivalent to the prior
form. Also make the bool-default branch explicit (loved only supports
equality operators) and share the COALESCE rendering via coalesceExpr.

Signed-off-by: Deluan <deluan@deluan.com>

* refactor(smartplaylist): make coalesced() a field method

Thermo-nuclear review follow-up: promote the free coalesceExpr(f) to a
smartPlaylistField.coalesced() method that returns the bare expression when
there is no default. This lets sortExpr call field.coalesced() unconditionally
and drop its 'if coalesceDefault != nil' branch, removing the 'only annotation
fields get coalesced' special case from the sort path. Behavior unchanged.

Signed-off-by: Deluan <deluan@deluan.com>

* fix(smartplaylist): keep COALESCE for LIKE, bool-ordering, and tag ranges

Code review (xhigh) found the index-friendly rewrite did not cover every
operator, breaking result-set equivalence on a few reachable raw-JSON paths:

- LIKE family (contains/startsWith/endsWith/notContains) on annotation fields
  used the bare column, so a NULL column never matched and missing-annotation
  rows were dropped.
- Ordering comparators (gt/lt/...) on bool fields (loved) were decided as
  equality, wrongly including never-annotated rows.
- InTheRange on a numeric tag split into two independent json_tree EXISTS,
  letting different tag values satisfy each bound.

Centralize the decision in annotationCond via bareNullInclusion: emit the
index-friendly bare form only for scalar values under an exactly-orderable
comparator, otherwise fall back to the COALESCE form (always equivalent to the
original). Route LIKE through coalesced(); reject tag/role ranges. Replace the
local toFloat/toBool with spf13/cast (fixes unhandled numeric types and string
bool forms), and drop the redundant LookupField + double reflect.TypeOf.

A 17-case brute-force check confirms row-set equivalence to the prior
COALESCE form across all operators including the fixed LIKE/bool cases.

Signed-off-by: Deluan <deluan@deluan.com>

* fix(smartplaylist): keep COALESCE for list values on bool annotation fields

Second review found the bareNullInclusion bool branch missed the non-scalar
guard the numeric branch has: a list value on loved/albumloved/artistloved
(e.g. {"is":{"loved":[true]}}) coerced through toBool (which swallowed the
cast error) to false, emitting the bare/OR-IS-NULL form and wrongly including
never-annotated rows. Make toBool return (value, ok) like toFloat and bail to
COALESCE when the value isn't a scalar bool. Also add the missing test for the
tag/role range rejection. A 21-case brute-force confirms row-set equivalence to
the original COALESCE form across every operator, including the bool/numeric
list paths.

Signed-off-by: Deluan <deluan@deluan.com>

* refactor(smartplaylist): drop spf13/cast for stdlib value coercion

The value coercion only sees the handful of types criteria produces (int,
float64, string from JSON; bool already normalized at unmarshal), so cast's
broad conversion isn't needed. Use small explicit type switches over strconv
instead, keeping the string fallback (ParseFloat/ParseBool) that closes the
'1'/'t' gap. No dependency change — cast returns to indirect.

Signed-off-by: Deluan <deluan@deluan.com>

* refactor(smartplaylist): share bool coercion via criteria.ToBool

normalizeBoolValue (unmarshal-time) and the persistence bool guard both parsed
bool-ish values independently. Extract the shared logic into an exported
criteria.ToBool(any) (bool, ok): normalizeBoolValue delegates to it (behavior
unchanged), and the persistence layer reuses it via its existing model/criteria
import instead of a local helper. No behavior change.

Signed-off-by: Deluan <deluan@deluan.com>

* refactor(smartplaylist): trim sqlLiteral and dedup rationale comments

/simplify cleanup: fmt %v already renders bool defaults as false/true, so drop
sqlLiteral's redundant bool branch. Consolidate the COALESCE-vs-index rationale
to the smartPlaylistField comment instead of repeating it across annotationCond
and the struct. No behavior change.

Signed-off-by: Deluan <deluan@deluan.com>

* fix(smartplaylist): address review feedback on multi-field maps and *any

From the PR bot reviews:
- sqlFields now uses the field's coalesced() form, so annotation fields in a
  multi-field operator map (Is/Gt/Contains with >1 key) keep COALESCE and don't
  silently drop never-annotated rows. Covers both the comparison and LIKE
  fallback paths. (Gemini high, Copilot)
- Replace coalesceDefault *any with a plain any (0/false are non-nil
  interfaces, so nil still means 'no default'); drop the coalesce() boxing
  helper and the pointer indirection. (Gemini)
- Give rangeExpr clear, range-specific errors for the multi-field and malformed
  -pair cases instead of an empty-field / 'in operator' message. (Copilot)

Adds tests for the multi-field COALESCE behavior and the new range errors.

Signed-off-by: Deluan <deluan@deluan.com>

* Revert multi-field COALESCE handling (YAGNI)

The multi-field operator map case the bots flagged is unreachable: marshalExpression
rejects any operator map with more than one field, so a multi-field map can never be
persisted or loaded. Revert the sqlFields change and its tests rather than harden a
code path no supported input can reach. Keep the two reachable improvements from the
review: coalesceDefault any (not *any), and the clearer malformed-range error.

Signed-off-by: Deluan <deluan@deluan.com>

* refactor(persistence): model comparator as a behavior-carrying struct

The smart-playlist comparator was a bare string alias, forcing two parallel
switches over the same six operators: squirrelCmp mapped each to its squirrel
constructor, and bareNullInclusion restated each as a float predicate. Adding
or changing an operator meant editing both in sync.

Make comparator a struct that bundles those facts per operator (the squirrel
builder, the operator as a float predicate, and whether it's an ordering op).
Both switches collapse: squirrelCmp is deleted in favor of cmp.build, and
bareNullInclusion's numeric switch becomes a single cmp.satisfy call. Generated
SQL is unchanged, as the existing table-driven tests confirm.

* docs(smartplaylist): trim comments that restate the code

Remove or tighten comments that describe what the code already says (likeCond and
comparisonExpr doc lines, redundant clauses in annotationField/coalesced/ToBool/
normalizeBoolValue). Keep the comments that explain non-obvious rationale: the
COALESCE-vs-index tradeoff, the bareNullInclusion/annotationCond contracts, and the
why-we-fall-back notes.

* docs(smartplaylist): collapse coalesceDefault comment to one line

The field's six-line block duplicated the COALESCE-vs-index rationale that already
lives on annotationCond. Reduce it to a one-line description plus a pointer there.

---------

Signed-off-by: Deluan <deluan@deluan.com>
2026-06-24 22:26:41 -04:00
..
e2e fix(smartplaylist): support isMissing/isPresent operators on ReplayGain fields (#5585) 2026-06-10 21:12:31 -04:00
album_repository.go fix(subsonic): always emit required created field on AlbumID3 (#5340) 2026-04-10 19:29:20 -04:00
album_repository_test.go fix(subsonic): always emit required created field on AlbumID3 (#5340) 2026-04-10 19:29:20 -04:00
artist_repository.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
artist_repository_test.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
collation_test.go fix(ui): make playlist name sorting case-insensitive (#4845) 2026-01-05 19:05:11 -05:00
criteria_sql.go perf(smartplaylist): use annotation index for playcount/rating/loved filters (#5662) 2026-06-24 22:26:41 -04:00
criteria_sql_benchmark_test.go perf(smartplaylists): merge negated artist/tag rules into one NOT EXISTS 2026-06-14 10:47:11 -04:00
criteria_sql_test.go perf(smartplaylist): use annotation index for playcount/rating/loved filters (#5662) 2026-06-24 22:26:41 -04:00
export_test.go feat(bfr): Big Refactor: new scanner, lots of new fields and tags, improvements and DB schema changes (#2709) 2025-02-19 20:35:17 -05:00
folder_repository.go fix(scanner): import playlists skipped when no admin existed yet (#5609) 2026-06-14 13:39:16 -04:00
folder_repository_test.go fix(scanner): import playlists skipped when no admin existed yet (#5609) 2026-06-14 13:39:16 -04:00
genre_repository.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
genre_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
helpers.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
helpers_test.go feat(bfr): Big Refactor: new scanner, lots of new fields and tags, improvements and DB schema changes (#2709) 2025-02-19 20:35:17 -05:00
library_repository.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
library_repository_test.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
mediafile_repository.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
mediafile_repository_test.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
persistence.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
persistence_suite_test.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
persistence_test.go fix(server): play queue should not return empty entries for deleted tracks 2024-09-20 11:22:37 -04:00
player_repository.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
player_repository_test.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
playlist_repository.go fix(playlists): allow toggling auto-import and avoid unnecessary artwork reloads (#5421) 2026-04-27 12:20:27 -04:00
playlist_repository_test.go refactor: centralize criteria sort parsing and extract smart playlist logic (#5415) 2026-04-26 14:49:59 -04:00
playlist_track_repository.go refactor: move playlist business logic from repositories to service layer (#5027) 2026-02-21 19:57:13 -05:00
playqueue_repository.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
playqueue_repository_test.go fix(server): ensure single record per user by reusing existing playqueue ID 2025-06-11 17:26:13 -04:00
plugin_cleanup.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_cleanup_test.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_repository.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
plugin_repository_test.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
property_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
property_repository_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
radio_repository.go feat(ui): add cover art support for internet radio stations (#5229) 2026-03-18 18:57:33 -04:00
radio_repository_test.go refactor: more warnings clean up 2026-05-20 17:43:12 -03:00
scrobble_buffer_repository.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_buffer_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_repository.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_repository_test.go feat(server): track scrobble/linstens history (#4770) 2025-12-06 11:07:18 -05:00
share_repository.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
share_repository_test.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
smart_playlist_repository.go refactor: centralize criteria sort parsing and extract smart playlist logic (#5415) 2026-04-26 14:49:59 -04:00
smart_playlist_repository_test.go fix(smartplaylists): coerce string booleans in smart playlist rules (#5450) 2026-05-01 19:21:48 -04:00
sql_annotations.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
sql_annotations_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
sql_base_repository.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
sql_base_repository_test.go fix(server): headless library access improvements (#4362) 2025-07-20 15:58:21 -04:00
sql_bookmarks.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
sql_bookmarks_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
sql_participations.go fix(scanner): remove stale role associations when artist role changes. Fix #4242 2025-12-16 06:38:50 -05:00
sql_restful.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00
sql_restful_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
sql_search_fts.go fix(opensubsonic): make search3 empty-query pagination fast at large offsets (#5601) 2026-06-12 15:53:37 -04:00
sql_search_fts_test.go fix(search): normalization for non-NFKD Unicode letters (ø, æ, œ, ß) (#5413) 2026-04-25 20:27:38 -04:00
sql_search_like.go chore: go fix 2026-05-28 22:13:05 -03:00
sql_search_like_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_tags.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
tag_library_filtering_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
tag_repository.go feat(scanner): improve error messages for cleanup operations in annotations, bookmarks, and tags 2025-11-20 09:27:42 -05:00
tag_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
transcoding_repository.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
transcoding_repository_test.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
user_props_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
user_repository.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00
user_repository_test.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00