navidrome/persistence
Deluan Quintão d4387c5502
Some checks are pending
Pipeline: Test, Lint, Build / Get version info (push) Waiting to run
Pipeline: Test, Lint, Build / Lint Go code (push) Waiting to run
Pipeline: Test, Lint, Build / Test Go code (push) Waiting to run
Pipeline: Test, Lint, Build / Test Go code (Windows) (push) Waiting to run
Pipeline: Test, Lint, Build / Test JS code (push) Waiting to run
Pipeline: Test, Lint, Build / Lint i18n files (push) Waiting to run
Pipeline: Test, Lint, Build / Check Docker configuration (push) Waiting to run
Pipeline: Test, Lint, Build / Build (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-1 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-2 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-3 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-4 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-5 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-6 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-7 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-8 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-9 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build-10 (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Push to GHCR (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Push to Docker Hub (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Cleanup digest artifacts (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Build Windows installers (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Package/Release (push) Blocked by required conditions
Pipeline: Test, Lint, Build / Upload Linux PKG (push) Blocked by required conditions
perf(db): index media_file album/artist sort orders (#5706)
* perf(db): add composite indexes for song list album/artist sorts

The media_file sort mappings for album, artist and albumArtist expand to
multi-column ORDER BY clauses that no existing index could satisfy, so SQLite
fell back to a full table scan plus a temp B-tree sort of every row (including
the large lyrics/tags/full_text columns) even for a single 15-item page. On a
96K-track library this made /api/song?_sort=album take 3.6s on a cold cache.

Add composite indexes matching the three sort mappings, allowing the query to
walk the index and stop at the page size, in both directions. Drop the now
redundant single-column order_album_name/order_artist_name indexes (strict
prefixes of the new composites) and three indexes with no query path:
birth_time is only read in Go code, and artist/album_artist text column
lookups go through the media_file_artists table instead.

* fix(ui): make composer and track number columns non-sortable in song list

Clicking the Composer header was a silent no-op: composer is not a media_file
column, so the native API's sanitizeSort drops the sort and returns rows in
table order. Track number sorting across the whole library is not meaningful
and cannot use an index (the existing index leads with disc_number). Mark both
columns sortable={false}, like quality and mood.

* test(persistence): add sort index coverage test for large tables

Guard against sort options silently losing index support: every sort mapping
on media_file, album and artist is now verified with EXPLAIN QUERY PLAN to be
satisfiable by an index (both directions), so adding a mapping or dropping an
index that reintroduces a full-table temp B-tree sort fails the test. Sorts
that genuinely cannot use an index (random, annotation-join columns, JSON
expressions) must be declared in an exceptions list with the reason, keeping
the trade-off visible in review.

To make the sort mappings the complete declared sort surface, add identity
mappings for the media_file columns the UI sorts by without a mapping (year,
genre, duration, channels, bpm, path, comment, play_count, play_date, rating).
These are behaviorally no-ops: the same ORDER BY was previously produced by
the field whitelist fallback.

* perf(db): drop PreferSortTags expression indexes from media_file

The media_file sort_title/sort_artist_name/sort_album_name expression indexes
are only usable when PreferSortTags is enabled - a config reported by ~0.1% of
installations (insights, week of 2026-06-22) - yet every install pays their
storage (~8.6MB on a 96K-track library) and scanner write overhead. Drop them:
PreferSortTags installs fall back to a full sort for title/artist/album orders,
everyone else gets smaller DBs and cheaper writes. The order_album_name and
order_artist_name collation checks remain valid, now satisfied by the composite
sort indexes.

Signed-off-by: Deluan <deluan@navidrome.org>

---------

Signed-off-by: Deluan <deluan@navidrome.org>
2026-07-03 08:58:55 -04:00
..
e2e fix(smartplaylist): support isMissing/isPresent operators on ReplayGain fields (#5585) 2026-06-10 21:12:31 -04:00
album_repository.go fix(search): artists with atomic non-ASCII names unfindable after FTS5 migration (#5703) 2026-07-02 12:53:10 -04:00
album_repository_test.go fix(subsonic): make "recently added" order reproducible and consistent with RecentlyAddedByModTime (#5678) 2026-06-29 16:18:29 -04:00
artist_repository.go fix(search): artists with atomic non-ASCII names unfindable after FTS5 migration (#5703) 2026-07-02 12:53:10 -04:00
artist_repository_test.go perf(db): skip annotation join in CountAll when unused (#5694) 2026-06-30 22:50:43 -04:00
collation_test.go perf(db): index media_file album/artist sort orders (#5706) 2026-07-03 08:58:55 -04:00
criteria_sql.go perf(smartplaylist): use annotation index for playcount/rating/loved filters (#5662) 2026-06-24 22:26:41 -04:00
criteria_sql_benchmark_test.go perf(smartplaylists): merge negated artist/tag rules into one NOT EXISTS 2026-06-14 10:47:11 -04:00
criteria_sql_test.go perf(smartplaylist): use annotation index for playcount/rating/loved filters (#5662) 2026-06-24 22:26:41 -04:00
export_test.go feat(bfr): Big Refactor: new scanner, lots of new fields and tags, improvements and DB schema changes (#2709) 2025-02-19 20:35:17 -05:00
folder_repository.go fix(scanner): import playlists skipped when no admin existed yet (#5609) 2026-06-14 13:39:16 -04:00
folder_repository_test.go fix(scanner): import playlists skipped when no admin existed yet (#5609) 2026-06-14 13:39:16 -04:00
genre_repository.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
genre_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
helpers.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
helpers_test.go feat(bfr): Big Refactor: new scanner, lots of new fields and tags, improvements and DB schema changes (#2709) 2025-02-19 20:35:17 -05:00
library_repository.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
library_repository_test.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
mediafile_repository.go perf(db): index media_file album/artist sort orders (#5706) 2026-07-03 08:58:55 -04:00
mediafile_repository_test.go perf(db): skip annotation join in CountAll when unused (#5694) 2026-06-30 22:50:43 -04:00
persistence.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
persistence_suite_test.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
persistence_test.go fix(server): play queue should not return empty entries for deleted tracks 2024-09-20 11:22:37 -04:00
player_repository.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
player_repository_test.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
playlist_repository.go fix(playlists): allow toggling auto-import and avoid unnecessary artwork reloads (#5421) 2026-04-27 12:20:27 -04:00
playlist_repository_test.go refactor: centralize criteria sort parsing and extract smart playlist logic (#5415) 2026-04-26 14:49:59 -04:00
playlist_track_repository.go refactor: move playlist business logic from repositories to service layer (#5027) 2026-02-21 19:57:13 -05:00
playqueue_repository.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
playqueue_repository_test.go fix(server): ensure single record per user by reusing existing playqueue ID 2025-06-11 17:26:13 -04:00
plugin_cleanup.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_cleanup_test.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_repository.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
plugin_repository_test.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
property_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
property_repository_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
radio_repository.go feat(ui): add cover art support for internet radio stations (#5229) 2026-03-18 18:57:33 -04:00
radio_repository_test.go refactor: more warnings clean up 2026-05-20 17:43:12 -03:00
scrobble_buffer_repository.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_buffer_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_repository.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
scrobble_repository_test.go feat(server): track scrobble/linstens history (#4770) 2025-12-06 11:07:18 -05:00
share_repository.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
share_repository_test.go fix(playlist/share): apply user library access to import and sharing paths (#5640) 2026-06-23 18:53:51 -04:00
smart_playlist_repository.go refactor: centralize criteria sort parsing and extract smart playlist logic (#5415) 2026-04-26 14:49:59 -04:00
smart_playlist_repository_test.go fix(smartplaylists): coerce string booleans in smart playlist rules (#5450) 2026-05-01 19:21:48 -04:00
sort_index_coverage_test.go perf(db): index media_file album/artist sort orders (#5706) 2026-07-03 08:58:55 -04:00
sql_annotations.go perf(db): skip annotation join in CountAll when unused (#5694) 2026-06-30 22:50:43 -04:00
sql_annotations_test.go perf(db): skip annotation join in CountAll when unused (#5694) 2026-06-30 22:50:43 -04:00
sql_base_repository.go perf(db): skip library filter when a non-admin sees all libraries (#5696) 2026-07-01 13:12:29 -04:00
sql_base_repository_test.go test(db): fix flaky applyLibraryFilter specs on shared DB (#5697) 2026-07-01 13:58:33 -04:00
sql_bookmarks.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
sql_bookmarks_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
sql_participations.go fix(scanner): remove stale role associations when artist role changes. Fix #4242 2025-12-16 06:38:50 -05:00
sql_restful.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00
sql_restful_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
sql_search_fts.go feat(search): rank exact matches above prefix matches (#5704) 2026-07-02 15:51:03 -04:00
sql_search_fts_test.go feat(search): rank exact matches above prefix matches (#5704) 2026-07-02 15:51:03 -04:00
sql_search_like.go chore: go fix 2026-05-28 22:13:05 -03:00
sql_search_like_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_tags.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
tag_library_filtering_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
tag_repository.go feat(scanner): improve error messages for cleanup operations in annotations, bookmarks, and tags 2025-11-20 09:27:42 -05:00
tag_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
transcoding_repository.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
transcoding_repository_test.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
user_props_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
user_repository.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00
user_repository_test.go fix(nativeapi): make /api/song path filter work and use startsWith (#5566) 2026-06-05 14:00:08 -04:00