navidrome/plugins/cmd/ndpgen/internal/parser.go
Deluan Quintão f580d3ea76
feat(plugins): expose the song Matcher as a host service (#5643)
* feat(plugins): add public Track and Artist DTOs for host services

* feat(plugins): add Matcher host-service interface and MatchSong DTO

* feat(plugins): generate Matcher host wrappers, PDK clients, and matcher permission

* feat(plugins): implement Matcher host service and MediaFile-to-Track converter

Also fixes an ndpgen bug where ParseDirectory parsed each host-service file
in isolation, so a service method referencing a struct defined in another
file of the same package (host.Track in track.go) could not be resolved.
ParseDirectory now collects package-wide structs in a first pass, mirroring
ParseCapabilities; PDK clients regenerated cleanly via make gen.

* feat(plugins): register Matcher host service in the manager

* test(plugins): add Matcher host service integration test plugin

* refactor(plugins): simplify matcher converter and parser file collection

- toTrack: use gg.V for nil-able field derefs and slice.Map for genres/
  participants, removing the repeated nil-guard blocks and inner loop
- manager_loader: drop the redundant ds==nil guard (loadEnabledPlugins
  already gates a nil DataStore), matching the other service entries
- ndpgen parser: extract collectGoFiles, shared by ParseDirectory and
  ParseCapabilities instead of duplicating the file-filter loop

* fix(plugins): keep nullable Track numerics as pointers

ReplayGain values, BitDepth, and BPM are nullable in model.MediaFile, and 0
is a valid measured ReplayGain value. Flattening them to value types with
omitempty made a real 0 indistinguishable from absent. Model them as *float64
/*int32 so plugins can tell 'no data' from a measured 0. Regenerated PDK
clients; converter passes the model pointers through (RG) or maps *int->*int32
(BitDepth/BPM).

* refactor(plugins): trim redundant pass labels in ndpgen ParseDirectory

The function doc already explains the two-pass approach; the inline labels
restated it. Reduce to bare waypoints.

* fix(plugins): gate Track.Path on library filesystem permission

MatchSongs copied mf.Path into every result unconditionally, letting a plugin
with only the matcher permission enumerate on-disk file paths by matching known
songs. Gate Path behind library.filesystem, matching the Library host service.
toTrack is now a method carrying the permission flag.

* refactor(plugins): align MatchSong JSON casing and parse Go files once

- MatchSong: artistMBID/albumMBID JSON tags -> artistMbid/albumMbid so the Go
  wire format matches the Rust SDK's camelCase serialization (cross-SDK fix)
- MatchSongs doc reworded to language-neutral 'empty (absent)' so generated
  Rust/Python client docs no longer say Go-specific 'nil'
- ndpgen: parse each package file once (parseGoFiles) and reuse the ASTs across
  both passes in ParseDirectory and ParseCapabilities, instead of re-parsing

* refactor(plugins): use shared types for Matcher host service

Move the Matcher host service onto the shared plugins/types package instead
of the host-local MatchSong and Track structs. MatchSongs now takes
[]types.SongRef and returns []*types.Track, dropping host.MatchSong and moving
host.Track (with its host.Artist dependency collapsed onto types.ArtistRef) into
plugins/types. ArtistRef gains SortName and SubRole so it can back a track's
Participants.

SongRef gains a millisecond-precision DurationMs field that supersedes the now
deprecated seconds-based Duration, with DurationInMs() resolving the effective
value and SetDurationMs() keeping both fields in sync when populating a SongRef
to send to a plugin.

The ndpgen host-wrapper template only ever imported context, json and extism, so
a host service referencing the shared types package produced uncompilable code.
Emit the plugins/types import when the service references shared types directly
(gated on the existing Service.ImportsSharedTypes), matching the client template,
and cover it with GenerateHost tests. This removes the need for host-local
re-export aliases. Regenerated the Go/Rust/Python PDK and capability schemas
accordingly.

* test(plugins): cover SongRef duration and artist conversion

Add unit coverage for the new SongRef behavior: SetDurationMs populating both
DurationMs and the deprecated seconds field, and the SongRef-to-agents.Song
conversion preferring DurationMs over Duration and the Artists list over the
scalar Artist/ArtistMBID.

Extract the inline SongRef-to-agents.Song closure in MatchSongs into a named
toAgentSong function so the conversion can be asserted directly rather than only
through the opaque matcher. The end-to-end wire shape of the moved types is
already validated by the existing MatcherService integration test, so no new
WASM-boundary test is needed.

* fix(plugins): harden and unify SongRef-to-agents.Song duration conversion

Address findings from a code review of the matcher host service:

- DurationInMs now clamps a negative deprecated-seconds value to 0 instead of
  converting it through uint32, which previously wrapped a value like -1s into a
  ~49-day duration that corrupted the matcher's duration-proximity tiebreaker.
- Replace the unused SetDurationMs(uint32) with SetDuration(seconds float32),
  which takes the unit callers actually hold (model.MediaFile.Duration is
  float32 seconds) and centralizes the seconds-to-ms conversion. Wire it into
  mediaFileToSongRef so outbound SongRefs carry both duration fields in sync.
- Make the metadata-agent path use DurationInMs() so every consumer of the
  shared SongRef honors the DurationMs-over-Duration precedence contract; a
  plugin sending only DurationMs no longer loses its duration on that path.
- Collapse the matcher's duplicate toAgentSong/agentArtists helpers into the
  existing songRefToAgentSong converter, so there is a single SongRef-to-Song
  mapping. Tests narrowed to the duration cases, with artist precedence still
  covered in metadata_agent_test.go.

* feat(plugins): allow Matcher host service to scope a match to a user

Add an options struct to the Matcher host service so a plugin can run a match as
a specific user. When MatchOptions.Username is set, the match is run in that
user's context: their favourites and ratings inform the matcher's tiebreaker, and
the returned tracks carry that user's per-user annotations (Starred, StarredAt,
Rating, PlayCount, PlayDate, added to types.Track). An empty username preserves
the previous unscoped behaviour.

Cross-user access is gated by the same allowedUsers/allUsers permission the Users
and SubsonicAPI host services use: an unknown username, or one the plugin is not
permitted to act as, returns an error. User-library access applies automatically
once the user is in context (applyLibraryFilter). Independently, results are now
restricted to the libraries the plugin itself may access via the precomputed
libraryAccess set, dropping any matched track outside that set (the input index
stays unmatched) — this applies even without a username and even for an
admin-scoped user.

core/matcher is unchanged: it already loads and uses annotations and applies
user-library filtering from context, so the feature works by deriving the request
context and post-filtering by plugin library access in the host adapter. The new
opts parameter and the Track annotation fields are propagated to all PDK clients
(Go/Rust/Python) by make gen.

* fix(plugins): correct Matcher library scope and unify user-access checks

Address findings from a code review of the user-scoped Matcher host service:

- The plugin-library post-filter previously dropped every match for a plugin that
  holds only the matcher permission, because library config is tied to the Library
  permission and a matcher-only plugin has none (empty allowedLibraries,
  AllLibraries=false). Gate the filter on whether the plugin actually declared the
  Library permission: matcher-only plugins are no longer library-restricted, while
  plugins that opt into a library scope are enforced as before. The per-user
  library filter (applyLibraryFilter) still applies whenever a non-admin user is
  scoped.

- resolveUser collapsed every FindByUsername error (including transient DB
  failures) into a misleading "not found". Extract a shared userAccess type
  (alongside libraryAccess) whose resolve() distinguishes model.ErrNotFound from a
  real backend error and authorizes the user against the allowed set. The Matcher
  service now uses it, and host_subsonicapi shares the same userAccess type for its
  permission check (preserving its existing error messages), removing a third
  divergent copy of the resolve-and-authorize logic.

- Document in the matcher tests that the mock MediaFileRepo returns annotations
  unconditionally, so the unit tests cover the adapter's scoped-flag gating and
  access checks but not the SQL per-user join. Add tests for the library-permission
  gating and for surfacing a backend error instead of masking it as not-found.

* fix(plugins): require a library scope for Matcher, fail closed

Reverse the permissive default introduced when fixing the library post-filter: a
Matcher plugin now must be granted a library scope (all libraries, or at least one
specific library) and MatchSongs rejects the request with "no libraries
configured" when it has none, instead of either silently matching nothing or
defaulting to every library.

This mirrors how the SubsonicAPI host service requires a user scope
(checkPermissions errors with "no users configured" when none is set): the check
is a runtime guard via libraryAccess.configured(), needs no manifest changes, and
keeps the failure loud rather than silent. The per-match library post-filter then
always applies, and the restrictLibraries flag added in the previous commit is
removed.

* fix(plugins): require library permission for matcher; guard nil user

Close the gap where a plugin declaring only the matcher permission loaded
successfully but failed every MatchSongs call with "no libraries configured",
with no way for an admin to grant a library scope (the library-config UI is gated
on the library permission). Add a cross-field manifest rule, mirroring the
existing "subsonicapi requires users" rule, so the matcher permission requires
the library permission to be declared. A matcher plugin therefore also surfaces
the library-config panel and is subject to the existing load/enable-time library
configuration gate, making the fail-closed library check reachable and fixable
rather than a silent dead end. The test plugin manifest now declares the library
permission accordingly.

Also restore a defensive nil-user guard in userAccess.resolve: if a DataStore's
FindByUsername ever returns (nil, nil) instead of model.ErrNotFound, return a
clean "not found" error rather than dereferencing a nil *model.User.

* feat(plugins): expose track AverageRating in Matcher results

Add AverageRating to the Matcher's Track DTO. Unlike the per-user annotations
(Starred, Rating, PlayCount, ...), AverageRating is an aggregate stored on the
track itself and is loaded regardless of the request user, so it is populated
unconditionally rather than gated on a scoped username. Propagated to the PDK
types by make gen.

Signed-off-by: Deluan <deluan@navidrome.org>

* style(plugins): trim verbose comments in matcher host service

Condense the over-long explanatory comments added across the matcher host
service to one-liners that state the why, and simplify the ptrInt32/unixPtr
helpers to Go 1.26's new(value). No behavior change.

* refactor(plugins): pass userAccess into newSubsonicAPIService

Move newUserAccess construction to the loader call site so the SubsonicAPI service
constructor takes a userAccess value directly, matching newMatcherService. Pure
refactor: the service already stored a userAccess internally, so behavior and error
messages are unchanged.

* fix(plugins): regenerate PDK and drop omitempty from AverageRating

Re-run make gen so the generated PDK doc comments match the source comment
trimmed in an earlier commit (the source was simplified but the PDK was not
regenerated, leaving the committed files stale — a 'generated files up to date'
hazard).

Also drop omitempty from Track.AverageRating: it is always set (0 when unrated),
so it should be present in the payload like the other always-set fields
(BirthTime/CreatedAt/UpdatedAt), not dropped at zero. Tag change propagated to the
PDK by the same regeneration.

* fix(plugins): reject user-scoped match before lookup when plugin has no user scope

A matcher plugin requires the library permission but not the users permission, so
a matcher-only plugin always has an empty user scope (allUsers=false, no allowed
users). MatchSongs still ran FindByUsername for any opts.Username before checking
authorization and returned distinguishable errors ('user X not found' vs 'not
allowed to act as user X'), letting such a plugin enumerate account names from the
error text.

Guard userAccess.resolve to reject with a single fixed error before the lookup when
the plugin has no user scope, mirroring how the SubsonicAPI service short-circuits
with 'no users configured'. The unscoped match path (no username) is unaffected, so
matcher-only plugins still match normally.

* fix(plugins): run unscoped matcher as admin, not the inherited request user

A matcher host call can arrive on a context that already carries a request user
(e.g. a plugin capability invoked while serving that user's request — extism
propagates the call context into host functions). With no opts.Username, MatchSongs
passed that context straight through, so the media-file repository applied the
caller's library filter and per-user annotation ranking to an explicitly unscoped
match.

Set the user context explicitly: a username scopes to that user (overriding any
inherited one), and an unscoped match runs under adminContext so only the plugin's
own library scope constrains results. Adds tests using a context-capturing
DataStore to assert the user the matcher resolves in both cases.

* chore(plugins): drop the generated Python matcher PDK

The Python plugin PDK is no longer supported (ndpgen generates only Go and Rust
clients), so remove the stale generated nd_host_matcher.py rather than leave a
client that drifts from the host interface.

* docs(plugins): deprecate SongRef.Artist/ArtistMBID in favor of Artists

Mark the scalar single-artist fields deprecated; Artists (the ArtistRef list) is
the preferred way to supply artist data and already takes precedence for matching.
Propagated to the PDK and capability schemas by make gen.

* refactor(plugins): flatten Track.Participants and add Role to ArtistRef

Change Track.Participants from map[role][]ArtistRef to a flat []ArtistRef, and give
ArtistRef a Role field (the participation category: artist/composer/performer/...)
alongside SubRole (a specialization within a role, e.g. the instrument for a
performer). In the flat list each entry now self-describes its role rather than
relying on a map key, matching how SongRef.Artists is already a flat list; the
converter tags each entry with its role and emits them in a stable role order.
Propagated to the PDK and capability schemas by make gen.

---------

Signed-off-by: Deluan <deluan@navidrome.org>
2026-07-01 11:19:15 -04:00

1046 lines
30 KiB
Go

package internal
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"maps"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
)
// Annotation patterns
var (
// //nd:hostservice name=ServiceName permission=key
hostServicePattern = regexp.MustCompile(`//nd:hostservice\s+(.*)`)
// //nd:hostfunc [name=CustomName]
hostFuncPattern = regexp.MustCompile(`//nd:hostfunc(?:\s+(.*))?`)
// //nd:capability name=PackageName [required=true]
capabilityPattern = regexp.MustCompile(`//nd:capability\s+(.*)`)
// //nd:export name=ExportName
exportPattern = regexp.MustCompile(`//nd:export\s+(.*)`)
// key=value pairs
keyValuePattern = regexp.MustCompile(`(\w+)=(\S+)`)
)
// parsedGoFile pairs a source path with its already-parsed AST.
type parsedGoFile struct {
path string
file *ast.File
}
// parseGoFiles returns the eligible Go source files in dir, each parsed once.
func parseGoFiles(dir string, fset *token.FileSet) ([]parsedGoFile, error) {
paths, err := goSourceFiles(dir)
if err != nil {
return nil, err
}
out := make([]parsedGoFile, 0, len(paths))
for _, p := range paths {
f, err := parser.ParseFile(fset, p, nil, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(p), err)
}
out = append(out, parsedGoFile{path: p, file: f})
}
return out, nil
}
// goSourceFiles returns the Go source file paths in dir, excluding generated,
// test, and doc files.
func goSourceFiles(dir string) ([]string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("reading directory: %w", err)
}
var paths []string
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || !strings.HasSuffix(name, ".go") {
continue
}
if strings.HasSuffix(name, "_gen.go") || strings.HasSuffix(name, "_test.go") || name == "doc.go" {
continue
}
paths = append(paths, filepath.Join(dir, name))
}
return paths, nil
}
// ParseDirectory parses all Go source files in a directory and extracts host services.
func ParseDirectory(dir string) ([]Service, error) {
return ParseDirectoryWithShared(dir, nil)
}
// ParseDirectoryWithShared parses all Go source files in a directory, resolving any
// type aliases that reference the shared `types` package against the provided registry.
func ParseDirectoryWithShared(dir string, shared map[string]StructDef) ([]Service, error) {
fset := token.NewFileSet()
parsed, err := parseGoFiles(dir, fset)
if err != nil {
return nil, err
}
// First pass: collect all struct definitions and type aliases from every file
// so that a struct or alias declared in one file is visible when resolving
// types in a sibling file.
pkgStructMap := make(map[string]StructDef)
pkgAliasMap := make(map[string]TypeAlias)
for _, pf := range parsed {
for _, s := range parseStructs(pf.file) {
pkgStructMap[s.Name] = s
}
for _, a := range parseTypeAliases(pf.file) {
pkgAliasMap[a.Name] = a
}
}
// Second pass: parse services using the package-level maps.
var services []Service
for _, pf := range parsed {
svcList, err := parseServiceFile(pf.file, pkgStructMap, pkgAliasMap, shared)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(pf.path), err)
}
services = append(services, svcList...)
}
return services, nil
}
// ParseCapabilities parses all Go source files in a directory and extracts capabilities.
func ParseCapabilities(dir string) ([]Capability, error) {
return ParseCapabilitiesWithShared(dir, nil)
}
// ParseCapabilitiesWithShared parses all Go source files in a directory, resolving any
// type aliases that reference the shared `types` package against the provided registry.
func ParseCapabilitiesWithShared(dir string, shared map[string]StructDef) ([]Capability, error) {
fset := token.NewFileSet()
parsed, err := parseGoFiles(dir, fset)
if err != nil {
return nil, err
}
// First pass: collect all structs, type aliases, and const groups.
pkgStructMap := make(map[string]StructDef)
pkgAliasMap := make(map[string]TypeAlias)
var allConstGroups []ConstGroup
for _, pf := range parsed {
for _, s := range parseStructs(pf.file) {
pkgStructMap[s.Name] = s
}
for _, a := range parseTypeAliases(pf.file) {
pkgAliasMap[a.Name] = a
}
allConstGroups = append(allConstGroups, parseConstGroups(pf.file)...)
}
// Second pass: parse capabilities using the package-level type maps.
var capabilities []Capability
for _, pf := range parsed {
capList, err := parseCapabilityFile(pf.path, pf.file, pkgStructMap, pkgAliasMap, allConstGroups, shared)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(pf.path), err)
}
capabilities = append(capabilities, capList...)
}
return capabilities, nil
}
// LoadSharedTypes parses every struct defined in dir (the shared `types` source
// package) and returns them keyed by name. dir == "" yields an empty map.
func LoadSharedTypes(dir string) (map[string]StructDef, error) {
result := map[string]StructDef{}
if dir == "" {
return result, nil
}
paths, err := goSourceFiles(dir)
if err != nil {
return nil, fmt.Errorf("reading shared types directory: %w", err)
}
fset := token.NewFileSet()
for _, path := range paths {
f, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(path), err)
}
for _, s := range parseStructs(f) {
result[s.Name] = s
}
}
return result, nil
}
// parseCapabilityFile parses a single Go source file and extracts capabilities.
func parseCapabilityFile(path string, f *ast.File, structMap map[string]StructDef, aliasMap map[string]TypeAlias, allConstGroups []ConstGroup, shared map[string]StructDef) ([]Capability, error) {
var capabilities []Capability
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
interfaceType, ok := typeSpec.Type.(*ast.InterfaceType)
if !ok {
continue
}
// Check for //nd:capability annotation in doc comment
docText, rawDoc := getDocComment(genDecl, typeSpec)
capAnnotation := parseCapabilityAnnotation(rawDoc)
if capAnnotation == nil {
continue
}
// Extract source file base name (e.g., "websocket_callback" from "websocket_callback.go")
baseName := filepath.Base(path)
sourceFile := strings.TrimSuffix(baseName, ".go")
capability := Capability{
Name: capAnnotation["name"],
Interface: typeSpec.Name.Name,
Required: capAnnotation["required"] == "true",
Doc: cleanDoc(docText),
SourceFile: sourceFile,
}
// Parse methods and collect referenced types
referencedTypes := make(map[string]bool)
for _, method := range interfaceType.Methods.List {
if len(method.Names) == 0 {
continue // Embedded interface
}
funcType, ok := method.Type.(*ast.FuncType)
if !ok {
continue
}
// Check for //nd:export annotation
methodDocText, methodRawDoc := getMethodDocComment(method)
exportAnnotation := parseExportAnnotation(methodRawDoc)
if exportAnnotation == nil {
continue
}
export, err := parseExport(method.Names[0].Name, funcType, exportAnnotation, cleanDoc(methodDocText))
if err != nil {
return nil, fmt.Errorf("parsing export %s.%s: %w", typeSpec.Name.Name, method.Names[0].Name, err)
}
capability.Methods = append(capability.Methods, export)
// Collect referenced types from input and output
collectReferencedTypes(export.Input.Type, referencedTypes)
collectReferencedTypes(export.Output.Type, referencedTypes)
}
// Recursively collect all struct dependencies
collectAllStructDependencies(referencedTypes, structMap)
// Resolve shared-type aliases against the registry
sharedAliases, sharedTypes, err := resolveSharedAliases(referencedTypes, aliasMap, shared)
if err != nil {
return nil, err
}
capability.SharedAliases = sharedAliases
capability.SharedTypes = sharedTypes
// Build a set of names already covered by SharedAliases so we don't
// emit them again in TypeAliases (which would cause a redeclaration).
sharedAliasNames := make(map[string]bool, len(capability.SharedAliases))
for _, sa := range capability.SharedAliases {
sharedAliasNames[sa.Name] = true
}
// Sort type names for stable output order
sortedTypeNames := slices.Sorted(maps.Keys(referencedTypes))
// Attach referenced structs to the capability
for _, typeName := range sortedTypeNames {
if s, exists := structMap[typeName]; exists {
capability.Structs = append(capability.Structs, s)
}
}
// Attach referenced type aliases (skip those already in SharedAliases)
for _, typeName := range sortedTypeNames {
if sharedAliasNames[typeName] {
continue
}
if a, exists := aliasMap[typeName]; exists {
capability.TypeAliases = append(capability.TypeAliases, a)
}
}
// Also attach type aliases prefixed with interface name (e.g., ScrobblerError for Scrobbler interface)
// This supports error types that are not directly referenced in method signatures
interfaceName := typeSpec.Name.Name
for _, typeName := range slices.Sorted(maps.Keys(aliasMap)) {
if sharedAliasNames[typeName] {
continue
}
a := aliasMap[typeName]
if strings.HasPrefix(typeName, interfaceName) && !referencedTypes[typeName] {
capability.TypeAliases = append(capability.TypeAliases, a)
referencedTypes[typeName] = true // Mark as referenced for const lookup
}
}
// Attach const groups that match referenced type aliases
for _, group := range allConstGroups {
if group.Type == "" {
continue
}
if referencedTypes[group.Type] {
capability.Consts = append(capability.Consts, group)
}
}
if len(capability.Methods) > 0 {
capabilities = append(capabilities, capability)
}
}
}
return capabilities, nil
}
// resolveSharedAliases determines which shared `types` package structs a host
// service or capability uses and returns the deprecated re-export aliases to emit
// for them.
//
// A shared type counts as used when a field references it by qualified name
// (e.g. types.Track) or via a declared alias used by bare name (e.g. a field of
// type TrackInfo where `type TrackInfo = types.Track`). The shared struct's own
// fields are followed transitively so nested shared types are picked up too. For
// every used canonical type, each declared `type X = types.Canonical` alias is
// emitted as a SharedAlias so the generated PDK keeps re-exporting it for
// backwards compatibility.
//
// It returns the deprecated re-export aliases to emit and the resolved shapes of
// every used shared type (alias or not, for schema inlining).
//
// Returns an error if a referenced shared type cannot be found in the shared registry.
func resolveSharedAliases(referenced map[string]bool, aliasMap map[string]TypeAlias, shared map[string]StructDef) ([]SharedAlias, []StructDef, error) {
// Index declared shared aliases by the canonical type they target, e.g.
// "Track" -> [TrackInfo]. A canonical type may have more than one alias.
aliasesByCanonical := map[string][]TypeAlias{}
for _, a := range aliasMap {
if a.IsSharedAlias() {
canonical := strings.TrimPrefix(a.Type, sharedTypesPrefix)
aliasesByCanonical[canonical] = append(aliasesByCanonical[canonical], a)
}
}
// Walk the referenced types, following nested shared references inside the
// shared structs, to find the set of canonical shared types used.
used := map[string]bool{}
var queue []string
for name := range referenced {
if c, ok := seedSharedCanonical(name, aliasMap); ok {
queue = append(queue, c)
}
}
fieldRefs := map[string]bool{}
for len(queue) > 0 {
canonical := queue[0]
queue = queue[1:]
if used[canonical] {
continue
}
def, ok := shared[canonical]
if !ok {
return nil, nil, fmt.Errorf(
"shared type %q could not be resolved: pass -shared=<dir> pointing at the shared types package, and ensure %s is defined there",
canonical, sharedTypesPrefix+canonical,
)
}
used[canonical] = true
for _, f := range def.Fields {
clear(fieldRefs)
collectReferencedTypes(f.Type, fieldRefs)
for t := range fieldRefs {
if c, ok := nestedSharedCanonical(t, shared); ok {
queue = append(queue, c)
}
}
}
}
var out []SharedAlias
var usedDefs []StructDef
for canonical := range used {
usedDefs = append(usedDefs, shared[canonical])
for _, a := range aliasesByCanonical[canonical] {
out = append(out, SharedAlias{Name: a.Name, Target: a.Type, Doc: a.Doc, Def: shared[canonical]})
}
}
slices.SortFunc(out, func(a, b SharedAlias) int { return strings.Compare(a.Name, b.Name) })
slices.SortFunc(usedDefs, func(a, b StructDef) int { return strings.Compare(a.Name, b.Name) })
return out, usedDefs, nil
}
// seedSharedCanonical maps a type token referenced by a capability/service field
// to the canonical shared type it denotes. It recognizes qualified references
// (types.X -> X) and declared shared aliases used by bare name (X where
// `type X = types.Y` -> Y). A bare name that is not a declared shared alias is
// not treated as shared, so a local struct sharing a name with a shared type is
// never misclassified.
func seedSharedCanonical(name string, aliasMap map[string]TypeAlias) (string, bool) {
if rest, ok := strings.CutPrefix(name, sharedTypesPrefix); ok {
return rest, true
}
if a, ok := aliasMap[name]; ok && a.IsSharedAlias() {
return strings.TrimPrefix(a.Type, sharedTypesPrefix), true
}
return "", false
}
// nestedSharedCanonical maps a type token found inside a shared struct's own
// fields to a canonical shared type. Within the shared package, types reference
// each other by bare name (e.g. Track.Artists is []ArtistRef), so any bare name
// present in the shared registry counts.
func nestedSharedCanonical(name string, shared map[string]StructDef) (string, bool) {
if rest, ok := strings.CutPrefix(name, sharedTypesPrefix); ok {
return rest, true
}
if _, ok := shared[name]; ok {
return name, true
}
return "", false
}
// collectAllStructDependencies recursively collects all struct types referenced by other structs.
func collectAllStructDependencies(referencedTypes map[string]bool, structMap map[string]StructDef) {
// Keep iterating until no new types are added
for {
newTypes := make(map[string]bool)
for typeName := range referencedTypes {
if s, exists := structMap[typeName]; exists {
for _, field := range s.Fields {
collectReferencedTypes(field.Type, newTypes)
}
}
}
// Check if any new types were found
foundNew := false
for t := range newTypes {
if !referencedTypes[t] {
referencedTypes[t] = true
foundNew = true
}
}
if !foundNew {
break
}
}
}
// parseExport parses an export method signature into an Export struct.
func parseExport(name string, funcType *ast.FuncType, annotation map[string]string, doc string) (Export, error) {
export := Export{
Name: name,
ExportName: annotation["name"],
Doc: doc,
}
// Capability exports have exactly one input parameter (the struct type)
if funcType.Params != nil && len(funcType.Params.List) == 1 {
field := funcType.Params.List[0]
typeName := typeToString(field.Type)
paramName := "input"
if len(field.Names) > 0 {
paramName = field.Names[0].Name
}
export.Input = NewParam(paramName, typeName)
}
// Capability exports return (OutputType, error)
if funcType.Results != nil {
for _, field := range funcType.Results.List {
typeName := typeToString(field.Type)
if typeName == "error" {
continue // Skip error return
}
paramName := "output"
if len(field.Names) > 0 {
paramName = field.Names[0].Name
}
export.Output = NewParam(paramName, typeName)
break // Only take the first non-error return
}
}
return export, nil
}
// parseServiceFile parses a single Go source file and extracts host services.
// pkgStructMap and pkgAliasMap are the package-wide struct and alias maps built
// from all files in the package, so a host-service interface can reference types
// defined in a sibling file.
func parseServiceFile(f *ast.File, pkgStructMap map[string]StructDef, pkgAliasMap map[string]TypeAlias, shared map[string]StructDef) ([]Service, error) {
var services []Service
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
interfaceType, ok := typeSpec.Type.(*ast.InterfaceType)
if !ok {
continue
}
// Check for //nd:hostservice annotation in doc comment
docText, rawDoc := getDocComment(genDecl, typeSpec)
svcAnnotation := parseHostServiceAnnotation(rawDoc)
if svcAnnotation == nil {
continue
}
service := Service{
Name: svcAnnotation["name"],
Permission: svcAnnotation["permission"],
Interface: typeSpec.Name.Name,
Doc: cleanDoc(docText),
}
// Parse methods and collect referenced types
referencedTypes := make(map[string]bool)
for _, method := range interfaceType.Methods.List {
if len(method.Names) == 0 {
continue // Embedded interface
}
funcType, ok := method.Type.(*ast.FuncType)
if !ok {
continue
}
// Check for //nd:hostfunc annotation
methodDocText, methodRawDoc := getMethodDocComment(method)
methodAnnotation := parseHostFuncAnnotation(methodRawDoc)
if methodAnnotation == nil {
continue
}
m, err := parseMethod(method.Names[0].Name, funcType, methodAnnotation, cleanDoc(methodDocText))
if err != nil {
return nil, fmt.Errorf("parsing method %s.%s: %w", typeSpec.Name.Name, method.Names[0].Name, err)
}
service.Methods = append(service.Methods, m)
// Collect referenced types from params and returns
for _, p := range m.Params {
collectReferencedTypes(p.Type, referencedTypes)
}
for _, r := range m.Returns {
collectReferencedTypes(r.Type, referencedTypes)
}
}
// Resolve shared-type aliases against the registry. Host-service schemas
// are not generated (the -schemas pass is capability-only), so the resolved
// shared shapes are not needed here.
sharedAliases, _, err := resolveSharedAliases(referencedTypes, pkgAliasMap, shared)
if err != nil {
return nil, err
}
service.SharedAliases = sharedAliases
// Recursively collect all struct dependencies so types referenced only
// transitively (e.g. a field type of a referenced struct) are attached.
collectAllStructDependencies(referencedTypes, pkgStructMap)
// Attach referenced structs to the service (sorted for stable output)
for _, typeName := range slices.Sorted(maps.Keys(referencedTypes)) {
if s, exists := pkgStructMap[typeName]; exists {
service.Structs = append(service.Structs, s)
}
}
if len(service.Methods) > 0 {
services = append(services, service)
}
}
}
return services, nil
}
// parseStructs extracts all struct type definitions from a parsed Go file.
func parseStructs(f *ast.File) []StructDef {
var structs []StructDef
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
structType, ok := typeSpec.Type.(*ast.StructType)
if !ok {
continue
}
docText, _ := getDocComment(genDecl, typeSpec)
s := StructDef{
Name: typeSpec.Name.Name,
Doc: cleanDoc(docText),
}
// Parse struct fields
for _, field := range structType.Fields.List {
if len(field.Names) == 0 {
continue // Embedded field
}
fieldDef := parseStructField(field)
s.Fields = append(s.Fields, fieldDef...)
}
structs = append(structs, s)
}
}
return structs
}
// parseTypeAliases extracts all type alias definitions from a parsed Go file.
// Type aliases are non-struct type declarations like: type MyType string
func parseTypeAliases(f *ast.File) []TypeAlias {
var aliases []TypeAlias
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
// Skip struct and interface types
if _, isStruct := typeSpec.Type.(*ast.StructType); isStruct {
continue
}
if _, isInterface := typeSpec.Type.(*ast.InterfaceType); isInterface {
continue
}
docText, _ := getDocComment(genDecl, typeSpec)
aliases = append(aliases, TypeAlias{
Name: typeSpec.Name.Name,
Type: typeToString(typeSpec.Type),
Doc: cleanDoc(docText),
IsAlias: typeSpec.Assign.IsValid(),
})
}
}
return aliases
}
// parseConstGroups extracts const groups from a parsed Go file.
func parseConstGroups(f *ast.File) []ConstGroup {
var groups []ConstGroup
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.CONST {
continue
}
group := ConstGroup{}
for _, spec := range genDecl.Specs {
valueSpec, ok := spec.(*ast.ValueSpec)
if !ok {
continue
}
// Get type if specified
if valueSpec.Type != nil && group.Type == "" {
group.Type = typeToString(valueSpec.Type)
}
// Extract values
for i, name := range valueSpec.Names {
def := ConstDef{
Name: name.Name,
}
// Get value if present
if i < len(valueSpec.Values) {
def.Value = exprToString(valueSpec.Values[i])
}
// Get doc comment
if valueSpec.Doc != nil {
def.Doc = cleanDoc(valueSpec.Doc.Text())
} else if valueSpec.Comment != nil {
def.Doc = cleanDoc(valueSpec.Comment.Text())
}
group.Values = append(group.Values, def)
}
}
if len(group.Values) > 0 {
groups = append(groups, group)
}
}
return groups
}
// exprToString converts an AST expression to a Go source string.
func exprToString(expr ast.Expr) string {
switch e := expr.(type) {
case *ast.BasicLit:
return e.Value
case *ast.Ident:
return e.Name
default:
return ""
}
}
// parseStructField parses a struct field and returns FieldDef for each name.
func parseStructField(field *ast.Field) []FieldDef {
var fields []FieldDef
typeName := typeToString(field.Type)
// Parse struct tag for JSON field name and omitempty
jsonTag := ""
omitEmpty := false
if field.Tag != nil {
tag := field.Tag.Value
// Remove backticks
tag = strings.Trim(tag, "`")
// Parse json tag
jsonTag, omitEmpty = parseJSONTag(tag)
}
// Get doc comment
var doc string
if field.Doc != nil {
doc = cleanDoc(field.Doc.Text())
}
for _, name := range field.Names {
fieldJSONTag := jsonTag
if fieldJSONTag == "" {
// Default to field name with camelCase
fieldJSONTag = toJSONName(name.Name)
}
fields = append(fields, FieldDef{
Name: name.Name,
Type: typeName,
JSONTag: fieldJSONTag,
OmitEmpty: omitEmpty,
Doc: doc,
})
}
return fields
}
// parseJSONTag extracts the json field name and omitempty flag from a struct tag.
func parseJSONTag(tag string) (name string, omitEmpty bool) {
// Find json:"..." in the tag
for _, part := range strings.Split(tag, " ") {
if strings.HasPrefix(part, `json:"`) {
value := strings.TrimPrefix(part, `json:"`)
value = strings.TrimSuffix(value, `"`)
parts := strings.Split(value, ",")
if len(parts) > 0 && parts[0] != "-" {
name = parts[0]
}
for _, opt := range parts[1:] {
if opt == "omitempty" {
omitEmpty = true
}
}
return
}
}
return "", false
}
// collectReferencedTypes extracts custom type names from a Go type string.
// It handles pointers, slices, and maps, collecting base type names.
func collectReferencedTypes(goType string, refs map[string]bool) {
// Strip pointer
if strings.HasPrefix(goType, "*") {
collectReferencedTypes(goType[1:], refs)
return
}
// Strip slice
if strings.HasPrefix(goType, "[]") {
if goType != "[]byte" {
collectReferencedTypes(goType[2:], refs)
}
return
}
// Handle map
if strings.HasPrefix(goType, "map[") {
rest := goType[4:] // Remove "map["
depth := 1
keyEnd := 0
for i, r := range rest {
if r == '[' {
depth++
} else if r == ']' {
depth--
if depth == 0 {
keyEnd = i
break
}
}
}
keyType := rest[:keyEnd]
valueType := rest[keyEnd+1:]
collectReferencedTypes(keyType, refs)
collectReferencedTypes(valueType, refs)
return
}
// Qualified reference to the shared `types` package (e.g. types.Track).
// These start with a lowercase package selector, so they must be collected
// before the uppercase check below would skip them.
if strings.HasPrefix(goType, sharedTypesPrefix) {
refs[goType] = true
return
}
// Check if it's a custom type (starts with uppercase, not a builtin)
if len(goType) > 0 && goType[0] >= 'A' && goType[0] <= 'Z' {
switch goType {
case "String", "Bool", "Int", "Int32", "Int64", "Float32", "Float64":
// Not custom types (just capitalized for some reason)
default:
refs[goType] = true
}
}
}
// toJSONName is imported from types.go via the same package
// getDocComment extracts the doc comment for a type spec.
// Returns both the readable doc text and the raw comment text (which includes pragma-style comments).
func getDocComment(genDecl *ast.GenDecl, typeSpec *ast.TypeSpec) (docText, rawText string) {
var docGroup *ast.CommentGroup
// First check the TypeSpec's own doc (when multiple types in one block)
if typeSpec.Doc != nil {
docGroup = typeSpec.Doc
} else if genDecl.Doc != nil {
// Fall back to GenDecl doc (single type declaration)
docGroup = genDecl.Doc
}
if docGroup == nil {
return "", ""
}
return docGroup.Text(), commentGroupRaw(docGroup)
}
// commentGroupRaw returns all comment text including pragma-style comments (//nd:...).
// Go's ast.CommentGroup.Text() strips comments without a space after //, so we need this.
func commentGroupRaw(cg *ast.CommentGroup) string {
if cg == nil {
return ""
}
var lines []string
for _, c := range cg.List {
lines = append(lines, c.Text)
}
return strings.Join(lines, "\n")
}
// getMethodDocComment extracts the doc comment for a method.
func getMethodDocComment(field *ast.Field) (docText, rawText string) {
if field.Doc == nil {
return "", ""
}
return field.Doc.Text(), commentGroupRaw(field.Doc)
}
// parseHostServiceAnnotation extracts //nd:hostservice annotation parameters.
func parseHostServiceAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := hostServicePattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseHostFuncAnnotation extracts //nd:hostfunc annotation parameters.
func parseHostFuncAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := hostFuncPattern.FindStringSubmatch(line)
if match != nil {
params := parseKeyValuePairs(match[1])
if params == nil {
params = make(map[string]string)
}
return params
}
}
return nil
}
// parseCapabilityAnnotation extracts //nd:capability annotation parameters.
func parseCapabilityAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := capabilityPattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseExportAnnotation extracts //nd:export annotation parameters.
func parseExportAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := exportPattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseKeyValuePairs extracts key=value pairs from annotation text.
func parseKeyValuePairs(text string) map[string]string {
matches := keyValuePattern.FindAllStringSubmatch(text, -1)
if len(matches) == 0 {
return nil
}
result := make(map[string]string)
for _, m := range matches {
result[m[1]] = m[2]
}
return result
}
// parseMethod parses a method signature into a Method struct.
func parseMethod(name string, funcType *ast.FuncType, annotation map[string]string, doc string) (Method, error) {
m := Method{
Name: name,
ExportName: annotation["name"],
Doc: doc,
}
// Parse parameters (skip context.Context)
if funcType.Params != nil {
for _, field := range funcType.Params.List {
typeName := typeToString(field.Type)
if typeName == "context.Context" {
continue // Skip context parameter
}
for _, name := range field.Names {
m.Params = append(m.Params, NewParam(name.Name, typeName))
}
}
}
// Parse return values
if funcType.Results != nil {
for _, field := range funcType.Results.List {
typeName := typeToString(field.Type)
if typeName == "error" {
m.HasError = true
continue // Track error but don't include in Returns
}
// Handle anonymous returns
if len(field.Names) == 0 {
// Generate a name based on position
m.Returns = append(m.Returns, NewParam("result", typeName))
} else {
for _, name := range field.Names {
m.Returns = append(m.Returns, NewParam(name.Name, typeName))
}
}
}
}
return m, nil
}
// typeToString converts an AST type expression to a string.
func typeToString(expr ast.Expr) string {
switch t := expr.(type) {
case *ast.Ident:
return t.Name
case *ast.SelectorExpr:
return typeToString(t.X) + "." + t.Sel.Name
case *ast.StarExpr:
return "*" + typeToString(t.X)
case *ast.ArrayType:
if t.Len == nil {
return "[]" + typeToString(t.Elt)
}
return fmt.Sprintf("[%s]%s", typeToString(t.Len), typeToString(t.Elt))
case *ast.MapType:
return fmt.Sprintf("map[%s]%s", typeToString(t.Key), typeToString(t.Value))
case *ast.BasicLit:
return t.Value
case *ast.InterfaceType:
// Empty interface (interface{} or any)
if t.Methods == nil || len(t.Methods.List) == 0 {
return "any"
}
// Non-empty interfaces can't be easily represented
return "any"
default:
return fmt.Sprintf("%T", expr)
}
}
// cleanDoc removes annotation lines from documentation.
func cleanDoc(doc string) string {
var lines []string
for _, line := range strings.Split(doc, "\n") {
trimmed := strings.TrimSpace(line)
if strings.HasPrefix(trimmed, "//nd:") {
continue
}
lines = append(lines, line)
}
return strings.TrimSpace(strings.Join(lines, "\n"))
}