fix: reduce retry log noise during concurrent chunk processing

Addresses issue #362 - users were seeing hundreds of ERROR/WARNING logs
when processing large documents due to SurrealDB v2 transaction conflicts
during concurrent chunk embedding operations.

Changes:
- Upgraded to surreal-commands v1.3.0 which includes retry_log_level feature
- Increased retry attempts from 5 to 15 with max wait time 120s (from 30s)
  to handle deep queues during concurrent processing
- Set retry_log_level to "debug" in embed_chunk and process_source commands
- Changed repository.py RuntimeError logging from ERROR to DEBUG level
- Updated command exception handlers to log retries at DEBUG level
- Updated documentation to reflect retry strategy

This is a temporary workaround for SurrealDB v2.x transaction conflict
issues with SEARCH indexes. Settings can be reduced after migrating to
SurrealDB v3 which fixes the underlying concurrency issue.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
LUIS NOVO 2026-01-05 11:30:55 -03:00
parent b76af505b2
commit 48e2800211
7 changed files with 53 additions and 44 deletions

View file

@ -74,8 +74,8 @@ async def repo_query(
raise RuntimeError(result)
return result
except RuntimeError as e:
# RuntimeError is raised for retriable transaction conflicts - log without stack trace
logger.error(str(e))
# RuntimeError is raised for retriable transaction conflicts - log at debug to avoid noise
logger.debug(str(e))
raise
except Exception as e:
logger.exception(e)