Commit graph

11166 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
4a05a114ab chore: plain Apache LICENSE, and the site and README say the plugins have shipped
- LICENSE is the plain Apache 2.0 text, without the "Portions of this
  software" preamble, so GitHub's About panel names the license. The two BSL
  folders keep their own LICENSE, which names each folder as its scope; the
  README's License section and the Terms page now name both, and the Terms
  point at the LICENSE files rather than the root one alone.
- /plugins: the eight plugins shipped in 2.2.1, with what each reads, how to
  start (download, free 30-day licence, Settings › Plugins, @ in a chat),
  third-party plugins and the registry, and why Reddit and Instagram are not
  offered.
- /pricing: 7 platforms and the web crawler, not 9; the plugin note, the
  licence and trial answers and the expiry answer describe the shipped
  plugins; third-party plugins need no licence.
- Homepage plugin band: no longer "Coming soon".
- README: a Plugins section linking /downloads for the free licence, in place
  of Star history.
2026-10-15 14:49:37 -07:00
Rohan Verma
c45d0fb7d6
Merge pull request #2203 from TonyWu2333/fix/script-traceback-relative-paths
fix(local): print a script's traceback relative to its run folder
2026-10-15 14:03:27 -07:00
Rohan Verma
947e9f495b
Merge pull request #2235 from MannXo/fix/2228-sidecar-spawn-errors
fix(agent): back off opencode restarts and fail Agentic fast when it cannot stay up
2026-10-15 14:00:53 -07:00
Rohan Verma
6382dec425
Merge pull request #2170 from MohammadHijjawi97/fix/mid-stream-error-ends-reply
fix(local): fail a reply the endpoint cuts off with an in-stream error
2026-10-15 13:23:32 -07:00
Rohan Verma
b43bee9e7d
Merge pull request #2183 from Cedric921/feat/podcast-mp3
feat(local): encode a podcast episode as MP3
2026-10-15 13:21:37 -07:00
Rohan Verma
23eaead701
Merge pull request #2248 from fukalous/docs/import-thread-recovery-followup
docs(import): align thread recovery documentation
2026-10-15 13:21:12 -07:00
Rohan Verma
25a8acc351
Merge pull request #2271 from MODSetter/fix/audiocpp-cmake-generator
fix(local): name Visual Studio's generator when compiling audio.cpp on Windows
2026-10-15 13:16:42 -07:00
DESKTOP-RTLN3BA\$punk
8afccad660 fix(local): name Visual Studio's generator when compiling audio.cpp on Windows
A CMake from MinGW or MSYS2, first on PATH on many Windows machines,
defaults to Ninja, which refuses -A x64 and finds no compiler outside a
Visual Studio prompt, so pnpm dev stopped at configuring audio.cpp. The
recipe now names the generator vswhere finds, such as "Visual Studio 17
2022", instead of leaving CMake to choose.
2026-10-15 12:59:34 -07:00
Anish Sarkar
1c3b52939f
Merge pull request #2270 from AnishSarkar22/feat/desktop-ui-changes
feat(local): three-way tool permissions, one shared Button, nested dialogs that block
2026-10-16 00:57:56 +05:30
Anish Sarkar
62107da0f8 fix(local): a group permission never switches on a tool that cannot take it
Picking Needs approval for a first-party Read-only group switched blocked
tools on, and they then ran without asking. A group choice now skips any
tool the approval rules keep from it, and the menu closes a choice no tool
in the group can take.
2026-10-16 00:55:20 +05:30
Anish Sarkar
ca154e95fd fix(local): widen the conversation column to max-w-2xl 2026-10-16 00:32:59 +05:30
Anish Sarkar
88ec68e1c0 fix(local): border the left-sidebar file viewer on its right 2026-10-16 00:25:52 +05:30
Anish Sarkar
cbf028d20e fix(local): narrow the settings dialog to w-5xl 2026-10-16 00:13:37 +05:30
Anish Sarkar
299666170a fix(local): a nested dialog blocks the dialog under it
Base UI stacks popups by portal order and covers everything opened before
a modal dialog with its own click blocker, which carries no z-index. Our
wrappers put z-50 on every popup, lifting a parent dialog above its child's
blocker so it stayed clickable. Follow Base UI's setup instead: #root gets
isolation: isolate and the popup wrappers drop z-50, so every nested
dialog, alert dialog and popup stacks by the order it opened. The toaster
portals to <body>, outside #root, to stay above dialogs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 00:04:32 +05:30
Anish Sarkar
a7c57daede refactor(local): every button is the shared Button, with one press motion
Raw <button> elements become Button; ones with their own layout (cards,
stretched row links, flip surfaces) use a new bare variant that keeps the
focus ring, transition and press without the look. Press feedback lives
only in Button: a 0.95 scale on every button, menu triggers included, in
over 150ms and back over 250ms on a no-overshoot curve. Wide buttons and
rows pass static and press by colour instead. The per-button press nudges
are gone, and ESLint now rejects a raw <button> or press motion in a
Button's className.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 23:47:57 +05:30
chenminghao.99
7897d0f70e docs(import): align thread recovery documentation
Signed-off-by: chenminghao.99 <fukalous@gmail.com>
2026-10-16 01:24:52 +08:00
Anish Sarkar
96290c2a4f feat(local): allow, ask or block each plugin tool, as Claude does
Each tool row trades its switch and Ask/Always allow control for three
icons with tooltips that say what each does, or why the approval rules
close it. Descriptions move to a hover on the name, and a blocked tool's
name is muted. Read-only and Can make changes fold away, with a count and
a menu of the state their tools share (Custom while they differ) that
sets every tool at once. A pick is one write of enabled and approval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 20:29:04 +05:30
Anish Sarkar
4bf6b1cf9c fix(local): enlarge the settings and add-plugin dialogs
The settings dialog grows to 1072x768 and its sidebar to 200px so long
labels fit; both are capped to the window. The add-plugin dialog moves
from max-w-md to max-w-lg. Sizes use Tailwind scale classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 20:04:53 +05:30
Anish Sarkar
cec4ba1190 fix(local): make secondary buttons stand out from cards and rows
Raise the secondary token in both themes so secondary buttons no longer
blend into hovered rows. Browse cards move off the token to a muted fill
with a softer hover, and their Connect button is now secondary like the
connected table's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 19:57:14 +05:30
Mohammad Hijjawi
eec650e0d6 fix(local): answer the chat test's in-stream error with a 502
The Model setup chat test caught httpx errors but not the new
StreamedError, so a model that failed mid-reply answered 500 instead of
the 502 "chat request failed: ..." a refused request gets.
2026-10-15 14:50:50 +01:00
Mohammad Hijjawi
1914117e65 Merge remote-tracking branch 'upstream/dev' into fix/mid-stream-error-ends-reply
# Conflicts:
#	docs/architecture/chat.md
#	surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py
2026-10-15 14:37:34 +01:00
Parman Mohammadalizadeh
a83135164f Merge remote-tracking branch 'upstream/dev' into fix/2228-sidecar-spawn-errors
# Conflicts:
#	surfsense_local/electron/src/main/sidecars/boot.ts
2026-10-15 08:58:50 +02:00
TonyWu
e7d4e05116 fix(local): write run-folder paths relative across a script's whole stderr 2026-10-15 09:46:47 +08:00
Cedric Karungu
15c6447e2d Merge branch 'dev' of github.com:MODSetter/SurfSense into feat/podcast-mp3
# Conflicts:
#	surfsense_local/backend/bundling/worker.spec
2026-10-15 03:34:39 +02:00
Rohan Verma
b17b3e9e0c
Merge pull request #2185 from Cedric921/feat/third-party-notices
Some checks are pending
Desktop Tests / Changes (push) Waiting to run
Desktop Tests / Backend (macos-latest) (push) Blocked by required conditions
Desktop Tests / Backend (ubuntu-latest) (push) Blocked by required conditions
Desktop Tests / Backend (windows-latest) (push) Blocked by required conditions
Desktop Tests / Frontend (push) Blocked by required conditions
Desktop Tests / Electron (macos-latest) (push) Blocked by required conditions
Desktop Tests / Electron (ubuntu-latest) (push) Blocked by required conditions
Desktop Tests / Electron (windows-latest) (push) Blocked by required conditions
Desktop Tests / Desktop Gate (push) Blocked by required conditions
feat(local): generate third-party notices at build time
2026-10-14 17:46:01 -07:00
Rohan Verma
2f631628b6
Merge pull request #2240 from MannXo/ci/2226-desktop-tests-dev-and-os-matrix
ci(desktop): run desktop tests on pushes to dev, and on Windows and macOS
2026-10-14 17:43:00 -07:00
Cedric Karungu
9ab8af0bf2 fix(local): list the Silero VAD pack in the notices, with its own licence 2026-10-15 02:31:28 +02:00
Cedric Karungu
6207e516b1 Merge branch 'dev' of github.com:MODSetter/SurfSense into feat/third-party-notices
# Conflicts:
#	docs/architecture/packaging.md
#	surfsense_local/electron/package.json
2026-10-15 02:14:57 +02:00
Rohan Verma
6765eb5a08
Merge pull request #2247 from ybai08/feat/studio-refine-refusal-codes
feat(local): give a refine's refusals codes the interface can translate
2026-10-14 17:08:57 -07:00
Rohan Verma
fd3fe4f42c
Merge pull request #2260 from MODSetter/perf/plugin-calls-app
perf(agent): get plugins ready beside the sync, and stop waiting on the plugin check
2026-10-14 16:13:19 -07:00
Rohan Verma
297c05775f
Merge pull request #2262 from MODSetter/perf/plugins-speed
perf(plugins): make the scraper plugins fast, and let the agent read their results
2026-10-14 16:13:14 -07:00
Rohan Verma
56f9b43a07
Merge pull request #2264 from MODSetter/feat/local-speech-recognition
feat(local): local speech recognition, for recordings as sources and voice input
2026-10-14 15:47:11 -07:00
DESKTOP-RTLN3BA\$punk
531c1f6810 fix(local): keep a ChatGPT plan out of speech settings, and docs that match v1
After dev's one-click ChatGPT sign-in, the local-only speech section must
still offer no remote model; a test now proves it with a plan that claims
the slot, and fails without the localOnly gate. The docs drop two stale
Known gaps (no speech screen; no transcription test), name the settings nav
as it reads, count migrations to 0030, mark steps 2 to 7 done in the
proposal, and note the curated recogniser's type in the catalog proposal.
2026-10-14 15:04:38 -07:00
DESKTOP-RTLN3BA\$punk
433d2664fd fix(plugins): review fixes before merge
- Google Search: Google's sorry wall is recognised by host and path, not a
  "/sorry" substring, so a result linking to a page with "sorry" in its URL
  no longer burns the wave's exit (goto and the captcha check share
  captcha.is_sorry_url).
- Walmart: a redirect still pending after the last allowed hop is a refused
  try, so the next exit is tried instead of reading an empty page.
- Amazon: a hedge whose sticky lease is unavailable falls back to the
  gateway instead of spending the call's hedge budget on nothing.
- Fixtures: live tracking and session tokens scrubbed from the captured
  pages (Indeed, Amazon, Google, Google Maps, TikTok, YouTube).
- plugin-host.md: shared_access() everywhere, and the Reddit and Instagram
  gaps now describe their fail-fast latches.
2026-10-14 15:01:56 -07:00
DESKTOP-RTLN3BA\$punk
ad52df29d9 fix(agent): fail a turn closed when opencode cannot list its servers, and always close the proxy's upstream
- When GET /mcp fails, the turn used to go on with only SurfSense's tools
  re-registered, so the session kept the last turn's rules and a held plugin's
  tool switched off since could still be offered. Without the server list
  there is nothing to deny by name, and a "*" deny would also block read and
  edit, so the turn now ends with a network error before the model is asked,
  as it does when the rules cannot be set. This predates the PR.
- The plugin proxy closes its upstream connection on every path, including
  a failed 200 write and cancellation.
2026-10-14 15:01:55 -07:00
DESKTOP-RTLN3BA\$punk
d9c0a86d98 Merge remote-tracking branch 'origin/dev' into feat/local-speech-recognition
dev took ADR 0055 for the plugin registry's Verified rule, so the speech
recognition record is now 0056, Accepted with the change it decides, and
every link to it follows. Its 'Where the code stands' section went: the
code now holds the whole decision.
2026-10-14 14:51:34 -07:00
Parman Mohammadalizadeh
f993974f0d fix(agent): earn a given-up opencode another start only on a turn or a new session
The history read that follows a failed Agentic message reached
ready_opencode too and touched the configuration again, so one failed
message restarted opencode twice and raised two toasts. A history read
now leaves the configuration alone and fails at once with Electron's
reason.

Also backdates the earlier failure file in its test, since Windows file
times can tie it with the configuration written right after.
2026-10-14 20:43:46 +02:00
Parman Mohammadalizadeh
b10d56813f Merge remote-tracking branch 'upstream/dev' into ci/2226-desktop-tests-dev-and-os-matrix 2026-10-14 20:29:27 +02:00
DESKTOP-RTLN3BA\$punk
5f393ac2cd Merge branch 'feat/local-speech-recognition-ui' into feat/local-speech-recognition
The interface half of local speech recognition: Settings > Speech recognition
and its set-up popover, recordings in the sources panel, the microphone in
Electron, and dictation in the composer.
2026-10-14 08:59:57 -07:00
DESKTOP-RTLN3BA\$punk
0de99e8a7f fix(local): decode Ogg recordings under any path and clear a stopped recording's audio
Review of the step 5 backend half (553048c02).

- An Ogg Vorbis recording whose path held a character outside the Windows
  ANSI code page (a file named 会議.ogg, or a data folder under an account
  named José) was refused for good as "This recording could not be decoded.":
  miniaudio's vorbis_get_file_info opens the path with fopen, while its stream
  reader opens a wide one. The decode child now reads an Ogg file's length from
  its own pages, the last page's granule position over the rate its first
  packet names; it matched miniaudio's reading on every fixture.
- A speech model chosen midway restarted the recording under the old language
  intent, so the header read "Japanese" and the checkpoint kept "ja" although
  the new model, whose settings start empty, detected the language. The job now
  asks the status route again and starts over under what it reports.
- A cancel of a recording queued again after giving way left its decoded audio,
  115 MB an hour, with no run left to clear it. The cancel route now drops it
  and progress.json, keeping the plan and windows for a retry, as a run's own
  end does (recordings/ended_run.py, shared with the worker, best-effort for a
  file another process holds). A cancel that lands just as a job would give
  way now ends it, instead of pause_job refusing quietly with the audio kept.
- build-audiocpp.yml's path filter takes modules/documents/recordings/, which
  the end-to-end gate reads too.
2026-10-14 08:58:40 -07:00
DESKTOP-RTLN3BA\$punk
553048c02e feat(local): transcribe recordings as sources in the ingest worker
Step 5 of the local speech recognition proposal, backend half. A recording
uploaded as a source becomes a timestamped transcript in its content, so
chunking, search, citations and the agent take it unchanged, without one long
file holding every other upload behind it on the one-thread ingest queue.

- Upload: .wav .mp3 .flac .ogg .oga with server-owned audio/* types, checked
  by their bytes (RIFF/WAVE, ID3 or an MPEG frame, fLaC, Ogg with a Vorbis
  first packet). Ogg Opus is rejected per file: "Opus support comes in a
  later update". can_hold_figures is false for audio/*, so Docling never
  reads a recording.
- worker/audio_decode/: the worker's own binary, started as
  `worker --decode-audio <in> <out>` ahead of the consumer's imports, checks
  the disk from the file's header length, streams it through pyminiaudio
  (MIT) to 16 kHz mono s16 and exits when its parent does. A malformed file
  kills the child, not the consumer. miniaudio joins pyproject, uv.lock and
  worker.spec (with its licence metadata).
- worker/ingestion/audio/, reached from parsing before Docling: readiness
  through the API's status route, the decode, Silero over the file a minute at
  a time (the API's reader, now able to stream), spans padded and joined,
  windows of at most 28 s cut at the longest pause or a run's quietest
  100 ms, each window sent to /internal/transcription/windows with retries and
  waits, windows.jsonl and progress.json as the checkpoint, a progress event
  per window, then the header and [hh:mm:ss] paragraphs, segments.json and
  transcription {language, duration_s, model} in the row's metadata.
- Yielding: after a window or a 503 waiting, once a minute has passed and a
  source that is not a recording is pending, the job raises JobYieldedError;
  the pipeline sets the row back to pending (jobs.pause_job) and queues it at
  PRIORITY_BULK. Recordings never yield to each other.
- No speech model: the job fails without retry and marks
  transcription.awaiting; choosing a speech model, by the route or an install
  with select, queues every waiting recording again. Refusals (no room, no
  speech, no audio, undecodable, memory short for 10 minutes) are
  CancelExecution(retry=False); every other retry resumes from windows.jsonl
  when the model and language intent match.
- DocumentRead gains transcription {language, progress, awaiting} for audio
  rows, progress read from progress.json; InternalEvent and the SSE frame
  gain an optional transient progress {done_s, total_s}.

Measured here: an 8.6-minute file from pending to ready in 60 s at 5.83% word
error; an hour of MP3 decodes in 3.6 s; miniaudio's linear resampler read the
44.1 kHz copy at 5.91% against scipy's 5.83%, under the proposal's 2% relative,
so it stays. Docs: documents.md gains Recordings; data-model.md, overview.md,
selection.md and packaging.md follow; the proposal records the two step 5
measurements.
2026-10-14 08:31:45 -07:00
DESKTOP-RTLN3BA\$punk
6c952c226b fix(local): answer a hung speech server with 503 and keep numbers in CJK text
Review of the speech recognition API slice.

- A transcription request past its deadline raised httpx's ReadTimeout out of
  the client, so the phrase and windows routes answered a bare 500 instead of
  503 transcription_unavailable, which the worker retries and the composer
  explains. Any HTTP error the client meets, and a 200 with no JSON text, now
  become TranscriptionUnavailableError; a timed-out request is not asked again.
- The cleanup's character-level collapse for zh, ja, yue and th counted a
  number's digits as characters, so 100000 in a Chinese transcript became 10,
  and Hmmmm became Hm. A run of digits or Latin letters now counts as one unit.
- A WAV cut mid-sample passed the check with half a sample, which made Silero
  fail (the clip then skipped the speech check) and reached the server as is.
  The check keeps whole samples only.
- The keepalive and close routes were sync handlers, so they changed the
  dictation sessions from the threadpool while the gate read them on the event
  loop. They are async now, like every other use.

Docs: the proposal's Transport table gains the windows route's `busy` wait and
a phrase's 409 transcription_needs_memory after a hold, both of which the step
documented in admission.md only; the cleanup rule names the unit; admission.md
says a timed-out request answers 503.
2026-10-14 06:39:44 -07:00
Anish Sarkar
af8eef8434
Merge pull request #2263 from AnishSarkar22/feat/homepage-new
Some checks failed
Build and Push Docker Images / image_version (push) Has been cancelled
Plugins Registry / Detect changes (push) Has been cancelled
Plugins Registry / Sign the registry (push) Has been cancelled
Build and Push Docker Images / build (./docker/sandbox, cpu, ./docker/sandbox/Dockerfile, sandbox, surfsense-sandbox, ubuntu-24.04-arm, linux/arm64, arm64, , false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./docker/sandbox, cpu, ./docker/sandbox/Dockerfile, sandbox, surfsense-sandbox, ubuntu-latest, linux/amd64, amd64, , false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (sandbox, surfsense-sandbox, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Plugins Registry / Refuse hand edits to the signed list (push) Has been cancelled
Plugins Registry / Check registry (push) Has been cancelled
feat(web): redesign the marketing site
2026-10-14 19:08:45 +05:30
Anish Sarkar
567b7275c9 feat(web): icons on the Resources menu items
Each item gets a Hugeicons mark in a tile beside its title: book for the
blog, megaphone for announcements, history for the changelog, envelope
for contact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:50:58 +05:30
Anish Sarkar
99e7bd8bcf feat(web): footer on the Golden Gate engraving
Links sit straight on the page ground over a full-width engraved scene,
replacing the closing download pitch and the card on the sky painting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:42:59 +05:30
DESKTOP-RTLN3BA\$punk
b249438b69 Merge remote-tracking branch 'origin/dev' into perf/plugins-speed
# Conflicts:
#	docs/architecture/plugin-host.md
#	plugins/remote/host/app.py
#	plugins/remote/proprietary/license_check/tests/test_license_access.py
2026-10-14 05:59:47 -07:00
DESKTOP-RTLN3BA\$punk
d908693ef6 fix(reddit, instagram): fail fast where anonymous reads are closed
Reddit refuses logged-out .json and Instagram's profiles sit behind a login.
Reddit now gives up after one rotation instead of about 2m50s and latches the
refusal, so calls answer at once with an error naming the countries tried.
Instagram no longer rotates on its require_login 401 and points to /p/ and
/reel/ URLs, which still work. Both use the shared license cache, progress and
text results.
2026-10-14 05:56:49 -07:00
DESKTOP-RTLN3BA\$punk
affc5b1325 perf(web-crawl): crawl pages concurrently, cap the HTML before extraction, bound the output
- Pages are crawled level by level, several at once, with an exact page budget.
- HTML is capped at 1 MB before trafilatura and parsed once: a 7.3 MB page
  took 17-25 s to extract, now about 1.5 s. Extractors warm up at startup.
- The tier ladder follows the HTTP status and challenge markers: a 404 fails
  in about 1 s instead of trying the browser tiers. The dynamic tier runs on
  the shared browser pool with a bounded settle.
- Output is bounded: links and contacts are capped with totals, and every
  page gets its share of the agent's text. A 3-page crawl reply went from
  1.85 MB to 49 KB.
- Fixes: GitHub and GitLab URLs filed as social, 404s and Cloudflare pages
  returned as success, loadedUrl ignoring redirects, links resolved against
  the pre-redirect URL, the oldest date on the page used as its date,
  permalink anchors in titles, and raw HTML from the stealth tier.

A 3-page docs crawl went from about 28 s to 4-8 s.
2026-10-14 05:56:49 -07:00
DESKTOP-RTLN3BA\$punk
e9da4198e4 perf(tiktok): comments and trending over TikTok's JSON API, listings on the shared browser pool
- Comments and trending read TikTok's unsigned JSON endpoints, with the
  browser as fallback: comments went from 25-96 s to about 2 s.
- Listings run on the shared browser pool, with event-driven waits, a JS
  click, no home-page warm-up, media blocked, a hedged second country, and
  concurrent targets; profile and video pages go over HTTP.
- Geo-blocked and login-bounced exits end the attempt at once.
- user_search recovers an empty first body inside the page instead of
  starting a new browser, which made it intermittent.
- Fixes: textLanguage, isAd, isPinned, isSlideshow, mentions and slideshow
  media dropped by parse_video, business accounts marked verified, withheld
  comment pages, and the profile item counting toward max_items.
2026-10-14 05:56:49 -07:00
DESKTOP-RTLN3BA\$punk
92a52c8001 perf(walmart): review pages in parallel, fail-fast exits, and ratings for variants
- Review pages after the first are fetched in parallel and deduplicated;
  start URLs and search terms run together, in order.
- Fail-fast transport with hedging; 444, 403, 5xx and a /blocked redirect are
  blocks that rotate, not pages; raw bytes and orjson instead of the lxml
  round trip.
- A failed detail returns the listing card with detailsError instead of an
  error item and an extra listing fetch.
- Fixes: variant products with 0 stars and 0 reviews, the sponsored flag and
  search provenance lost on details, duplicate reviews at page boundaries,
  reviews of other variants unlabelled, specifications and fulfillment always
  null, and lone surrogates in review text.

200 reviews went from 34-99 s to about 10-13 s. WALMART_PARALLEL=0 restores
sequential fetches; WALMART_CANONICAL_URL=1 (off) skips the /ip redirect.
2026-10-14 05:56:49 -07:00