269 commits

Author	SHA1	Message	Date
mickael	6cecec0fa7	perf: implement enhanced memory management utilities ... - Added aggressive garbage collection and memory checks. - Introduced token counting optimizations with caching and chunking. - Integrated content caching and progressive cleanup for large repos. - Adjusted memory thresholds for better handling of OOM risks.	2025-08-14 10:40:05 +02:00
mickael	d1d7abb3fc	fix: ensure memory metrics are recorded for all ingestion paths	2025-08-12 19:13:28 +02:00
mickael	8e4e37c311	feat: add prometheus memory metrics for ingestion monitoring	2025-08-12 19:01:12 +02:00
mickael	d17d3329f4	feat: add memory optimizations to prevent OOM issues	2025-08-12 18:46:44 +02:00
Nicolas Iragne	c057f6e062	feat: use gitpython for git stuff (#504) ... Co-authored-by: Iwan Burel <iwan.burel@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-08-11 17:27:42 +02:00
Mickael	867c2d904e	feat(web): add version info display (#483) ... Co-authored-by: Nicolas IRAGNE <nicoragne@hotmail.fr>	2025-08-05 18:20:13 +02:00
Nicolas Iragne	768895a07b	chore: update fastapi link (#486)	2025-08-03 10:31:49 +02:00
Nicolas Iragne	8b59bef541	fix: make cache aware of subpaths (#481)	2025-07-31 14:43:54 +02:00
Romain Courtois	2df0eb4398	fix: improved server side cleanup after ingest (#477)	2025-07-30 23:21:05 +00:00
Mickael	7886b5d84f	refactor(metrics): simplify Prometheus counters usage (#475)	2025-07-30 17:03:24 +00:00
Mickael	d061b4877a	feat(logging): implement loguru (#473)	2025-07-30 16:42:48 +00:00
Napuh	5fbb445cd8	fix: handle network errors gracefully in token count estimation (#437) ... Co-authored-by: ix-56h <n.guintini@protonmail.com> Co-authored-by: Zarial <39010759+ix-56h@users.noreply.github.com>	2025-07-30 14:49:12 +02:00
Mickael	efe5a26861	feat: serve cached digest if available (#462) ... Co-authored-by: Nicolas IRAGNE <nicoragne@hotmail.fr>	2025-07-29 18:02:57 +02:00
Zarial	932bfef85d	fix: remove logarithm conversion from the backend and correctly process max file size in kb (#464) ... Co-authored-by: Nicolas Iragne <nicoragne@hotmail.fr>	2025-07-27 16:36:47 +00:00
Mickael	414e85189f	feat: implement S3 integration for storing and retrieving digest files (#427) ... Co-authored-by: Filip Christiansen <22807962+filipchristiansen@users.noreply.github.com> Co-authored-by: Nicolas Iragne <nicoragne@hotmail.fr>	2025-07-26 16:28:17 +02:00
Zarial	7085e138a7	fix: reset pattern form when switching between include/exclude patterns (#417) ... Co-authored-by: Filip Christiansen <22807962+filipchristiansen@users.noreply.github.com>	2025-07-24 11:01:49 +02:00
Filip Christiansen	a99089ab11	refactor: consistent cloning & pattern-handling (#388) ... Co-authored-by: ix-56h <n.guintini@protonmail.com>	2025-07-23 16:35:32 +02:00
Filip Christiansen	9ceaf6cbbb	fix: Potential fix for code scanning alert no. 75: Uncontrolled data used in path expression (#421) ... Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-16 20:50:45 +02:00
Zarial	74e503fa11	fix: gitignore and gitingestignore files are now correctly processed … (#416)	2025-07-15 18:11:29 +02:00
Mickael	590e55a4d2	feat: integrate Sentry for error tracking and performance monitoring (#408)	2025-07-13 17:02:00 +02:00
Nicolas Iragne	fa2e192c05	fix: enable metrics if env var is defined instead of being "True" (#407)	2025-07-13 16:24:44 +02:00
Nicolas Iragne	1016f6ecb3	feat: implement prometheus exporter (#406) ... Co-authored-by: mickael <contact@mickael-caudrelier.fr>	2025-07-13 16:09:20 +02:00
Romain Courtois	f5b20dec44	chore: replace org name in repo urls (#396)	2025-07-09 11:30:37 +02:00
Zarial	6fbc296e7b	New swagger UI integration, improve frontend, remove repo_url from api error response, add api endpoints for ingestion and api config json, better docstring for endpoints, fix include/exclude frontend behavior, new download endpoint (#366)	2025-07-08 15:13:18 +02:00
Zarial	84450506c9	rename llm.txt to llms.txt (#390)	2025-07-06 23:19:19 +02:00
Vitalii Barenin	9dfb279ae3	Merge main into fix/windows-support	2025-07-06 15:55:23 +03:00
Vitalii Barenin	56ce0ed19e	Use 'git config core.longpaths' without level	2025-07-06 14:51:17 +03:00
Vitalii Barenin	e10bfc02f7	Check core.longpaths using --global instead of --system on Windows	2025-07-06 14:10:58 +03:00
Vitalii Barenin	d284d758c9	Check core.longpaths using --global instead of --system on Windows	2025-07-06 14:10:57 +03:00
Filip Christiansen	b683e59b5b	feat: add Tailwind CSS pipeline, tag-aware cloning & overhaul CI/CD (#352) ... Frontend * introduce Tailwind CSS (package.json, tailwind.config.js, input CSS) * build site.css on-the-fly (removed tracked artefact; added .gitignore) * new favicon/icon assets & template cleanup * split JS into modular files Docker * replace single-stage image with 3-stage build • css-builder (Node 20 alpine) → compiles Tailwind • python-builder installs project with PEP 621 metadata • runtime image copies site-packages + compiled CSS, runs as uid 1000 CI/CD * ci.yml: cache by pyproject.toml, install with `pip -e .[dev]` * new frontend job builds/archives CSS after tests * publish.yml: build CSS first, then wheel/sdist; trusted OIDC upload * tidy scorecard workflow Core library * clone.py, parser & utils now resolve tags in addition to branches/commits * fallback branch/tag discovery when `git ls-remote` fails * compat\_func.py back-ports Path.readlink / str.removesuffix for Py 3.8 Tooling & docs * add `[dev]` extra, drop requirements-dev.txt & its pre-commit fixer * refreshed CONTRIBUTING.md with Node/Tailwind instructions * updated tests for new tag logic	2025-07-06 14:10:42 +03:00
Vitalii Barenin	b8e375f71c	fix(windows): warn if Git long path support is disabled, do not fail	2025-07-06 14:00:35 +03:00
Filip Christiansen	cea0eddce8	refactor: centralize PAT validation, streamline repo checks & misc cleanup (#349) ... * refactor: centralize PAT validation, streamline repo checks & housekeeping * `.venv*` to `.gitignore` * `# type: ignore[attr-defined]` hints in `compat_typing.py` for IDE-agnostic imports * Helpful PAT string in `InvalidGitHubTokenError` for easier debugging * Bump **ruff-pre-commit** hook → `v0.12.1` * CONTRIBUTING: * Require **Python 3.9+** * Recommend signed (`-S`) commits * PAT validation now happens **only** in entry points (`utils.auth.resolve_token` for CLI/lib, `server.process_query` for Web UI) * Unified `_check_github_repo_exists` into `check_repo_exists`, replacing `curl -I` with `curl --silent --location --write-out %{http_code} -o /dev/null` * Broaden `_GITHUB_PAT_PATTERN` * `create_git_auth_header` raises `ValueError` when hostname is missing * Tests updated to expect raw HTTP-code output * Superfluous “token can be set via `GITHUB_TOKEN`” notes in docstrings * `.gitingestignore` & `.terraform` from `DEFAULT_IGNORE_PATTERNS` * Token validation inside `create_git_command` * Obsolete `test_create_git_command_invalid_token` * Adjust `test_clone.py` and `test_git_utils.py` for new status-code handling * Consolidate mocks after token-validation relocation BREAKING CHANGE: `create_git_command` no longer validates GitHub tokens; callers must ensure tokens are valid (via `validate_github_token`) before invoking lower-level git helpers. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-06 14:00:20 +03:00
BareninVitalya	6fe62cc2da	Update src/gitingest/utils/git_utils.py ... Co-authored-by: Nicolas Iragne <nicoragne@hotmail.fr>	2025-07-06 13:43:29 +03:00
Vitalii Barenin	390a591dc5	Update git_utils.py: warn on long paths issue on Windows	2025-07-06 13:43:29 +03:00
Vitalii Barenin	99aaef0c0a	Fix Windows compatibility: subprocess support and long path handling	2025-07-06 13:31:19 +03:00
Zarial	e023af309e	replace curl by httpx (#374) ... * replace curl by httpx --------- Co-authored-by: Filip Christiansen <22807962+filipchristiansen@users.noreply.github.com>	2025-07-06 11:39:54 +02:00
Nicolas Iragne	bc8cdb4594	feat(ci): build Docker Image on PRs (#382) ... * feat(ci): add GitHub Actions workflows for Docker image build and rename PyPI publish * chore: rename cli module to __main__ so it can be executed without having to build * Update .github/workflows/docker_image.yml Co-authored-by: Filip Christiansen <22807962+filipchristiansen@users.noreply.github.com>	2025-07-05 21:04:20 +02:00
Filip Christiansen	b1eebaec88	Remove helper function _normalize_pattern (and its use inside _parse_patterns) (#386)	2025-07-05 13:25:28 +02:00
Filip Christiansen	dead917060	chore(pre-commit) / refactor(templates): add ESLint pre-commit hook + refactor result.jinja (#379) ... * chore(pre-commit) / refactor(templates): add ESLint hook + refactor `result.jinja` * Add `eslint` to `pre-commit` hooks * `getFileName` + `toggleFile` moved to `utils.js` * Run linter	2025-07-04 18:32:34 +02:00
Filip Christiansen	3e83ba3936	refactor(tailwind,build,ci,docker): drop local Tailwind build & Node toolchain (#373) ... * refactor(build,ci,docker): drop local Tailwind build & Node toolchain * Remove npm ecosystem from Dependabot, workflows and Dockerfile. * Delete package.json, tailwind.config.js and generated CSS sources. * Replace compiled CSS with Tailwind CDN plus inline component layer. * Simplify CI: eliminate frontend build job; Python-only release pipeline. * Shrink Docker image to single Python stage. * Trim .gitignore and CONTRIBUTING docs accordingly. * fix max_file_size limit * fix #375	2025-07-04 01:16:06 +02:00
jpotw	38c23171a1	feat: include_submodules option (#313) ... * feat: add optional --include-submodules flag to CLI and ingestion - Adds --include-submodules CLI flag to control submodule analysis - Propagates include_submodules through ingestion, schemas, and clone logic - Updates tests to cover submodule inclusion - Adds a helper function (_checkout_partial_clone) to avoid repetition - Adds include_submodules example in README.md - Web UI for this option is not implemented for now (https://github.com/cyclotruc/gitingest/pull/313#issuecomment-3019912523) --------- Co-authored-by: Filip Christiansen <22807962+filipchristiansen@users.noreply.github.com>	2025-07-03 19:19:11 +02:00
Filip Christiansen	149c8e95e1	extract navbar svgs (#367)	2025-07-03 15:34:07 +02:00
Zarial	0fcf8a956f	fix: frontend directory tree (#363) ... Co-authored-by: ix-56h <n.guintini@protonmail.com>	2025-07-03 03:51:59 +02:00
Filip Christiansen	789d44e986	fix the bg color of the input text field (#364)	2025-07-02 23:13:51 +02:00
Filip Christiansen	d8d0e2ee29	fix lock file (#362)	2025-07-02 22:53:41 +02:00
Filip Christiansen	016817d559	feat: add Tailwind CSS pipeline, tag-aware cloning & overhaul CI/CD (#352) ... Frontend * introduce Tailwind CSS (package.json, tailwind.config.js, input CSS) * build site.css on-the-fly (removed tracked artefact; added .gitignore) * new favicon/icon assets & template cleanup * split JS into modular files Docker * replace single-stage image with 3-stage build • css-builder (Node 20 alpine) → compiles Tailwind • python-builder installs project with PEP 621 metadata • runtime image copies site-packages + compiled CSS, runs as uid 1000 CI/CD * ci.yml: cache by pyproject.toml, install with `pip -e .[dev]` * new frontend job builds/archives CSS after tests * publish.yml: build CSS first, then wheel/sdist; trusted OIDC upload * tidy scorecard workflow Core library * clone.py, parser & utils now resolve tags in addition to branches/commits * fallback branch/tag discovery when `git ls-remote` fails * compat\_func.py back-ports Path.readlink / str.removesuffix for Py 3.8 Tooling & docs * add `[dev]` extra, drop requirements-dev.txt & its pre-commit fixer * refreshed CONTRIBUTING.md with Node/Tailwind instructions * updated tests for new tag logic	2025-07-02 21:31:14 +02:00
Zarial	2b1f228ae1	feat: Refactor backend to a rest api (#346) ... * refactor: Refactor backend to a rest api and make weak changes to the front to make it works * fix: remove result bool in Success response object since it's useless, remove sensitive data from the Success Response object, make the integration unit test compatible * fix: clean query processor return types, remove deprecated fields, better handling for few errors * fix: unit tests, remove deprecated is_index from jinja templates --------- Co-authored-by: ix <n.guintini@protonmail.com>	2025-07-02 16:36:39 +02:00
Filip Christiansen	f8d397e66e	refactor: centralize PAT validation, streamline repo checks & misc cleanup (#349) ... * refactor: centralize PAT validation, streamline repo checks & housekeeping * `.venv*` to `.gitignore` * `# type: ignore[attr-defined]` hints in `compat_typing.py` for IDE-agnostic imports * Helpful PAT string in `InvalidGitHubTokenError` for easier debugging * Bump **ruff-pre-commit** hook → `v0.12.1` * CONTRIBUTING: * Require **Python 3.9+** * Recommend signed (`-S`) commits * PAT validation now happens **only** in entry points (`utils.auth.resolve_token` for CLI/lib, `server.process_query` for Web UI) * Unified `_check_github_repo_exists` into `check_repo_exists`, replacing `curl -I` with `curl --silent --location --write-out %{http_code} -o /dev/null` * Broaden `_GITHUB_PAT_PATTERN` * `create_git_auth_header` raises `ValueError` when hostname is missing * Tests updated to expect raw HTTP-code output * Superfluous “token can be set via `GITHUB_TOKEN`” notes in docstrings * `.gitingestignore` & `.terraform` from `DEFAULT_IGNORE_PATTERNS` * Token validation inside `create_git_command` * Obsolete `test_create_git_command_invalid_token` * Adjust `test_clone.py` and `test_git_utils.py` for new status-code handling * Consolidate mocks after token-validation relocation BREAKING CHANGE: `create_git_command` no longer validates GitHub tokens; callers must ensure tokens are valid (via `validate_github_token`) before invoking lower-level git helpers. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-01 14:21:13 +02:00
Filip Christiansen	25923037ea	feat(ui): add inline PAT info tooltip inside token field (#348)	2025-07-01 10:30:51 +02:00
Sendi John	fdcbc53cad	docs: Fix CLI help text accuracy (#332) ... * docs: Fix CLI help text accuracy - Add stdout documentation to --output option help text - Update default filename to 'digest.txt' consistently - Enhance docstrings with comprehensive usage examples - Improve GitHub token documentation with environment variable support - Fix inconsistencies between help text and actual CLI behavior Co-authored-by: Filip Christiansen	2025-07-01 09:18:37 +02:00