vrr/gitingest
2
0
Fork 0
mirror of https://github.com/cyclotruc/gitingest.git synced 2026-04-28 08:39:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

chore: add pre-commit config, type hints, badges, and lint codebase (#57)

Browse source 
* chore: add pre-commit config, type hints, badges, and lint codebase

- Add .pre-commit-config.yaml and pyproject.toml for Black and isort
- Add missing type hints throughout the code (Dict[...] for Python 3.8 compatibility)
- Added badges and convert existing badges to use <a><img></a> format
- Lint Markdown files
- Lint Jinja templates with djlint

* Resolve error and fix remaining type hint violations

* Fix absolute imports and mock paths in test_clone.py to resolve test failures.

* Replace deprecated 'dotenv' with 'python-dotenv' in requirements.txt to resolve installation errors.
This commit is contained in:
Filip Christiansen 2024-12-28 05:59:11 +01:00 • committed by GitHub
parent 96fb7fe4fb
commit eb73a0cc1f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
41 changed files with 1050 additions and 725 deletions
Download patch file Download diff file Expand all files Collapse all files

6
.github/workflows/unitest.yml vendored
View file

@ -15,19 +15,19 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }} - name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4 uses: actions/setup-python@v4
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install pytest pytest-asyncio pip install pytest pytest-asyncio
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install -e . pip install -e .
- name: Run tests - name: Run tests
run: | run: |
pytest pytest

1
.gitignore vendored
View file

@ -131,6 +131,7 @@ venv/
ENV/ ENV/
env.bak/ env.bak/
venv.bak/ venv.bak/
.python-version
# Spyder project settings # Spyder project settings
.spyderproject .spyderproject

78
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,78 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
# Files
- id: check-added-large-files
description: 'Prevent large files from being committed.'
args: ['--maxkb=10000']
- id: check-case-conflict
description: 'Check for files that would conflict in case-insensitive filesystems.'
- id: fix-byte-order-marker
description: 'Remove utf-8 byte order marker.'
- id: mixed-line-ending
description: 'Replace mixed line ending.'
# Links
- id: destroyed-symlinks
description: 'Detect symlinks which are changed to regular files with a content of a path which that symlink was pointing to.'
# File files for parseable syntax: python
- id: check-ast
# File and line endings
- id: end-of-file-fixer
description: 'Ensure that a file is either empty, or ends with one newline.'
- id: trailing-whitespace
description: 'Trim trailing whitespace.'
# Python
- id: check-docstring-first
description: 'Check a common error of defining a docstring after code.'
- id: requirements-txt-fixer
description: 'Sort entries in requirements.txt.'
- repo: https://github.com/MarcoGorelli/absolufy-imports
rev: v0.3.1
hooks:
- id: absolufy-imports
description: 'Automatically convert relative imports to absolute. (Use `args: [--never]` to revert.)'
- repo: https://github.com/psf/black
rev: 24.10.0
hooks:
- id: black
- repo: https://github.com/asottile/pyupgrade
rev: v3.19.1
hooks:
- id: pyupgrade
description: 'Automatically upgrade syntax for newer versions.'
args: [--py3-plus, --py36-plus, --py38-plus]
- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.10.0
hooks:
- id: python-check-blanket-noqa
description: 'Enforce that `noqa` annotations always occur with specific codes. Sample annotations: `# noqa: F401`, `# noqa: F401,W203`.'
- id: python-check-blanket-type-ignore
description: 'Enforce that `# type: ignore` annotations always occur with specific codes. Sample annotations: `# type: ignore[attr-defined]`, `# type: ignore[attr-defined, name-defined]`.'
- id: python-use-type-annotations
description: 'Enforce that python3.6+ type annotations are used instead of type comments.'
- repo: https://github.com/PyCQA/isort
rev: 5.13.2
hooks:
- id: isort
description: 'Sort imports alphabetically, and automatically separated into sections and by type.'
- repo: https://github.com/hadialqattan/pycln
rev: v2.4.0
hooks:
- id: pycln
description: 'Remove unused import statements.'
- repo: https://github.com/djlint/djLint
rev: v1.36.4
hooks:
- id: djlint-reformat-jinja

12
CODE_OF_CONDUCT.md
View file

@ -60,7 +60,7 @@ representative at an online or offline event.
Instances of abusive, harassing, or otherwise unacceptable behavior may be Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at reported to the community leaders responsible for enforcement at
romain@coderamp.io. <romain@coderamp.io>.
All complaints will be reviewed and investigated promptly and fairly. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the All community leaders are obligated to respect the privacy and security of the
@ -114,15 +114,13 @@ the community.
## Attribution ## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
version 2.0, available at version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. <https://www.contributor-covenant.org/version/2/0/code_of_conduct.html>.
Community Impact Guidelines were inspired by [Mozilla's code of conduct Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity). enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at <https://www.contributor-covenant.org/faq>. Translations are available at
https://www.contributor-covenant.org/translations. <https://www.contributor-covenant.org/translations>.

78
README.md
View file

@ -1,34 +1,56 @@
[![Image](./docs/frontpage.png "GitIngest main page")](https://gitingest.com/) [![Image](./docs/frontpage.png "GitIngest main page")](https://gitingest.com)
![License](https://img.shields.io/badge/license-MIT-blue.svg) <!-- License -->
<a href="https://github.com/cyclotruc/gitingest/blob/main/LICENSE">
<img alt="License" src="https://img.shields.io/badge/license-MIT-blue.svg" />
</a>
<!-- PyPI version -->
<a href="https://badge.fury.io/py/gitingest">
<img src="https://badge.fury.io/py/gitingest.svg" alt="PyPI version" />
</a>
<!-- Downloads -->
<a href="https://pepy.tech/project/gitingest">
<img src="https://pepy.tech/badge/gitingest" alt="Downloads" />
</a>
<!-- GitHub issues -->
<a href="https://github.com/cyclotruc/gitingest/issues">
<img src="https://img.shields.io/github/issues/cyclotruc/gitingest" alt="GitHub issues" />
</a>
<!-- Black code style -->
<a href="https://github.com/psf/black">
<img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg" />
</a>
<!-- Discord -->
<a href="https://discord.com/invite/zerRaGK9EC">
<img src="https://dcbadge.limes.pink/api/server/https://discord.com/invite/zerRaGK9EC" alt="Discord" />
</a>
# GitIngest
# GitIngest 🔍
Turn any Git repository into a prompt-friendly text ingest for LLMs. Turn any Git repository into a prompt-friendly text ingest for LLMs.
You can also replace `hub` with `ingest` in any github url to access the coresponding digest You can also replace `hub` with `ingest` in any github url to access the coresponding digest
[gitingest.com](https://gitingest.com/) [gitingest.com](https://gitingest.com)
## 🚀 Features ## 🚀 Features
- **Easy code context**: Get a text digest from a git repository URL or a directory - **Easy code context**: Get a text digest from a git repository URL or a directory
- **Smart Formatting**: Optimized output format for LLM prompts - **Smart Formatting**: Optimized output format for LLM prompts
- **Statistics about**: : - **Statistics about**:
- File and directory structure - File and directory structure
- Size of the extract - Size of the extract
- Token count - Token count
- **CLI tool**: Run it as a command (Currently on Linux only) - **CLI tool**: Run it as a command (Currently on Linux only)
- **Python package**: Import it in your code - **Python package**: Import it in your code
## 📦 Installation ## 📦 Installation
``` ``` bash
pip install gitingest pip install gitingest
``` ```
## 💡 Command Line usage ## 💡 Command Line usage
The `gitingest` command line tool allows you to analyze codebases and create a text dump of their contents. The `gitingest` command line tool allows you to analyze codebases and create a text dump of their contents.
@ -46,60 +68,62 @@ gitingest --help
This will write the digest in a text file (default `digest.txt`) in your current working directory. This will write the digest in a text file (default `digest.txt`) in your current working directory.
## 🐛 Python package usage ## 🐛 Python package usage
```python ```python
from gitingest import ingest from gitingest import ingest
summary, tree, content = ingest("path/to/directory") summary, tree, content = ingest("path/to/directory")
#or from URL # or from URL
summary, tree, content = ingest("https://github.com/cyclotruc/gitingest") summary, tree, content = ingest("https://github.com/cyclotruc/gitingest")
``` ```
By default, this won't write a file but can be enabled with the `output` argument By default, this won't write a file but can be enabled with the `output` argument
## 🛠️ Using ## 🛠️ Using
- Tailwind CSS - Frontend - Tailwind CSS - Frontend
- [FastAPI](https://github.com/fastapi/fastapi) - Backend framework - [FastAPI](https://github.com/fastapi/fastapi) - Backend framework
- [tiktoken](https://github.com/openai/tiktoken) - Token estimation - [tiktoken](https://github.com/openai/tiktoken) - Token estimation
- [apianalytics.dev](https://www.apianalytics.dev/) - Simple Analytics - [apianalytics.dev](https://www.apianalytics.dev/) - Simple Analytics
## 🌐 Self-host
## 🌐 Self-host
1. Build the image: 1. Build the image:
```
``` bash
docker build -t gitingest . docker build -t gitingest .
``` ```
2. Run the container: 2. Run the container:
```
``` bash
docker run -d --name gitingest -p 8000:8000 gitingest docker run -d --name gitingest -p 8000:8000 gitingest
``` ```
The application will be available at `http://localhost:8000` The application will be available at `http://localhost:8000`
Ensure environment variables are set before running the application or deploying it via Docker. Ensure environment variables are set before running the application or deploying it via Docker.
## ✔️ Contributing ## ✔️ Contributing
Contributions are welcome! Contributions are welcome!
Gitingest aims to be friendly for first time contributors, with a simple python and html codebase. If you need any help while working with the code, reach out to us on [discord](https://discord.com/invite/zerRaGK9EC) Gitingest aims to be friendly for first time contributors, with a simple python and html codebase. If you need any help while working with the code, reach out to us on [discord](https://discord.com/invite/zerRaGK9EC)
### Ways to contribute ### Ways to contribute
1. Provide your feedback and ideas on discord 1. Provide your feedback and ideas on discord
2. Open an Issue on github to report a bug 2. Open an Issue on github to report a bug
2. Create a Pull request 3. Create a Pull request
- Fork the repository - Fork the repository
- Make your changes and test them locally - Make your changes and test them locally
- Open a pull request for review and feedback - Open a pull request for review and feedback
### 🔧 Local dev ### 🔧 Local dev
#### Environment Configuration #### Environment Configuration
- **`ALLOWED_HOSTS`**: Specify allowed hostnames for the application. Default: `"gitingest.com,*.gitingest.com,gitdigest.dev,localhost"`. - **`ALLOWED_HOSTS`**: Specify allowed hostnames for the application. Default: `"gitingest.com,*.gitingest.com,gitdigest.dev,localhost"`.
You can configure the application using the following environment variables: You can configure the application using the following environment variables:
@ -108,23 +132,25 @@ ALLOWED_HOSTS="gitingest.local,localhost"
``` ```
#### Run locally #### Run locally
1. Clone the repository
1. Clone the repository
```bash ```bash
git clone https://github.com/cyclotruc/gitingest.git git clone https://github.com/cyclotruc/gitingest.git
cd gitingest cd gitingest
``` ```
2. Install dependencies 2. Install dependencies
```bash ```bash
pip install -r requirements.txt pip install -r requirements.txt
``` ```
3. Run the application: 3. Run the application:
```bash ```bash
cd src cd src
uvicorn main:app --reload uvicorn main:app --reload
``` ```
The frontend will be available at `localhost:8000` The frontend will be available at `localhost:8000`

2
SECURITY.md
View file

@ -2,4 +2,4 @@
## Reporting a Vulnerability ## Reporting a Vulnerability
If you have discovered a vulnerability inside the project, report it privately at romain@coderamp.io. This way the maintainer can work on a proper fix without disclosing the problem to the public before it has been solved. If you have discovered a vulnerability inside the project, report it privately at <romain@coderamp.io>. This way the maintainer can work on a proper fix without disclosing the problem to the public before it has been solved.

17
pyproject.toml Normal file
View file

@ -0,0 +1,17 @@
[tool.pylint.format]
max-line-length = 119
[tool.pycln]
all = true
[tool.isort]
profile = "black"
line_length = 119
remove_redundant_aliases = true
float_to_top = true
order_by_type = true
filter_files = true
[tool.black]
line-length = 119
skip-string-normalization = true

3
pytest.ini
View file

@ -3,7 +3,6 @@ pythonpath = src
testpaths = src/gitingest/tests testpaths = src/gitingest/tests
asyncio_mode = auto asyncio_mode = auto
python_files = test_*.py python_files = test_*.py
python_classes = Test* python_classes = Test*
python_functions = test_* python_functions = test_*

15
requirements.txt
View file

@ -1,8 +1,13 @@
fastapi[standard] black
uvicorn click>=8.0.0
djlint
fastapi-analytics fastapi-analytics
slowapi fastapi[standard]
tiktoken pre-commit
pytest pytest
pytest-asyncio pytest-asyncio
click>=8.0.0 python-dotenv
slowapi
starlette
tiktoken
uvicorn

4
setup.py
View file

@ -1,4 +1,4 @@
from setuptools import setup, find_packages from setuptools import find_packages, setup
setup( setup(
name="gitingest", name="gitingest",
@ -28,4 +28,4 @@ setup(
"License :: OSI Approved :: MIT License", "License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3", "Programming Language :: Python :: 3",
], ],
) )

2
src/config.py
View file

@ -1,4 +1,4 @@
MAX_DISPLAY_SIZE = 300000 MAX_DISPLAY_SIZE = 300_000
TMP_BASE_PATH = "../tmp" TMP_BASE_PATH = "../tmp"
EXAMPLE_REPOS = [ EXAMPLE_REPOS = [

10
src/gitingest/__init__.py
View file

@ -1,6 +1,6 @@
from .ingest_from_query import ingest_from_query from gitingest.clone import clone_repo
from .clone import clone_repo from gitingest.ingest import ingest
from .parse_query import parse_query from gitingest.ingest_from_query import ingest_from_query
from .ingest import ingest from gitingest.parse_query import parse_query
__all__ = ["ingest_from_query", "clone_repo", "parse_query", "ingest"] __all__ = ["ingest_from_query", "clone_repo", "parse_query", "ingest"]

24
src/gitingest/cli.py
View file

@ -1,10 +1,12 @@
import os import os
import pathlib from typing import Optional, Tuple
import click import click
from gitingest.ignore_patterns import DEFAULT_IGNORE_PATTERNS
from gitingest.ingest import ingest from gitingest.ingest import ingest
from gitingest.ingest_from_query import MAX_FILE_SIZE from gitingest.ingest_from_query import MAX_FILE_SIZE
from gitingest.parse_query import DEFAULT_IGNORE_PATTERNS
def normalize_pattern(pattern: str) -> str: def normalize_pattern(pattern: str) -> str:
pattern = pattern.strip() pattern = pattern.strip()
@ -13,30 +15,38 @@ def normalize_pattern(pattern: str) -> str:
pattern += "*" pattern += "*"
return pattern return pattern
@click.command() @click.command()
@click.argument('source', type=str, required=True) @click.argument('source', type=str, required=True)
@click.option('--output', '-o', default=None, help='Output file path (default: <repo_name>.txt in current directory)') @click.option('--output', '-o', default=None, help='Output file path (default: <repo_name>.txt in current directory)')
@click.option('--max-size', '-s', default=MAX_FILE_SIZE, help='Maximum file size to process in bytes') @click.option('--max-size', '-s', default=MAX_FILE_SIZE, help='Maximum file size to process in bytes')
@click.option('--exclude-pattern', '-e', multiple=True, help='Patterns to exclude') @click.option('--exclude-pattern', '-e', multiple=True, help='Patterns to exclude')
@click.option('--include-pattern', '-i', multiple=True, help='Patterns to include') @click.option('--include-pattern', '-i', multiple=True, help='Patterns to include')
def main(source, output, max_size, exclude_pattern, include_pattern): def main(
source: str,
output: Optional[str],
max_size: int,
exclude_pattern: Tuple[str, ...],
include_pattern: Tuple[str, ...],
) -> None:
"""Analyze a directory and create a text dump of its contents.""" """Analyze a directory and create a text dump of its contents."""
try: try:
# Combine default and custom ignore patterns # Combine default and custom ignore patterns
exclude_patterns = list(exclude_pattern) exclude_patterns = list(exclude_pattern)
include_patterns = list(set(include_pattern)) include_patterns = list(set(include_pattern))
if not output: if not output:
output = "digest.txt" output = "digest.txt"
summary, tree, content = ingest(source, max_size, include_patterns, exclude_patterns, output=output) summary, tree, content = ingest(source, max_size, include_patterns, exclude_patterns, output=output)
click.echo(f"Analysis complete! Output written to: {output}") click.echo(f"Analysis complete! Output written to: {output}")
click.echo("\nSummary:") click.echo("\nSummary:")
click.echo(summary) click.echo(summary)
except Exception as e: except Exception as e:
click.echo(f"Error: {str(e)}", err=True) click.echo(f"Error: {str(e)}", err=True)
raise click.Abort() raise click.Abort()
if __name__ == '__main__': if __name__ == '__main__':
main() main()

22
src/gitingest/clone.py
View file

@ -1,10 +1,11 @@
import asyncio import asyncio
from typing import Tuple from typing import Any, Dict, Tuple
from gitingest.utils import async_timeout from gitingest.utils import async_timeout
CLONE_TIMEOUT = 20 CLONE_TIMEOUT = 20
async def check_repo_exists(url: str) -> bool: async def check_repo_exists(url: str) -> bool:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
"curl", "curl",
@ -20,14 +21,15 @@ async def check_repo_exists(url: str) -> bool:
stdout_str = stdout.decode() stdout_str = stdout.decode()
return "HTTP/1.1 404" not in stdout_str and "HTTP/2 404" not in stdout_str return "HTTP/1.1 404" not in stdout_str and "HTTP/2 404" not in stdout_str
@async_timeout(CLONE_TIMEOUT) @async_timeout(CLONE_TIMEOUT)
async def clone_repo(query: dict) -> str: async def clone_repo(query: Dict[str, Any]) -> Tuple[bytes, bytes]:
if not await check_repo_exists(query['url']): if not await check_repo_exists(query['url']):
raise ValueError("Repository not found, make sure it is public") raise ValueError("Repository not found, make sure it is public")
if query['commit']: if query['commit']:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
"git", "git",
"clone", "clone",
"--single-branch", "--single-branch",
query['url'], query['url'],
@ -36,21 +38,21 @@ async def clone_repo(query: dict) -> str:
stderr=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
) )
stdout, stderr = await proc.communicate() stdout, stderr = await proc.communicate()
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
"git", "git",
"-C", "-C",
query['local_path'], query['local_path'],
"checkout", "checkout",
query['branch'], query['branch'],
stdout=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
) )
stdout, stderr = await proc.communicate() stdout, stderr = await proc.communicate()
elif query['branch'] != 'main' and query['branch'] != 'master' and query['branch']: elif query['branch'] != 'main' and query['branch'] != 'master' and query['branch']:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
"git", "git",
"clone", "clone",
"--depth=1", "--depth=1",
"--single-branch", "--single-branch",
"--branch", "--branch",
@ -71,7 +73,7 @@ async def clone_repo(query: dict) -> str:
stdout=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
) )
stdout, stderr = await proc.communicate() stdout, stderr = await proc.communicate()
return stdout, stderr return stdout, stderr

102
src/gitingest/ignore_patterns.py Normal file
View file

@ -0,0 +1,102 @@
from typing import List
DEFAULT_IGNORE_PATTERNS: List[str] = [
# Python
'*.pyc',
'*.pyo',
'*.pyd',
'__pycache__',
'.pytest_cache',
'.coverage',
'.tox',
'.nox',
'.mypy_cache',
'.ruff_cache',
'.hypothesis',
'poetry.lock',
'Pipfile.lock',
# JavaScript/Node
'node_modules',
'bower_components',
'package-lock.json',
'yarn.lock',
'.npm',
'.yarn',
'.pnpm-store',
# Version control
'.git',
'.svn',
'.hg',
'.gitignore',
'.gitattributes',
'.gitmodules',
# Images and media
'*.svg',
'*.png',
'*.jpg',
'*.jpeg',
'*.gif',
'*.ico',
'*.pdf',
'*.mov',
'*.mp4',
'*.mp3',
'*.wav',
# Virtual environments
'venv',
'.venv',
'env',
'.env',
'virtualenv',
# IDEs and editors
'.idea',
'.vscode',
'.vs',
'*.swp',
'*.swo',
'*.swn',
'.settings',
'.project',
'.classpath',
'*.sublime-*',
# Temporary and cache files
'*.log',
'*.bak',
'*.swp',
'*.tmp',
'*.temp',
'.cache',
'.sass-cache',
'.eslintcache',
'.DS_Store',
'Thumbs.db',
'desktop.ini',
# Build directories and artifacts
'build',
'dist',
'target',
'out',
'*.egg-info',
'*.egg',
'*.whl',
'*.so',
'*.dylib',
'*.dll',
'*.class',
# Documentation
'site-packages',
'.docusaurus',
'.next',
'.nuxt',
# Other common patterns
## Minified files
'*.min.js',
'*.min.css',
## Source maps
'*.map',
## Terraform
'.terraform',
'*.tfstate*',
## Dependencies in various languages
'vendor/',
]

37
src/gitingest/ingest.py
View file

@ -1,18 +1,36 @@
import asyncio import asyncio
import inspect
import shutil import shutil
from typing import Union, List
from pathlib import Path from pathlib import Path
from typing import List, Optional, Tuple, Union
from .ingest_from_query import ingest_from_query from gitingest.clone import clone_repo
from .clone import clone_repo from gitingest.ingest_from_query import ingest_from_query
from .parse_query import parse_query from gitingest.parse_query import parse_query
def ingest(source: str, max_file_size: int = 10 * 1024 * 1024, include_patterns: Union[List[str], str] = None, exclude_patterns: Union[List[str], str] = None, output: str = None) -> str:
def ingest(
source: str,
max_file_size: int = 10 * 1024 * 1024,
include_patterns: Union[List[str], str, None] = None,
exclude_patterns: Union[List[str], str, None] = None,
output: Optional[str] = None,
) -> Tuple[str, str, str]:
try: try:
query = parse_query(source, max_file_size, False, include_patterns, exclude_patterns) query = parse_query(
source=source,
max_file_size=max_file_size,
from_web=False,
include_patterns=include_patterns,
ignore_patterns=exclude_patterns,
)
if query['url']: if query['url']:
asyncio.run(clone_repo(query)) clone_result = clone_repo(query)
if inspect.iscoroutine(clone_result):
asyncio.run(clone_result)
else:
raise TypeError("clone_repo did not return a coroutine as expected.")
summary, tree, content = ingest_from_query(query) summary, tree, content = ingest_from_query(query)
if output: if output:
@ -20,9 +38,10 @@ def ingest(source: str, max_file_size: int = 10 * 1024 * 1024, include_patterns:
f.write(tree + "\n" + content) f.write(tree + "\n" + content)
return summary, tree, content return summary, tree, content
finally: finally:
# Clean up the temporary directory if it was created # Clean up the temporary directory if it was created
if query['url']: if query['url']:
# Get parent directory two levels up from local_path (../tmp) # Get parent directory two levels up from local_path (../tmp)
cleanup_path = str(Path(query['local_path']).parents[1]) cleanup_path = str(Path(query['local_path']).parents[1])
shutil.rmtree(cleanup_path, ignore_errors=True) shutil.rmtree(cleanup_path, ignore_errors=True)

136
src/gitingest/ingest_from_query.py
View file

@ -1,13 +1,13 @@
import os import os
from fnmatch import fnmatch from fnmatch import fnmatch
from typing import Dict, List, Union from typing import Any, Dict, List, Optional, Set, Tuple
import tiktoken import tiktoken
MAX_FILE_SIZE = 10 * 1024 * 1024 # 10 MB
MAX_FILE_SIZE = 10 * 1024 * 1024 # 10MB
MAX_DIRECTORY_DEPTH = 20 # Maximum depth of directory traversal MAX_DIRECTORY_DEPTH = 20 # Maximum depth of directory traversal
MAX_FILES = 10000 # Maximum number of files to process MAX_FILES = 10_000 # Maximum number of files to process
MAX_TOTAL_SIZE_BYTES = 500 * 1024 * 1024 # 500MB MAX_TOTAL_SIZE_BYTES = 500 * 1024 * 1024 # 500 MB
def should_include(path: str, base_path: str, include_patterns: List[str]) -> bool: def should_include(path: str, base_path: str, include_patterns: List[str]) -> bool:
@ -18,6 +18,7 @@ def should_include(path: str, base_path: str, include_patterns: List[str]) -> bo
include = True include = True
return include return include
def should_exclude(path: str, base_path: str, ignore_patterns: List[str]) -> bool: def should_exclude(path: str, base_path: str, ignore_patterns: List[str]) -> bool:
rel_path = path.replace(base_path, "").lstrip(os.sep) rel_path = path.replace(base_path, "").lstrip(os.sep)
for pattern in ignore_patterns: for pattern in ignore_patterns:
@ -27,6 +28,7 @@ def should_exclude(path: str, base_path: str, ignore_patterns: List[str]) -> boo
return True return True
return False return False
def is_safe_symlink(symlink_path: str, base_path: str) -> bool: def is_safe_symlink(symlink_path: str, base_path: str) -> bool:
"""Check if a symlink points to a location within the base directory.""" """Check if a symlink points to a location within the base directory."""
try: try:
@ -37,23 +39,32 @@ def is_safe_symlink(symlink_path: str, base_path: str) -> bool:
# If there's any error resolving the paths, consider it unsafe # If there's any error resolving the paths, consider it unsafe
return False return False
def is_text_file(file_path: str) -> bool: def is_text_file(file_path: str) -> bool:
"""Determines if a file is likely a text file based on its content.""" """Determines if a file is likely a text file based on its content."""
try: try:
with open(file_path, 'rb') as file: with open(file_path, 'rb') as file:
chunk = file.read(1024) chunk = file.read(1024)
return not bool(chunk.translate(None, bytes([7, 8, 9, 10, 12, 13, 27] + list(range(0x20, 0x100))))) return not bool(chunk.translate(None, bytes([7, 8, 9, 10, 12, 13, 27] + list(range(0x20, 0x100)))))
except IOError: except OSError:
return False return False
def read_file_content(file_path: str) -> str: def read_file_content(file_path: str) -> str:
try: try:
with open(file_path, 'r', encoding='utf-8', errors='ignore') as f: with open(file_path, encoding='utf-8', errors='ignore') as f:
return f.read() return f.read()
except Exception as e: except Exception as e:
return f"Error reading file: {str(e)}" return f"Error reading file: {str(e)}"
def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int = 0, stats: Dict = None) -> Dict:
def scan_directory(
path: str,
query: Dict[str, Any],
seen_paths: Optional[Set[str]] = None,
depth: int = 0,
stats: Optional[Dict[str, int]] = None,
) -> Optional[Dict[str, Any]]:
"""Recursively analyzes a directory and its contents with safety limits.""" """Recursively analyzes a directory and its contents with safety limits."""
if seen_paths is None: if seen_paths is None:
seen_paths = set() seen_paths = set()
@ -76,6 +87,7 @@ def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int =
if real_path in seen_paths: if real_path in seen_paths:
print(f"Skipping already visited path: {path}") print(f"Skipping already visited path: {path}")
return None return None
seen_paths.add(real_path) seen_paths.add(real_path)
result = { result = {
@ -86,7 +98,7 @@ def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int =
"file_count": 0, "file_count": 0,
"dir_count": 0, "dir_count": 0,
"path": path, "path": path,
"ignore_content": False "ignore_content": False,
} }
ignore_patterns = query['ignore_patterns'] ignore_patterns = query['ignore_patterns']
@ -137,14 +149,20 @@ def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int =
"type": "file", "type": "file",
"size": file_size, "size": file_size,
"content": content, "content": content,
"path": item_path "path": item_path,
} }
result["children"].append(child) result["children"].append(child)
result["size"] += file_size result["size"] += file_size
result["file_count"] += 1 result["file_count"] += 1
elif os.path.isdir(real_path): elif os.path.isdir(real_path):
subdir = scan_directory(real_path, query, seen_paths, depth + 1, stats) subdir = scan_directory(
path=real_path,
query=query,
seen_paths=seen_paths,
depth=depth + 1,
stats=stats,
)
if subdir and (not include_patterns or subdir["file_count"] > 0): if subdir and (not include_patterns or subdir["file_count"] > 0):
subdir["name"] = item subdir["name"] = item
subdir["path"] = item_path subdir["path"] = item_path
@ -175,14 +193,20 @@ def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int =
"type": "file", "type": "file",
"size": file_size, "size": file_size,
"content": content, "content": content,
"path": item_path "path": item_path,
} }
result["children"].append(child) result["children"].append(child)
result["size"] += file_size result["size"] += file_size
result["file_count"] += 1 result["file_count"] += 1
elif os.path.isdir(item_path): elif os.path.isdir(item_path):
subdir = scan_directory(item_path, query, seen_paths, depth + 1, stats) subdir = scan_directory(
path=item_path,
query=query,
seen_paths=seen_paths,
depth=depth + 1,
stats=stats,
)
if subdir and (not include_patterns or subdir["file_count"] > 0): if subdir and (not include_patterns or subdir["file_count"] > 0):
result["children"].append(subdir) result["children"].append(subdir)
result["size"] += subdir["size"] result["size"] += subdir["size"]
@ -194,7 +218,13 @@ def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int =
return result return result
def extract_files_content(query: dict, node: Dict, max_file_size: int, files: List = None) -> List[Dict]:
def extract_files_content(
query: Dict[str, Any],
node: Dict[str, Any],
max_file_size: int,
files: Optional[List[Dict[str, Any]]] = None,
) -> List[Dict[str, Any]]:
"""Recursively collects all text files with their contents.""" """Recursively collects all text files with their contents."""
if files is None: if files is None:
files = [] files = []
@ -204,17 +234,21 @@ def extract_files_content(query: dict, node: Dict, max_file_size: int, files: Li
if node["size"] > max_file_size: if node["size"] > max_file_size:
content = None content = None
files.append({ files.append(
"path": node["path"].replace(query['local_path'], ""), {
"content": content, "path": node["path"].replace(query['local_path'], ""),
"size": node["size"] "content": content,
}) "size": node["size"],
},
)
elif node["type"] == "directory": elif node["type"] == "directory":
for child in node["children"]: for child in node["children"]:
extract_files_content(query, child, max_file_size, files) extract_files_content(query=query, node=child, max_file_size=max_file_size, files=files)
return files return files
def create_file_content_string(files: List[Dict]) -> str:
def create_file_content_string(files: List[Dict[str, Any]]) -> str:
"""Creates a formatted string of file contents with separators.""" """Creates a formatted string of file contents with separators."""
output = "" output = ""
separator = "=" * 48 + "\n" separator = "=" * 48 + "\n"
@ -223,6 +257,7 @@ def create_file_content_string(files: List[Dict]) -> str:
for file in files: for file in files:
if not file['content']: if not file['content']:
continue continue
if file['path'].lower() == '/readme.md': if file['path'].lower() == '/readme.md':
output += separator output += separator
output += f"File: {file['path']}\n" output += f"File: {file['path']}\n"
@ -234,6 +269,7 @@ def create_file_content_string(files: List[Dict]) -> str:
for file in files: for file in files:
if not file['content'] or file['path'].lower() == '/readme.md': if not file['content'] or file['path'].lower() == '/readme.md':
continue continue
output += separator output += separator
output += f"File: {file['path']}\n" output += f"File: {file['path']}\n"
output += separator output += separator
@ -241,12 +277,14 @@ def create_file_content_string(files: List[Dict]) -> str:
return output return output
def create_summary_string(query: dict, nodes: Dict, files: List[Dict]) -> str:
def create_summary_string(query: Dict[str, Any], nodes: Dict[str, Any], files: List[Dict[str, Any]]) -> str:
"""Creates a summary string with file counts and content size.""" """Creates a summary string with file counts and content size."""
if "user_name" in query: if "user_name" in query:
summary = f"Repository: {query['user_name']}/{query['repo_name']}\n" summary = f"Repository: {query['user_name']}/{query['repo_name']}\n"
else: else:
summary = f"Repository: {query['slug']}\n" summary = f"Repository: {query['slug']}\n"
summary += f"Files analyzed: {nodes['file_count']}\n" summary += f"Files analyzed: {nodes['file_count']}\n"
if 'subpath' in query and query['subpath'] != '/': if 'subpath' in query and query['subpath'] != '/':
@ -255,11 +293,19 @@ def create_summary_string(query: dict, nodes: Dict, files: List[Dict]) -> str:
summary += f"Commit: {query['commit']}\n" summary += f"Commit: {query['commit']}\n"
elif 'branch' in query and query['branch'] != 'main' and query['branch'] != 'master' and query['branch']: elif 'branch' in query and query['branch'] != 'main' and query['branch'] != 'master' and query['branch']:
summary += f"Branch: {query['branch']}\n" summary += f"Branch: {query['branch']}\n"
return summary return summary
def create_tree_structure(query: dict, node: Dict, prefix: str = "", is_last: bool = True) -> str:
def create_tree_structure(
query: Dict[str, Any],
node: Dict[str, Any],
prefix: str = "",
is_last: bool = True,
) -> str:
"""Creates a tree-like string representation of the file structure.""" """Creates a tree-like string representation of the file structure."""
tree = "" tree = ""
if not node["name"]: if not node["name"]:
node["name"] = query['slug'] node["name"] = query['slug']
@ -267,6 +313,7 @@ def create_tree_structure(query: dict, node: Dict, prefix: str = "", is_last: bo
current_prefix = "└── " if is_last else "├── " current_prefix = "└── " if is_last else "├── "
name = node["name"] + "/" if node["type"] == "directory" else node["name"] name = node["name"] + "/" if node["type"] == "directory" else node["name"]
tree += prefix + current_prefix + name + "\n" tree += prefix + current_prefix + name + "\n"
if node["type"] == "directory": if node["type"] == "directory":
# Adjust prefix only if we added a node name # Adjust prefix only if we added a node name
new_prefix = prefix + (" " if is_last else "│ ") if node["name"] else prefix new_prefix = prefix + (" " if is_last else "│ ") if node["name"] else prefix
@ -276,25 +323,29 @@ def create_tree_structure(query: dict, node: Dict, prefix: str = "", is_last: bo
return tree return tree
def generate_token_string(context_string: str) -> str:
def generate_token_string(context_string: str) -> Optional[str]:
"""Returns the number of tokens in a text string.""" """Returns the number of tokens in a text string."""
formatted_tokens = "" formatted_tokens = ""
try: try:
encoding = tiktoken.get_encoding("cl100k_base", ) encoding = tiktoken.get_encoding("cl100k_base")
total_tokens = len(encoding.encode(context_string, disallowed_special=())) total_tokens = len(encoding.encode(context_string, disallowed_special=()))
except Exception as e: except Exception as e:
print(e) print(e)
return None return None
if total_tokens > 1000000:
formatted_tokens = f"{total_tokens/1000000:.1f}M" if total_tokens > 1_000_000:
elif total_tokens > 1000: formatted_tokens = f"{total_tokens / 1_000_000:.1f}M"
formatted_tokens = f"{total_tokens/1000:.1f}k" elif total_tokens > 1_000:
formatted_tokens = f"{total_tokens / 1_000:.1f}k"
else: else:
formatted_tokens = f"{total_tokens}" formatted_tokens = f"{total_tokens}"
return formatted_tokens return formatted_tokens
def ingest_single_file(path: str, query: dict) -> Dict:
def ingest_single_file(path: str, query: Dict[str, Any]) -> Tuple[str, str, str]:
if not os.path.isfile(path): if not os.path.isfile(path):
raise ValueError(f"Path {path} is not a file") raise ValueError(f"Path {path} is not a file")
@ -310,7 +361,7 @@ def ingest_single_file(path: str, query: dict) -> Dict:
file_info = { file_info = {
"path": path.replace(query['local_path'], ""), "path": path.replace(query['local_path'], ""),
"content": content, "content": content,
"size": file_size "size": file_size,
} }
summary = ( summary = (
@ -326,11 +377,15 @@ def ingest_single_file(path: str, query: dict) -> Dict:
formatted_tokens = generate_token_string(files_content) formatted_tokens = generate_token_string(files_content)
if formatted_tokens: if formatted_tokens:
summary += f"\nEstimated tokens: {formatted_tokens}" summary += f"\nEstimated tokens: {formatted_tokens}"
return (summary, tree, files_content)
def ingest_directory(path: str, query: dict) -> Dict: return summary, tree, files_content
nodes = scan_directory(path, query)
files = extract_files_content(query, nodes, query['max_file_size'])
def ingest_directory(path: str, query: Dict[str, Any]) -> Tuple[str, str, str]:
nodes = scan_directory(path=path, query=query)
if not nodes:
raise ValueError(f"No files found in {path}")
files = extract_files_content(query=query, node=nodes, max_file_size=query['max_file_size'])
summary = create_summary_string(query, nodes, files) summary = create_summary_string(query, nodes, files)
tree = "Directory structure:\n" + create_tree_structure(query, nodes) tree = "Directory structure:\n" + create_tree_structure(query, nodes)
files_content = create_file_content_string(files) files_content = create_file_content_string(files)
@ -338,9 +393,11 @@ def ingest_directory(path: str, query: dict) -> Dict:
formatted_tokens = generate_token_string(tree + files_content) formatted_tokens = generate_token_string(tree + files_content)
if formatted_tokens: if formatted_tokens:
summary += f"\nEstimated tokens: {formatted_tokens}" summary += f"\nEstimated tokens: {formatted_tokens}"
return (summary, tree, files_content)
def ingest_from_query(query: dict) -> Dict: return summary, tree, files_content
def ingest_from_query(query: Dict[str, Any]) -> Tuple[str, str, str]:
"""Main entry point for analyzing a codebase directory or single file.""" """Main entry point for analyzing a codebase directory or single file."""
path = f"{query['local_path']}{query['subpath']}" path = f"{query['local_path']}{query['subpath']}"
if not os.path.exists(path): if not os.path.exists(path):
@ -348,6 +405,5 @@ def ingest_from_query(query: dict) -> Dict:
if query.get('type') == 'blob': if query.get('type') == 'blob':
return ingest_single_file(path, query) return ingest_single_file(path, query)
else:
return ingest_directory(path, query)
return ingest_directory(path, query)

95
src/gitingest/parse_query.py
View file

@ -1,55 +1,13 @@
from typing import List, Union import os
import uuid import uuid
import os from typing import Any, Dict, List, Optional, Union
from gitingest.ignore_patterns import DEFAULT_IGNORE_PATTERNS
DEFAULT_IGNORE_PATTERNS = [
# Python
'*.pyc', '*.pyo', '*.pyd', '__pycache__', '.pytest_cache', '.coverage',
'.tox', '.nox', '.mypy_cache', '.ruff_cache', '.hypothesis',
'poetry.lock', 'Pipfile.lock',
# JavaScript/Node
'node_modules', 'bower_components', 'package-lock.json', 'yarn.lock',
'.npm', '.yarn', '.pnpm-store',
# Version control
'.git', '.svn', '.hg', '.gitignore', '.gitattributes', '.gitmodules',
# Images and media
'*.svg', '*.png', '*.jpg', '*.jpeg', '*.gif', '*.ico', '*.pdf',
'*.mov', '*.mp4', '*.mp3', '*.wav',
# Virtual environments
'venv', '.venv', 'env', '.env', 'virtualenv',
# IDEs and editors
'.idea', '.vscode', '.vs', '*.swp', '*.swo', '*.swn',
'.settings', '.project', '.classpath', '*.sublime-*',
# Temporary and cache files
'*.log', '*.bak', '*.swp', '*.tmp', '*.temp',
'.cache', '.sass-cache', '.eslintcache',
'.DS_Store', 'Thumbs.db', 'desktop.ini',
# Build directories and artifacts
'build', 'dist', 'target', 'out',
'*.egg-info', '*.egg', '*.whl',
'*.so', '*.dylib', '*.dll', '*.class',
# Documentation
'site-packages', '.docusaurus', '.next', '.nuxt',
# Other common patterns
'*.min.js', '*.min.css', # Minified files
'*.map', # Source maps
'.terraform', '*.tfstate*', # Terraform
'vendor/', # Dependencies in various languages
]
TMP_BASE_PATH = "../tmp" TMP_BASE_PATH = "../tmp"
def parse_url(url: str) -> dict:
def parse_url(url: str) -> Dict[str, Any]:
parsed = { parsed = {
"user_name": None, "user_name": None,
"repo_name": None, "repo_name": None,
@ -62,22 +20,22 @@ def parse_url(url: str) -> dict:
"slug": None, "slug": None,
"id": None, "id": None,
} }
url = url.split(" ")[0] url = url.split(" ")[0]
if not url.startswith('https://'): if not url.startswith('https://'):
url = 'https://' + url url = 'https://' + url
# Extract domain and path # Extract domain and path
url_parts = url.split('/') url_parts = url.split('/')
domain = url_parts[2] domain = url_parts[2]
path_parts = url_parts[3:] path_parts = url_parts[3:]
if len(path_parts) < 2: if len(path_parts) < 2:
raise ValueError("Invalid repository URL. Please provide a valid Git repository URL.") raise ValueError("Invalid repository URL. Please provide a valid Git repository URL.")
parsed["user_name"] = path_parts[0] parsed["user_name"] = path_parts[0]
parsed["repo_name"] = path_parts[1] parsed["repo_name"] = path_parts[1]
# Keep original URL format # Keep original URL format
parsed["url"] = f"https://{domain}/{parsed['user_name']}/{parsed['repo_name']}" parsed["url"] = f"https://{domain}/{parsed['user_name']}/{parsed['repo_name']}"
parsed['slug'] = f"{parsed['user_name']}-{parsed['repo_name']}" parsed['slug'] = f"{parsed['user_name']}-{parsed['repo_name']}"
@ -89,10 +47,12 @@ def parse_url(url: str) -> dict:
parsed["branch"] = path_parts[3] parsed["branch"] = path_parts[3]
if len(parsed['branch']) == 40 and all(c in '0123456789abcdefABCDEF' for c in parsed['branch']): if len(parsed['branch']) == 40 and all(c in '0123456789abcdefABCDEF' for c in parsed['branch']):
parsed["commit"] = parsed['branch'] parsed["commit"] = parsed['branch']
parsed["subpath"] = "/" + "/".join(path_parts[4:]) parsed["subpath"] = "/" + "/".join(path_parts[4:])
return parsed return parsed
def normalize_pattern(pattern: str) -> str: def normalize_pattern(pattern: str) -> str:
pattern = pattern.strip() pattern = pattern.strip()
pattern = pattern.lstrip(os.sep) pattern = pattern.lstrip(os.sep)
@ -100,16 +60,21 @@ def normalize_pattern(pattern: str) -> str:
pattern += "*" pattern += "*"
return pattern return pattern
def parse_patterns(pattern: Union[List[str], str]) -> List[str]: def parse_patterns(pattern: Union[List[str], str]) -> List[str]:
if isinstance(pattern, list): if isinstance(pattern, list):
pattern = ",".join(pattern) pattern = ",".join(pattern)
for p in pattern.split(","): for p in pattern.split(","):
if not all(c.isalnum() or c in "-_./+*" for c in p.strip()): if not all(c.isalnum() or c in "-_./+*" for c in p.strip()):
raise ValueError(f"Pattern '{p}' contains invalid characters. Only alphanumeric characters, dash (-), underscore (_), dot (.), forward slash (/), plus (+), and asterisk (*) are allowed.") raise ValueError(
f"Pattern '{p}' contains invalid characters. Only alphanumeric characters, dash (-), "
"underscore (_), dot (.), forward slash (/), plus (+), and asterisk (*) are allowed."
)
patterns = [normalize_pattern(p) for p in pattern.split(",")] patterns = [normalize_pattern(p) for p in pattern.split(",")]
return patterns return patterns
def override_ignore_patterns(ignore_patterns: List[str], include_patterns: List[str]) -> List[str]: def override_ignore_patterns(ignore_patterns: List[str], include_patterns: List[str]) -> List[str]:
for pattern in include_patterns: for pattern in include_patterns:
if pattern in ignore_patterns: if pattern in ignore_patterns:
@ -117,8 +82,7 @@ def override_ignore_patterns(ignore_patterns: List[str], include_patterns: List[
return ignore_patterns return ignore_patterns
def parse_path(path: str) -> dict: def parse_path(path: str) -> Dict[str, Any]:
query = { query = {
"local_path": os.path.abspath(path), "local_path": os.path.abspath(path),
"slug": os.path.basename(os.path.dirname(path)) + "/" + os.path.basename(path), "slug": os.path.basename(os.path.dirname(path)) + "/" + os.path.basename(path),
@ -128,7 +92,14 @@ def parse_path(path: str) -> dict:
} }
return query return query
def parse_query(source: str, max_file_size: int, from_web: bool, include_patterns: Union[List[str], str] = None, ignore_patterns: Union[List[str], str] = None) -> dict:
def parse_query(
source: str,
max_file_size: int,
from_web: bool,
include_patterns: Optional[Union[List[str], str]] = None,
ignore_patterns: Optional[Union[List[str], str]] = None,
) -> Dict[str, Any]:
if from_web: if from_web:
query = parse_url(source) query = parse_url(source)
else: else:
@ -136,21 +107,21 @@ def parse_query(source: str, max_file_size: int, from_web: bool, include_pattern
query = parse_url(source) query = parse_url(source)
else: else:
query = parse_path(source) query = parse_path(source)
query['max_file_size'] = max_file_size query['max_file_size'] = max_file_size
if ignore_patterns and ignore_patterns != "": if ignore_patterns and ignore_patterns != "":
ignore_patterns = DEFAULT_IGNORE_PATTERNS + parse_patterns(ignore_patterns) ignore_patterns = DEFAULT_IGNORE_PATTERNS + parse_patterns(ignore_patterns)
else: else:
ignore_patterns = DEFAULT_IGNORE_PATTERNS ignore_patterns = DEFAULT_IGNORE_PATTERNS
if include_patterns and include_patterns != "": if include_patterns and include_patterns != "":
include_patterns = parse_patterns(include_patterns) include_patterns = parse_patterns(include_patterns)
ignore_patterns = override_ignore_patterns(ignore_patterns, include_patterns) ignore_patterns = override_ignore_patterns(ignore_patterns, include_patterns)
else: else:
include_patterns = None include_patterns = None
query['ignore_patterns'] = ignore_patterns query['ignore_patterns'] = ignore_patterns
query['include_patterns'] = include_patterns query['include_patterns'] = include_patterns
return query
return query

2
src/gitingest/tests/conftest.py
View file

@ -6,4 +6,4 @@ project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Add both the project root and src directory to PYTHONPATH # Add both the project root and src directory to PYTHONPATH
sys.path.insert(0, project_root) sys.path.insert(0, project_root)
sys.path.insert(0, os.path.join(project_root, 'src')) sys.path.insert(0, os.path.join(project_root, 'src'))

46
src/gitingest/tests/test_clone.py
View file

@ -1,72 +1,78 @@
from unittest.mock import AsyncMock, patch
import pytest import pytest
from clone import clone_repo, check_repo_exists
from unittest.mock import patch, AsyncMock from gitingest.clone import check_repo_exists, clone_repo
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_clone_repo_with_commit(): async def test_clone_repo_with_commit() -> None:
query = { query = {
'commit': 'a' * 40, # Simulating a valid commit hash 'commit': 'a' * 40, # Simulating a valid commit hash
'branch': 'main', 'branch': 'main',
'url': 'https://github.com/user/repo', 'url': 'https://github.com/user/repo',
'local_path': '/tmp/repo' 'local_path': '/tmp/repo',
} }
with patch('clone.check_repo_exists', return_value=True) as mock_check: with patch('gitingest.clone.check_repo_exists', return_value=True) as mock_check:
with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec: with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
mock_process = AsyncMock() mock_process = AsyncMock()
mock_process.communicate.return_value = (b'output', b'error') mock_process.communicate.return_value = (b'output', b'error')
mock_exec.return_value = mock_process mock_exec.return_value = mock_process
await clone_repo(query) await clone_repo(query)
mock_check.assert_called_once_with(query['url']) mock_check.assert_called_once_with(query['url'])
assert mock_exec.call_count == 2 # Clone and checkout calls assert mock_exec.call_count == 2 # Clone and checkout calls
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_clone_repo_without_commit(): async def test_clone_repo_without_commit() -> None:
query = { query = {
'commit': None, 'commit': None,
'branch': 'main', 'branch': 'main',
'url': 'https://github.com/user/repo', 'url': 'https://github.com/user/repo',
'local_path': '/tmp/repo' 'local_path': '/tmp/repo',
} }
with patch('clone.check_repo_exists', return_value=True) as mock_check: with patch('gitingest.clone.check_repo_exists', return_value=True) as mock_check:
with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec: with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
mock_process = AsyncMock() mock_process = AsyncMock()
mock_process.communicate.return_value = (b'output', b'error') mock_process.communicate.return_value = (b'output', b'error')
mock_exec.return_value = mock_process mock_exec.return_value = mock_process
await clone_repo(query) await clone_repo(query)
mock_check.assert_called_once_with(query['url']) mock_check.assert_called_once_with(query['url'])
assert mock_exec.call_count == 1 # Only clone call assert mock_exec.call_count == 1 # Only clone call
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_clone_repo_nonexistent_repository(): async def test_clone_repo_nonexistent_repository() -> None:
query = { query = {
'commit': None, 'commit': None,
'branch': 'main', 'branch': 'main',
'url': 'https://github.com/user/nonexistent-repo', 'url': 'https://github.com/user/nonexistent-repo',
'local_path': '/tmp/repo' 'local_path': '/tmp/repo',
} }
with patch('gitingest.clone.check_repo_exists', return_value=False) as mock_check: with patch('gitingest.clone.check_repo_exists', return_value=False) as mock_check:
with pytest.raises(ValueError, match="Repository not found"): with pytest.raises(ValueError, match="Repository not found"):
await clone_repo(query) await clone_repo(query)
mock_check.assert_called_once_with(query['url']) mock_check.assert_called_once_with(query['url'])
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_check_repo_exists(): async def test_check_repo_exists() -> None:
url = "https://github.com/user/repo" url = "https://github.com/user/repo"
with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec: with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
mock_process = AsyncMock() mock_process = AsyncMock()
mock_process.communicate.return_value = (b'HTTP/1.1 200 OK\n', b'') mock_process.communicate.return_value = (b'HTTP/1.1 200 OK\n', b'')
mock_exec.return_value = mock_process mock_exec.return_value = mock_process
# Test existing repository # Test existing repository
mock_process.returncode = 0 mock_process.returncode = 0
assert await check_repo_exists(url) is True assert await check_repo_exists(url) is True
# Test non-existing repository (404 response) # Test non-existing repository (404 response)
mock_process.communicate.return_value = (b'HTTP/1.1 404 Not Found\n', b'') mock_process.communicate.return_value = (b'HTTP/1.1 404 Not Found\n', b'')
mock_process.returncode = 0 mock_process.returncode = 0
@ -74,4 +80,4 @@ async def test_check_repo_exists():
# Test failed request # Test failed request
mock_process.returncode = 1 mock_process.returncode = 1
assert await check_repo_exists(url) is False assert await check_repo_exists(url) is False

72
src/gitingest/tests/test_ingest.py
View file

@ -1,12 +1,14 @@
from pathlib import Path
from typing import Any, Dict
import pytest import pytest
from src.gitingest.ingest_from_query import (
scan_directory, from gitingest.ingest_from_query import extract_files_content, scan_directory
extract_files_content,
)
# Test fixtures # Test fixtures
@pytest.fixture @pytest.fixture
def sample_query(): def sample_query() -> Dict[str, Any]:
return { return {
'user_name': 'test_user', 'user_name': 'test_user',
'repo_name': 'test_repo', 'repo_name': 'test_repo',
@ -14,16 +16,16 @@ def sample_query():
'subpath': '/', 'subpath': '/',
'branch': 'main', 'branch': 'main',
'commit': None, 'commit': None,
'max_file_size': 1000000, 'max_file_size': 1_000_000,
'slug': 'test_user/test_repo', 'slug': 'test_user/test_repo',
'ignore_patterns': ['*.pyc', '__pycache__', '.git'], 'ignore_patterns': ['*.pyc', '__pycache__', '.git'],
'include_patterns': None, 'include_patterns': None,
'pattern_type': 'exclude' 'pattern_type': 'exclude',
} }
@pytest.fixture @pytest.fixture
def temp_directory(tmp_path): def temp_directory(tmp_path: Path) -> Path:
# Creates the following structure: # Creates the following structure:
# test_repo/ # test_repo/
# ├── file1.txt # ├── file1.txt
@ -38,58 +40,57 @@ def temp_directory(tmp_path):
# | └── file_dir1.txt # | └── file_dir1.txt
# └── dir2/ # └── dir2/
# └── file_dir2.txt # └── file_dir2.txt
test_dir = tmp_path / "test_repo" test_dir = tmp_path / "test_repo"
test_dir.mkdir() test_dir.mkdir()
# Root files # Root files
(test_dir / "file1.txt").write_text("Hello World") (test_dir / "file1.txt").write_text("Hello World")
(test_dir / "file2.py").write_text("print('Hello')") (test_dir / "file2.py").write_text("print('Hello')")
# src directory and its files # src directory and its files
src_dir = test_dir / "src" src_dir = test_dir / "src"
src_dir.mkdir() src_dir.mkdir()
(src_dir / "subfile1.txt").write_text("Hello from src") (src_dir / "subfile1.txt").write_text("Hello from src")
(src_dir / "subfile2.py").write_text("print('Hello from src')") (src_dir / "subfile2.py").write_text("print('Hello from src')")
# src/subdir and its files # src/subdir and its files
subdir = src_dir / "subdir" subdir = src_dir / "subdir"
subdir.mkdir() subdir.mkdir()
(subdir / "file_subdir.txt").write_text("Hello from subdir") (subdir / "file_subdir.txt").write_text("Hello from subdir")
(subdir / "file_subdir.py").write_text("print('Hello from subdir')") (subdir / "file_subdir.py").write_text("print('Hello from subdir')")
# dir1 and its file # dir1 and its file
dir1 = test_dir / "dir1" dir1 = test_dir / "dir1"
dir1.mkdir() dir1.mkdir()
(dir1 / "file_dir1.txt").write_text("Hello from dir1") (dir1 / "file_dir1.txt").write_text("Hello from dir1")
# dir2 and its file # dir2 and its file
dir2 = test_dir / "dir2" dir2 = test_dir / "dir2"
dir2.mkdir() dir2.mkdir()
(dir2 / "file_dir2.txt").write_text("Hello from dir2") (dir2 / "file_dir2.txt").write_text("Hello from dir2")
return test_dir return test_dir
def test_scan_directory(temp_directory, sample_query):
result = scan_directory( def test_scan_directory(temp_directory: Path, sample_query: Dict[str, Any]) -> None:
str(temp_directory), result = scan_directory(str(temp_directory), query=sample_query)
query=sample_query if result is None:
) assert False, "Result is None"
assert result['type'] == 'directory' assert result['type'] == 'directory'
assert result['file_count'] == 8 # All .txt and .py files assert result['file_count'] == 8 # All .txt and .py files
assert result['dir_count'] == 4 # src, src/subdir, dir1, dir2 assert result['dir_count'] == 4 # src, src/subdir, dir1, dir2
assert len(result['children']) == 5 # file1.txt, file2.py, src, dir1, dir2 assert len(result['children']) == 5 # file1.txt, file2.py, src, dir1, dir2
def test_extract_files_content(temp_directory, sample_query):
nodes = scan_directory( def test_extract_files_content(temp_directory: Path, sample_query: Dict[str, Any]) -> None:
str(temp_directory), nodes = scan_directory(str(temp_directory), query=sample_query)
query=sample_query if nodes is None:
) assert False, "Nodes is None"
files = extract_files_content(query=sample_query, node=nodes, max_file_size=1_000_000)
files = extract_files_content(sample_query, nodes, max_file_size=1000000)
assert len(files) == 8 # All .txt and .py files assert len(files) == 8 # All .txt and .py files
# Check for presence of key files # Check for presence of key files
paths = [f['path'] for f in files] paths = [f['path'] for f in files]
assert any('file1.txt' in p for p in paths) assert any('file1.txt' in p for p in paths)
@ -101,22 +102,17 @@ def test_extract_files_content(temp_directory, sample_query):
assert any('file_dir2.txt' in p for p in paths) assert any('file_dir2.txt' in p for p in paths)
# TODO: test with include patterns: ['*.txt'] # TODO: test with include patterns: ['*.txt']
# TODO: test with wrong include patterns: ['*.qwerty'] # TODO: test with wrong include patterns: ['*.qwerty']
#single folder patterns # single folder patterns
# TODO: test with include patterns: ['src/*'] # TODO: test with include patterns: ['src/*']
# TODO: test with include patterns: ['/src/*'] # TODO: test with include patterns: ['/src/*']
# TODO: test with include patterns: ['/src/'] # TODO: test with include patterns: ['/src/']
# TODO: test with include patterns: ['/src*'] # TODO: test with include patterns: ['/src*']
#multiple patterns # multiple patterns
# TODO: test with multiple include patterns: ['*.txt', '*.py'] # TODO: test with multiple include patterns: ['*.txt', '*.py']
# TODO: test with multiple include patterns: ['/src/*', '*.txt'] # TODO: test with multiple include patterns: ['/src/*', '*.txt']
# TODO: test with multiple include patterns: ['/src*', '*.txt'] # TODO: test with multiple include patterns: ['/src*', '*.txt']

29
src/gitingest/tests/test_parse_query.py
View file

@ -1,12 +1,14 @@
import pytest import pytest
from gitingest.parse_query import parse_query, parse_url, DEFAULT_IGNORE_PATTERNS
from gitingest.ignore_patterns import DEFAULT_IGNORE_PATTERNS
from gitingest.parse_query import parse_query, parse_url
def test_parse_url_valid(): def test_parse_url_valid() -> None:
test_cases = [ test_cases = [
"https://github.com/user/repo", "https://github.com/user/repo",
"https://gitlab.com/user/repo", "https://gitlab.com/user/repo",
"https://bitbucket.org/user/repo" "https://bitbucket.org/user/repo",
] ]
for url in test_cases: for url in test_cases:
result = parse_url(url) result = parse_url(url)
@ -14,16 +16,15 @@ def test_parse_url_valid():
assert result["repo_name"] == "repo" assert result["repo_name"] == "repo"
assert result["url"] == url assert result["url"] == url
def test_parse_url_invalid():
def test_parse_url_invalid() -> None:
url = "https://only-domain.com" url = "https://only-domain.com"
with pytest.raises(ValueError, match="Invalid repository URL"): with pytest.raises(ValueError, match="Invalid repository URL"):
parse_url(url) parse_url(url)
def test_parse_query_basic():
test_cases = [ def test_parse_query_basic() -> None:
"https://github.com/user/repo", test_cases = ["https://github.com/user/repo", "https://gitlab.com/user/repo"]
"https://gitlab.com/user/repo"
]
for url in test_cases: for url in test_cases:
result = parse_query(url, max_file_size=50, from_web=True, ignore_patterns='*.txt') result = parse_query(url, max_file_size=50, from_web=True, ignore_patterns='*.txt')
assert result["user_name"] == "user" assert result["user_name"] == "user"
@ -31,13 +32,15 @@ def test_parse_query_basic():
assert result["url"] == url assert result["url"] == url
assert "*.txt" in result["ignore_patterns"] assert "*.txt" in result["ignore_patterns"]
def test_parse_query_include_pattern():
def test_parse_query_include_pattern() -> None:
url = "https://github.com/user/repo" url = "https://github.com/user/repo"
result = parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py') result = parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py')
assert result["include_patterns"] == ["*.py"] assert result["include_patterns"] == ["*.py"]
assert result["ignore_patterns"] == DEFAULT_IGNORE_PATTERNS assert result["ignore_patterns"] == DEFAULT_IGNORE_PATTERNS
def test_parse_query_invalid_pattern():
def test_parse_query_invalid_pattern() -> None:
url = "https://github.com/user/repo" url = "https://github.com/user/repo"
with pytest.raises(ValueError, match="Pattern.*contains invalid characters"): with pytest.raises(ValueError, match="Pattern.*contains invalid characters"):
parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py;rm -rf') parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py;rm -rf')

19
src/gitingest/utils.py
View file

@ -1,22 +1,27 @@
## Async Timeout decorator ## Async Timeout decorator
import asyncio import asyncio
import functools import functools
from typing import TypeVar, Callable from typing import Awaitable, Callable, ParamSpec, TypeVar
T = TypeVar("T") T = TypeVar("T")
P = ParamSpec("P")
class AsyncTimeoutError(Exception): class AsyncTimeoutError(Exception):
"""Raised when an async operation exceeds its timeout limit.""" """Raised when an async operation exceeds its timeout limit."""
pass pass
def async_timeout(seconds: int = 10):
def decorator(func: Callable[..., T]) -> Callable[..., T]: def async_timeout(seconds: int = 10) -> Callable[[Callable[P, Awaitable[T]]], Callable[P, Awaitable[T]]]:
def decorator(func: Callable[P, Awaitable[T]]) -> Callable[P, Awaitable[T]]:
@functools.wraps(func) @functools.wraps(func)
async def wrapper(*args, **kwargs) -> T: async def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
try: try:
return await asyncio.wait_for(func(*args, **kwargs), timeout=seconds) return await asyncio.wait_for(func(*args, **kwargs), timeout=seconds)
except asyncio.TimeoutError: except asyncio.TimeoutError:
raise AsyncTimeoutError(f"Clone timed out after {seconds} seconds") raise AsyncTimeoutError(f"Operation timed out after {seconds} seconds")
return wrapper return wrapper
return decorator
return decorator

62
src/main.py
View file

@ -1,27 +1,41 @@
import os import os
from dotenv import load_dotenv from typing import Dict
from fastapi import FastAPI, Request
from fastapi.templating import Jinja2Templates
from fastapi.responses import HTMLResponse, FileResponse, Response
from fastapi.staticfiles import StaticFiles
from starlette.middleware.trustedhost import TrustedHostMiddleware
from api_analytics.fastapi import Analytics from api_analytics.fastapi import Analytics
from dotenv import load_dotenv
from fastapi import FastAPI, Request
from fastapi.responses import FileResponse, HTMLResponse, Response
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
from slowapi import _rate_limit_exceeded_handler from slowapi import _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded from slowapi.errors import RateLimitExceeded
from starlette.middleware.trustedhost import TrustedHostMiddleware
from server_utils import limiter
from routers import download, dynamic, index from routers import download, dynamic, index
from server_utils import limiter
load_dotenv() load_dotenv()
app = FastAPI() app = FastAPI()
app.state.limiter = limiter app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
# Define a wrapper handler with the correct signature
async def rate_limit_exception_handler(request: Request, exc: Exception) -> Response:
if isinstance(exc, RateLimitExceeded):
# Delegate to the actual handler
return _rate_limit_exceeded_handler(request, exc)
# Optionally, handle other exceptions or re-raise
raise exc
# Register the wrapper handler
app.add_exception_handler(RateLimitExceeded, rate_limit_exception_handler)
app.mount("/static", StaticFiles(directory="static"), name="static") app.mount("/static", StaticFiles(directory="static"), name="static")
app.add_middleware(Analytics, api_key=os.getenv('API_ANALYTICS_KEY')) app_analytics_key = os.getenv('API_ANALYTICS_KEY')
if app_analytics_key:
app.add_middleware(Analytics, api_key=app_analytics_key)
# Define the default allowed hosts # Define the default allowed hosts
default_allowed_hosts = ["gitingest.com", "*.gitingest.com", "localhost", "127.0.0.1"] default_allowed_hosts = ["gitingest.com", "*.gitingest.com", "localhost", "127.0.0.1"]
@ -36,31 +50,29 @@ else:
app.add_middleware(TrustedHostMiddleware, allowed_hosts=allowed_hosts) app.add_middleware(TrustedHostMiddleware, allowed_hosts=allowed_hosts)
templates = Jinja2Templates(directory="templates") templates = Jinja2Templates(directory="templates")
@app.get("/health") @app.get("/health")
async def health_check(): async def health_check() -> Dict[str, str]:
return {"status": "healthy"} return {"status": "healthy"}
@app.head("/") @app.head("/")
async def head_root(): async def head_root() -> HTMLResponse:
"""Mirror the headers and status code of the index page""" """Mirror the headers and status code of the index page"""
return HTMLResponse( return HTMLResponse(content=None, headers={"content-type": "text/html; charset=utf-8"})
content=None,
headers={
"content-type": "text/html; charset=utf-8"
}
)
@app.get("/api/", response_class=HTMLResponse) @app.get("/api/", response_class=HTMLResponse)
@app.get("/api", response_class=HTMLResponse) @app.get("/api", response_class=HTMLResponse)
async def api_docs(request: Request): async def api_docs(request: Request) -> HTMLResponse:
return templates.TemplateResponse( return templates.TemplateResponse("api.jinja", {"request": request})
"api.jinja", {"request": request}
)
@app.get("/robots.txt") @app.get("/robots.txt")
async def robots(): async def robots() -> FileResponse:
return FileResponse('static/robots.txt') return FileResponse('static/robots.txt')
app.include_router(index) app.include_router(index)
app.include_router(download) app.include_router(download)
app.include_router(dynamic) app.include_router(dynamic)

105
src/process_query.py
View file

@ -1,16 +1,27 @@
from typing import List from typing import Any, Dict
from fastapi.templating import Jinja2Templates
from fastapi import Request
from config import MAX_DISPLAY_SIZE, EXAMPLE_REPOS from fastapi import Request
from gitingest import ingest_from_query, clone_repo, parse_query from fastapi.templating import Jinja2Templates
from server_utils import logSliderToSize, Colors from starlette.templating import _TemplateResponse
from config import EXAMPLE_REPOS, MAX_DISPLAY_SIZE
from gitingest.clone import clone_repo
from gitingest.ingest_from_query import ingest_from_query
from gitingest.parse_query import parse_query
from server_utils import Colors, logSliderToSize
templates = Jinja2Templates(directory="templates") templates = Jinja2Templates(directory="templates")
def print_query(query, request, max_file_size, pattern_type, pattern):
def print_query(
query: Dict[str, Any],
request: Request,
max_file_size: int,
pattern_type: str,
pattern: str,
) -> None:
print(f"{Colors.WHITE}{query['url']:<20}{Colors.END}", end="") print(f"{Colors.WHITE}{query['url']:<20}{Colors.END}", end="")
if int(max_file_size/1024) != 50: if int(max_file_size / 1024) != 50:
print(f" | {Colors.YELLOW}Size: {int(max_file_size/1024)}kb{Colors.END}", end="") print(f" | {Colors.YELLOW}Size: {int(max_file_size/1024)}kb{Colors.END}", end="")
if pattern_type == "include" and pattern != "": if pattern_type == "include" and pattern != "":
print(f" | {Colors.YELLOW}Include {pattern}{Colors.END}", end="") print(f" | {Colors.YELLOW}Include {pattern}{Colors.END}", end="")
@ -18,46 +29,74 @@ def print_query(query, request, max_file_size, pattern_type, pattern):
print(f" | {Colors.YELLOW}Exclude {pattern}{Colors.END}", end="") print(f" | {Colors.YELLOW}Exclude {pattern}{Colors.END}", end="")
def print_error(query, request, e, max_file_size, pattern_type, pattern): def print_error(
query: Dict[str, Any],
request: Request,
e: Exception,
max_file_size: int,
pattern_type: str,
pattern: str,
) -> None:
print(f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}<- {Colors.END}", end="") print(f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}<- {Colors.END}", end="")
print_query(query, request, max_file_size, pattern_type, pattern) print_query(query, request, max_file_size, pattern_type, pattern)
print(f" | {Colors.RED}{e}{Colors.END}") print(f" | {Colors.RED}{e}{Colors.END}")
def print_success(query, request, max_file_size, pattern_type, pattern, summary):
def print_success(
query: Dict[str, Any],
request: Request,
max_file_size: int,
pattern_type: str,
pattern: str,
summary: str,
) -> None:
estimated_tokens = summary[summary.index("Estimated tokens:") + len("Estimated ") :] estimated_tokens = summary[summary.index("Estimated tokens:") + len("Estimated ") :]
print(f"{Colors.GREEN}INFO{Colors.END}: {Colors.GREEN}<- {Colors.END}", end="") print(f"{Colors.GREEN}INFO{Colors.END}: {Colors.GREEN}<- {Colors.END}", end="")
print_query(query, request, max_file_size, pattern_type, pattern) print_query(query, request, max_file_size, pattern_type, pattern)
print(f" | {Colors.PURPLE}{estimated_tokens}{Colors.END}") print(f" | {Colors.PURPLE}{estimated_tokens}{Colors.END}")
async def process_query(
async def process_query(request: Request, input_text: str, slider_position: int, pattern_type: str = "exclude", pattern: str = "", is_index: bool = False) -> str: request: Request,
input_text: str,
slider_position: int,
pattern_type: str = "exclude",
pattern: str = "",
is_index: bool = False,
) -> _TemplateResponse:
template = "index.jinja" if is_index else "github.jinja" template = "index.jinja" if is_index else "github.jinja"
max_file_size = logSliderToSize(slider_position) max_file_size = logSliderToSize(slider_position)
if pattern_type == "include": if pattern_type == "include":
include_patterns = pattern include_patterns = pattern
exclude_patterns = None exclude_patterns = None
elif pattern_type == "exclude": elif pattern_type == "exclude":
exclude_patterns = pattern exclude_patterns = pattern
include_patterns = None include_patterns = None
try: try:
query = parse_query(input_text, max_file_size, True, include_patterns, exclude_patterns) query = parse_query(
source=input_text,
max_file_size=max_file_size,
from_web=True,
include_patterns=include_patterns,
ignore_patterns=exclude_patterns,
)
await clone_repo(query) await clone_repo(query)
summary, tree, content = ingest_from_query(query) summary, tree, content = ingest_from_query(query)
with open(f"{query['local_path']}.txt", "w") as f: with open(f"{query['local_path']}.txt", "w") as f:
f.write(tree + "\n" + content) f.write(tree + "\n" + content)
except Exception as e: except Exception as e:
#hack to print error message when query is not defined # hack to print error message when query is not defined
if 'query' in locals() and query is not None and isinstance(query, dict): if 'query' in locals() and query is not None and isinstance(query, dict):
print_error(query, request, e, max_file_size, pattern_type, pattern) print_error(query, request, e, max_file_size, pattern_type, pattern)
else: else:
print(f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}<- {Colors.END}", end="") print(f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}<- {Colors.END}", end="")
print(f"{Colors.RED}{e}{Colors.END}") print(f"{Colors.RED}{e}{Colors.END}")
return templates.TemplateResponse( return templates.TemplateResponse(
template, template,
{ {
"request": request, "request": request,
"github_url": input_text, "github_url": input_text,
@ -66,25 +105,37 @@ async def process_query(request: Request, input_text: str, slider_position: int,
"default_file_size": slider_position, "default_file_size": slider_position,
"pattern_type": pattern_type, "pattern_type": pattern_type,
"pattern": pattern, "pattern": pattern,
} },
) )
if len(content) > MAX_DISPLAY_SIZE: if len(content) > MAX_DISPLAY_SIZE:
content = f"(Files content cropped to {int(MAX_DISPLAY_SIZE/1000)}k characters, download full ingest to see more)\n" + content[:MAX_DISPLAY_SIZE] content = (
print_success(query, request, max_file_size, pattern_type, pattern, summary) f"(Files content cropped to {int(MAX_DISPLAY_SIZE / 1_000)}k characters, "
"download full ingest to see more)\n" + content[:MAX_DISPLAY_SIZE]
)
print_success(
query=query,
request=request,
max_file_size=max_file_size,
pattern_type=pattern_type,
pattern=pattern,
summary=summary,
)
return templates.TemplateResponse( return templates.TemplateResponse(
template, template,
{ {
"request": request, "request": request,
"github_url": input_text, "github_url": input_text,
"result": True, "result": True,
"summary": summary, "summary": summary,
"tree": tree, "tree": tree,
"content": content, "content": content,
"examples": EXAMPLE_REPOS if is_index else [], "examples": EXAMPLE_REPOS if is_index else [],
"ingest_id": query['id'], "ingest_id": query['id'],
"default_file_size": slider_position, "default_file_size": slider_position,
"pattern_type": pattern_type, "pattern_type": pattern_type,
"pattern": pattern, "pattern": pattern,
} },
) )

8
src/routers/__init__.py
View file

@ -1,5 +1,5 @@
from .download import router as download from routers.download import router as download
from .dynamic import router as dynamic from routers.dynamic import router as dynamic
from .index import router as index from routers.index import router as index
__all__ = ["download", "dynamic", "index"] __all__ = ["download", "dynamic", "index"]

25
src/routers/download.py
View file

@ -1,29 +1,30 @@
from fastapi import HTTPException, APIRouter
from fastapi.responses import Response
from config import TMP_BASE_PATH
import os import os
from fastapi import APIRouter, HTTPException
from fastapi.responses import Response
from config import TMP_BASE_PATH
router = APIRouter() router = APIRouter()
@router.get("/download/{digest_id}") @router.get("/download/{digest_id}")
async def download_ingest(digest_id: str): async def download_ingest(digest_id: str) -> Response:
try: try:
# Find the first .txt file in the directory # Find the first .txt file in the directory
directory = f"{TMP_BASE_PATH}/{digest_id}" directory = f"{TMP_BASE_PATH}/{digest_id}"
txt_files = [f for f in os.listdir(directory) if f.endswith('.txt')] txt_files = [f for f in os.listdir(directory) if f.endswith('.txt')]
if not txt_files: if not txt_files:
raise FileNotFoundError("No .txt file found") raise FileNotFoundError("No .txt file found")
with open(f"{directory}/{txt_files[0]}", "r") as f: with open(f"{directory}/{txt_files[0]}") as f:
content = f.read() content = f.read()
return Response( return Response(
content=content, content=content,
media_type="text/plain", media_type="text/plain",
headers={ headers={"Content-Disposition": f"attachment; filename={txt_files[0]}"},
"Content-Disposition": f"attachment; filename={txt_files[0]}"
}
) )
except FileNotFoundError: except FileNotFoundError:
raise HTTPException(status_code=404, detail="Digest not found") raise HTTPException(status_code=404, detail="Digest not found")

28
src/routers/dynamic.py
View file

@ -1,4 +1,4 @@
from fastapi import APIRouter, Request, Form from fastapi import APIRouter, Form, Request
from fastapi.responses import HTMLResponse from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
@ -8,26 +8,34 @@ from server_utils import limiter
router = APIRouter() router = APIRouter()
templates = Jinja2Templates(directory="templates") templates = Jinja2Templates(directory="templates")
@router.get("/{full_path:path}") @router.get("/{full_path:path}")
async def catch_all(request: Request, full_path: str): async def catch_all(request: Request, full_path: str) -> HTMLResponse:
return templates.TemplateResponse( return templates.TemplateResponse(
"github.jinja", "github.jinja",
{ {
"request": request, "request": request,
"github_url": f"https://github.com/{full_path}", "github_url": f"https://github.com/{full_path}",
"loading": True, "loading": True,
"default_file_size": 243 "default_file_size": 243,
} },
) )
@router.post("/{full_path:path}", response_class=HTMLResponse) @router.post("/{full_path:path}", response_class=HTMLResponse)
@limiter.limit("10/minute") @limiter.limit("10/minute")
async def process_catch_all( async def process_catch_all(
request: Request, request: Request,
input_text: str = Form(...), input_text: str = Form(...),
max_file_size: int = Form(...), max_file_size: int = Form(...),
pattern_type: str = Form(...), pattern_type: str = Form(...),
pattern: str = Form(...) pattern: str = Form(...),
): ) -> HTMLResponse:
return await process_query(request, input_text, max_file_size, pattern_type, pattern, is_index=False) return await process_query(
request,
input_text,
max_file_size,
pattern_type,
pattern,
is_index=False,
)

37
src/routers/index.py
View file

@ -1,40 +1,41 @@
from fastapi import APIRouter, Request, Form from fastapi import APIRouter, Form, Request
from fastapi.responses import HTMLResponse from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
from server_utils import limiter
from process_query import process_query
from config import EXAMPLE_REPOS from config import EXAMPLE_REPOS
from process_query import process_query
from server_utils import limiter
router = APIRouter() router = APIRouter()
templates = Jinja2Templates(directory="templates") templates = Jinja2Templates(directory="templates")
@router.get("/", response_class=HTMLResponse) @router.get("/", response_class=HTMLResponse)
async def home(request: Request): async def home(request: Request) -> HTMLResponse:
return templates.TemplateResponse( return templates.TemplateResponse(
"index.jinja", "index.jinja",
{ {
"request": request, "request": request,
"examples": EXAMPLE_REPOS, "examples": EXAMPLE_REPOS,
"default_file_size": 243 "default_file_size": 243,
} },
) )
@router.post("/", response_class=HTMLResponse) @router.post("/", response_class=HTMLResponse)
@limiter.limit("10/minute") @limiter.limit("10/minute")
async def index_post( async def index_post(
request: Request, request: Request,
input_text: str = Form(...), input_text: str = Form(...),
max_file_size: int = Form(...), max_file_size: int = Form(...),
pattern_type: str = Form(...), pattern_type: str = Form(...),
pattern: str = Form(...) pattern: str = Form(...),
): ) -> HTMLResponse:
return await process_query(request, input_text, max_file_size, pattern_type, pattern, is_index=True) return await process_query(
request,
input_text,
max_file_size,
pattern_type,
pattern,
is_index=True,
)

13
src/server_utils.py
View file

@ -1,21 +1,26 @@
import math
## Rate Limiter ## Rate Limiter
from slowapi import Limiter from slowapi import Limiter
from slowapi.util import get_remote_address from slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address) limiter = Limiter(key_func=get_remote_address)
## Logarithmic slider to file size
import math ## Logarithmic slider to file size conversion
def logSliderToSize(position): def logSliderToSize(position: int) -> int:
"""Convert slider position to file size in KB""" """Convert slider position to file size in KB"""
maxp = 500 maxp = 500
minv = math.log(1) minv = math.log(1)
maxv = math.log(102400) maxv = math.log(102400)
return round(math.exp(minv + (maxv - minv) * pow(position / maxp, 1.5))) * 1024 return round(math.exp(minv + (maxv - minv) * pow(position / maxp, 1.5))) * 1024
## Color printing utility ## Color printing utility
class Colors: class Colors:
"""ANSI color codes""" """ANSI color codes"""
BLACK = "\033[0;30m" BLACK = "\033[0;30m"
RED = "\033[0;31m" RED = "\033[0;31m"
GREEN = "\033[0;32m" GREEN = "\033[0;32m"

2
src/static/favicon.svg
View file

@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="512px" height="512px"><defs><style>.cls-1{fill:#d4e9ff;}.cls-2{fill:#f1f8ff;}.cls-3{fill:#b7daff;}</style></defs><title>1</title><g id="Layer_70" data-name="Layer 70"><path class="cls-1" d="M52,35V16L41,4H14.42A2.42,2.42,0,0,0,12,6.42V35H8V49h4v8.58A2.42,2.42,0,0,0,14.42,60H49.58A2.42,2.42,0,0,0,52,57.58V49h4V35Z"/><polygon class="cls-2" points="52 16 41 16 41 4 52 16"/><rect class="cls-3" x="12" y="49" width="40" height="4"/><path d="M19.76,46.06a.91.91,0,0,1-.63-.23.83.83,0,0,1-.26-.66.85.85,0,0,1,.25-.62.83.83,0,0,1,.62-.26.87.87,0,0,1,.63.26.83.83,0,0,1,.26.62.84.84,0,0,1-.26.65A.88.88,0,0,1,19.76,46.06Z"/><path d="M24.27,38.24v6.25H27.8a.93.93,0,0,1,.65.21.67.67,0,0,1,.23.52.65.65,0,0,1-.22.51,1,1,0,0,1-.65.2H23.6a.91.91,0,0,1-1.07-1.07V38.24a1.15,1.15,0,0,1,.24-.79.8.8,0,0,1,.62-.26.83.83,0,0,1,.64.26A1.13,1.13,0,0,1,24.27,38.24Z"/><path d="M31.52,38.24v6.25H35a.93.93,0,0,1,.65.21.67.67,0,0,1,.23.52.65.65,0,0,1-.22.51,1,1,0,0,1-.65.2h-4.2a.91.91,0,0,1-1.07-1.07V38.24a1.15,1.15,0,0,1,.24-.79.8.8,0,0,1,.62-.26.83.83,0,0,1,.64.26A1.13,1.13,0,0,1,31.52,38.24Z"/><path d="M40,44.62l-1.38-5.47v5.93a1.08,1.08,0,0,1-.22.74.81.81,0,0,1-1.16,0,1.07,1.07,0,0,1-.22-.74v-6.8a.85.85,0,0,1,.29-.76,1.4,1.4,0,0,1,.79-.2h.54a2.06,2.06,0,0,1,.71.09.59.59,0,0,1,.33.32,4.91,4.91,0,0,1,.24.74l1.25,4.71,1.25-4.71a4.91,4.91,0,0,1,.24-.74.59.59,0,0,1,.33-.32,2.06,2.06,0,0,1,.71-.09h.54a1.4,1.4,0,0,1,.79.2.85.85,0,0,1,.29.76v6.8a1.08,1.08,0,0,1-.22.74.75.75,0,0,1-.59.25.73.73,0,0,1-.57-.25,1.07,1.07,0,0,1-.22-.74V39.15L42.3,44.62c-.09.36-.16.62-.22.78a1.08,1.08,0,0,1-.31.45.91.91,0,0,1-.63.21.92.92,0,0,1-.84-.47,1.92,1.92,0,0,1-.18-.45Z"/><path d="M57,35a1,1,0,0,0-1-1H53V16s0,0,0-.06a1,1,0,0,0,0-.21s0-.05,0-.07l0,0a1,1,0,0,0-.18-.29l-11-12a1,1,0,0,0-.29-.21l0,0-.06,0A1,1,0,0,0,41.11,3H14.42A3.42,3.42,0,0,0,11,6.42V34H8a1,1,0,0,0-1,1V49s0,0,0,.07a1.08,1.08,0,0,0,.34.68l0,.05L11,52.5v5.08A3.42,3.42,0,0,0,14.42,61H49.58A3.42,3.42,0,0,0,53,57.58V52.5l3.6-2.7,0-.05a1.08,1.08,0,0,0,.34-.68s0,0,0-.07ZM21,34V5H40V16a1,1,0,0,0,1,1H51V34ZM42,6.57,49.73,15H42ZM13,6.42A1.42,1.42,0,0,1,14.42,5H19V34H13ZM9,36H55V48H9Zm4,21.58V50h6v9H14.42A1.42,1.42,0,0,1,13,57.58ZM49.58,59H21V50H51v7.58A1.42,1.42,0,0,1,49.58,59Z"/></g></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="512px" height="512px"><defs><style>.cls-1{fill:#d4e9ff;}.cls-2{fill:#f1f8ff;}.cls-3{fill:#b7daff;}</style></defs><title>1</title><g id="Layer_70" data-name="Layer 70"><path class="cls-1" d="M52,35V16L41,4H14.42A2.42,2.42,0,0,0,12,6.42V35H8V49h4v8.58A2.42,2.42,0,0,0,14.42,60H49.58A2.42,2.42,0,0,0,52,57.58V49h4V35Z"/><polygon class="cls-2" points="52 16 41 16 41 4 52 16"/><rect class="cls-3" x="12" y="49" width="40" height="4"/><path d="M19.76,46.06a.91.91,0,0,1-.63-.23.83.83,0,0,1-.26-.66.85.85,0,0,1,.25-.62.83.83,0,0,1,.62-.26.87.87,0,0,1,.63.26.83.83,0,0,1,.26.62.84.84,0,0,1-.26.65A.88.88,0,0,1,19.76,46.06Z"/><path d="M24.27,38.24v6.25H27.8a.93.93,0,0,1,.65.21.67.67,0,0,1,.23.52.65.65,0,0,1-.22.51,1,1,0,0,1-.65.2H23.6a.91.91,0,0,1-1.07-1.07V38.24a1.15,1.15,0,0,1,.24-.79.8.8,0,0,1,.62-.26.83.83,0,0,1,.64.26A1.13,1.13,0,0,1,24.27,38.24Z"/><path d="M31.52,38.24v6.25H35a.93.93,0,0,1,.65.21.67.67,0,0,1,.23.52.65.65,0,0,1-.22.51,1,1,0,0,1-.65.2h-4.2a.91.91,0,0,1-1.07-1.07V38.24a1.15,1.15,0,0,1,.24-.79.8.8,0,0,1,.62-.26.83.83,0,0,1,.64.26A1.13,1.13,0,0,1,31.52,38.24Z"/><path d="M40,44.62l-1.38-5.47v5.93a1.08,1.08,0,0,1-.22.74.81.81,0,0,1-1.16,0,1.07,1.07,0,0,1-.22-.74v-6.8a.85.85,0,0,1,.29-.76,1.4,1.4,0,0,1,.79-.2h.54a2.06,2.06,0,0,1,.71.09.59.59,0,0,1,.33.32,4.91,4.91,0,0,1,.24.74l1.25,4.71,1.25-4.71a4.91,4.91,0,0,1,.24-.74.59.59,0,0,1,.33-.32,2.06,2.06,0,0,1,.71-.09h.54a1.4,1.4,0,0,1,.79.2.85.85,0,0,1,.29.76v6.8a1.08,1.08,0,0,1-.22.74.75.75,0,0,1-.59.25.73.73,0,0,1-.57-.25,1.07,1.07,0,0,1-.22-.74V39.15L42.3,44.62c-.09.36-.16.62-.22.78a1.08,1.08,0,0,1-.31.45.91.91,0,0,1-.63.21.92.92,0,0,1-.84-.47,1.92,1.92,0,0,1-.18-.45Z"/><path d="M57,35a1,1,0,0,0-1-1H53V16s0,0,0-.06a1,1,0,0,0,0-.21s0-.05,0-.07l0,0a1,1,0,0,0-.18-.29l-11-12a1,1,0,0,0-.29-.21l0,0-.06,0A1,1,0,0,0,41.11,3H14.42A3.42,3.42,0,0,0,11,6.42V34H8a1,1,0,0,0-1,1V49s0,0,0,.07a1.08,1.08,0,0,0,.34.68l0,.05L11,52.5v5.08A3.42,3.42,0,0,0,14.42,61H49.58A3.42,3.42,0,0,0,53,57.58V52.5l3.6-2.7,0-.05a1.08,1.08,0,0,0,.34-.68s0,0,0-.07ZM21,34V5H40V16a1,1,0,0,0,1,1H51V34ZM42,6.57,49.73,15H42ZM13,6.42A1.42,1.42,0,0,1,14.42,5H19V34H13ZM9,36H55V48H9Zm4,21.58V50h6v9H14.42A1.42,1.42,0,0,1,13,57.58ZM49.58,59H21V50H51v7.58A1.42,1.42,0,0,1,49.58,59Z"/></g></svg>

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.2 KiB

After

Width:  |  Height:  |  Size: 2.2 KiB

Before After
Before After

2
src/static/js/snow.js
View file

@ -88,4 +88,4 @@ function initSnow() {
document.addEventListener('DOMContentLoaded', initSnow); document.addEventListener('DOMContentLoaded', initSnow);
// Also initialize when the HTMX content is swapped // Also initialize when the HTMX content is swapped
document.addEventListener('htmx:afterSettle', initSnow); document.addEventListener('htmx:afterSettle', initSnow);

3
src/static/robots.txt
View file

@ -1,5 +1,4 @@
User-agent: * User-agent: *
Allow: / Allow: /
Allow: /api/ Allow: /api/
Allow: /cyclotruc/gitingest/ Allow: /cyclotruc/gitingest/

60
src/templates/api.jinja
View file

@ -1,41 +1,35 @@
{% extends "base.jinja" %} {% extends "base.jinja" %}
{% block title %}Git ingest API{% endblock %} {% block title %}Git ingest API{% endblock %}
{% block content %} {% block content %}
<div class="relative"> <div class="relative">
<div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div> <div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div>
<div class="bg-[#fff4da] rounded-xl border-[3px] border-gray-900 p-8 relative z-20"> <div class="bg-[#fff4da] rounded-xl border-[3px] border-gray-900 p-8 relative z-20">
<h1 class="text-3xl font-bold text-gray-900 mb-4">API Documentation</h1> <h1 class="text-3xl font-bold text-gray-900 mb-4">API Documentation</h1>
<div class="prose prose-blue max-w-none">
<div class="bg-yellow-50 border-[3px] border-gray-900 p-4 mb-6 rounded-lg">
<div class="prose prose-blue max-w-none"> <div class="flex">
<div class="bg-yellow-50 border-[3px] border-gray-900 p-4 mb-6 rounded-lg"> <div class="flex-shrink-0">
<div class="flex"> <svg class="h-5 w-5 text-yellow-400"
<div class="flex-shrink-0"> viewBox="0 0 20 20"
<svg class="h-5 w-5 text-yellow-400" viewBox="0 0 20 20" fill="currentColor"> fill="currentColor">
<path fill-rule="evenodd" <path fill-rule="evenodd" d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z" clip-rule="evenodd" />
d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z" </svg>
clip-rule="evenodd" /> </div>
</svg> <div class="ml-3">
</div> <p class="text-sm text-gray-900">The API is currently under development..</p>
<div class="ml-3"> </div>
<p class="text-sm text-gray-900">
The API is currently under development..
</p>
</div> </div>
</div> </div>
<p class="text-gray-900">
We're working on making our API available to the public.
In the meantime, you can
<a href="https://github.com/cyclotruc/gitingest/issues/new"
target="_blank"
rel="noopener noreferrer"
class="text-[#6e5000] hover:underline">open an issue on github</a>
to suggest features.
</p>
</div> </div>
<p class="text-gray-900">
We're working on making our API available to the public.
In the meantime, you can
<a href="https://github.com/cyclotruc/gitingest/issues/new" target="_blank"
rel="noopener noreferrer" class="text-[#6e5000] hover:underline">
open an issue on github
</a>
to suggest features.
</p>
</div> </div>
</div> </div>
</div> {% endblock %}
{% endblock %}

111
src/templates/base.jinja
View file

@ -1,41 +1,44 @@
<!DOCTYPE html> <!DOCTYPE html>
<html lang="en"> <html lang="en">
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="icon" type="image/x-icon" href="/static/favicon.ico"> <link rel="icon" type="image/x-icon" href="/static/favicon.ico">
<!-- Search Engine Meta Tags -->
<!-- Search Engine Meta Tags --> <meta name="description"
<meta name="description" content="Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text"> content="Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text">
<meta name="keywords" content="GitIngest, AI tools, LLM integration, Ingest, Digest, Context, Prompt, Git workflow, codebase extraction, Git repository, Git automation, Summarize, prompt-friendly"> <meta name="keywords"
<meta name="robots" content="index, follow"> content="GitIngest, AI tools, LLM integration, Ingest, Digest, Context, Prompt, Git workflow, codebase extraction, Git repository, Git automation, Summarize, prompt-friendly">
<meta name="robots" content="index, follow">
<!-- Favicons --> <!-- Favicons -->
<link rel="icon" type="image/svg+xml" href="/static/favicon.svg"> <link rel="icon" type="image/svg+xml" href="/static/favicon.svg">
<link rel="icon" type="image/png" sizes="64x64" href="/static/favicon-64.png"> <link rel="icon"
<link rel="apple-touch-icon" sizes="180x180" href="/static/apple-touch-icon.png"> type="image/png"
sizes="64x64"
<!-- Web App Meta --> href="/static/favicon-64.png">
<meta name="apple-mobile-web-app-title" content="GitIngest"> <link rel="apple-touch-icon"
<meta name="application-name" content="GitIngest"> sizes="180x180"
<meta name="theme-color" content="#FCA847"> href="/static/apple-touch-icon.png">
<meta name="apple-mobile-web-app-capable" content="yes"> <!-- Web App Meta -->
<meta name="apple-mobile-web-app-status-bar-style" content="default"> <meta name="apple-mobile-web-app-title" content="GitIngest">
<meta name="application-name" content="GitIngest">
<meta name="theme-color" content="#FCA847">
<!-- OpenGraph Meta Tags --> <meta name="apple-mobile-web-app-capable" content="yes">
<meta property="og:title" content="Git ingest"> <meta name="apple-mobile-web-app-status-bar-style" content="default">
<meta property="og:description" content="Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text"> <!-- OpenGraph Meta Tags -->
<meta property="og:type" content="website"> <meta property="og:title" content="Git ingest">
<meta property="og:url" content="{{ request.url }}"> <meta property="og:description"
<meta property="og:image" content="/static/og-image.png"> content="Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text">
<meta property="og:type" content="website">
<title>{% block title %}Git ingest{% endblock %}</title> <meta property="og:url" content="{{ request.url }}">
<meta property="og:image" content="/static/og-image.png">
<script src="https://cdn.tailwindcss.com"></script> <title>
<script src="/static/js/utils.js"></script> {% block title %}Git ingest{% endblock %}
<script src="/static/js/snow.js"></script> </title>
<style> <script src="https://cdn.tailwindcss.com"></script>
<script src="/static/js/utils.js"></script>
<script src="/static/js/snow.js"></script>
<style>
#snow-canvas { #snow-canvas {
position: fixed; position: fixed;
top: 0; top: 0;
@ -45,23 +48,19 @@
width: 100%; width: 100%;
height: 100%; height: 100%;
} }
</style> </style>
{% block extra_head %}{% endblock %} {% block extra_head %}{% endblock %}
</head> </head>
<body class="bg-[#FFFDF8] min-h-screen flex flex-col">
<body class="bg-[#FFFDF8] min-h-screen flex flex-col"> <canvas id="snow-canvas"></canvas>
<canvas id="snow-canvas"></canvas> {% include 'components/navbar.jinja' %}
{% include 'components/navbar.jinja' %} <!-- Main content wrapper -->
<main class="flex-1 w-full">
<!-- Main content wrapper --> <div class="max-w-4xl mx-auto px-4 py-8">
<main class="flex-1 w-full"> {% block content %}{% endblock %}
<div class="max-w-4xl mx-auto px-4 py-8"> </div>
{% block content %}{% endblock %} </main>
</div> {% include 'components/footer.jinja' %}
</main> {% block extra_scripts %}{% endblock %}
</body>
{% include 'components/footer.jinja' %} </html>
{% block extra_scripts %}{% endblock %}
</body>
</html>

22
src/templates/components/footer.jinja
View file

@ -4,19 +4,23 @@
<div class="flex flex-col items-center"> <div class="flex flex-col items-center">
<div class="flex items-center"> <div class="flex items-center">
made with ❤️ by made with ❤️ by
<a href="https://bsky.app/profile/yasbaltrine.bsky.social" target="_blank" rel="noopener noreferrer" <a href="https://bsky.app/profile/yasbaltrine.bsky.social"
class="ml-1 hover:underline"> target="_blank"
@rom2 rel="noopener noreferrer"
</a> class="ml-1 hover:underline">@rom2</a>
</div> </div>
<a href="https://discord.gg/zerRaGK9EC" target="_blank" rel="noopener noreferrer" <a href="https://discord.gg/zerRaGK9EC"
class="mt-1 hover:underline flex items-center"> target="_blank"
<svg class="w-4 h-4 mr-1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"> rel="noopener noreferrer"
<path fill="currentColor" d="M524.531,69.836a1.5,1.5,0,0,0-.764-.7A485.065,485.065,0,0,0,404.081,32.03a1.816,1.816,0,0,0-1.923.91,337.461,337.461,0,0,0-14.9,30.6,447.848,447.848,0,0,0-134.426,0,309.541,309.541,0,0,0-15.135-30.6,1.89,1.89,0,0,0-1.924-.91A483.689,483.689,0,0,0,116.085,69.137a1.712,1.712,0,0,0-.788.676C39.068,183.651,18.186,294.69,28.43,404.354a2.016,2.016,0,0,0,.765,1.375A487.666,487.666,0,0,0,176.02,479.918a1.9,1.9,0,0,0,2.063-.676A348.2,348.2,0,0,0,208.12,430.4a1.86,1.86,0,0,0-1.019-2.588,321.173,321.173,0,0,1-45.868-21.853,1.885,1.885,0,0,1-.185-3.126c3.082-2.309,6.166-4.711,9.109-7.137a1.819,1.819,0,0,1,1.9-.256c96.229,43.917,200.41,43.917,295.5,0a1.812,1.812,0,0,1,1.924.233c2.944,2.426,6.027,4.851,9.132,7.16a1.884,1.884,0,0,1-.162,3.126,301.407,301.407,0,0,1-45.89,21.83,1.875,1.875,0,0,0-1,2.611,391.055,391.055,0,0,0,30.014,48.815,1.864,1.864,0,0,0,2.063.7A486.048,486.048,0,0,0,610.7,405.729a1.882,1.882,0,0,0,.765-1.352C623.729,277.594,590.933,167.465,524.531,69.836ZM222.491,337.58c-28.972,0-52.844-26.587-52.844-59.239S193.056,219.1,222.491,219.1c29.665,0,53.306,26.82,52.843,59.239C275.334,310.993,251.924,337.58,222.491,337.58Zm195.38,0c-28.971,0-52.843-26.587-52.843-59.239S388.437,219.1,417.871,219.1c29.667,0,53.307,26.82,52.844,59.239C470.715,310.993,447.538,337.58,417.871,337.58Z"/> class="mt-1 hover:underline flex items-center">
<svg class="w-4 h-4 mr-1"
xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 640 512">
<path fill="currentColor" d="M524.531,69.836a1.5,1.5,0,0,0-.764-.7A485.065,485.065,0,0,0,404.081,32.03a1.816,1.816,0,0,0-1.923.91,337.461,337.461,0,0,0-14.9,30.6,447.848,447.848,0,0,0-134.426,0,309.541,309.541,0,0,0-15.135-30.6,1.89,1.89,0,0,0-1.924-.91A483.689,483.689,0,0,0,116.085,69.137a1.712,1.712,0,0,0-.788.676C39.068,183.651,18.186,294.69,28.43,404.354a2.016,2.016,0,0,0,.765,1.375A487.666,487.666,0,0,0,176.02,479.918a1.9,1.9,0,0,0,2.063-.676A348.2,348.2,0,0,0,208.12,430.4a1.86,1.86,0,0,0-1.019-2.588,321.173,321.173,0,0,1-45.868-21.853,1.885,1.885,0,0,1-.185-3.126c3.082-2.309,6.166-4.711,9.109-7.137a1.819,1.819,0,0,1,1.9-.256c96.229,43.917,200.41,43.917,295.5,0a1.812,1.812,0,0,1,1.924.233c2.944,2.426,6.027,4.851,9.132,7.16a1.884,1.884,0,0,1-.162,3.126,301.407,301.407,0,0,1-45.89,21.83,1.875,1.875,0,0,0-1,2.611,391.055,391.055,0,0,0,30.014,48.815,1.864,1.864,0,0,0,2.063.7A486.048,486.048,0,0,0,610.7,405.729a1.882,1.882,0,0,0,.765-1.352C623.729,277.594,590.933,167.465,524.531,69.836ZM222.491,337.58c-28.972,0-52.844-26.587-52.844-59.239S193.056,219.1,222.491,219.1c29.665,0,53.306,26.82,52.843,59.239C275.334,310.993,251.924,337.58,222.491,337.58Zm195.38,0c-28.971,0-52.843-26.587-52.843-59.239S388.437,219.1,417.871,219.1c29.667,0,53.307,26.82,52.844,59.239C470.715,310.993,447.538,337.58,417.871,337.58Z" />
</svg> </svg>
Discord Discord
</a> </a>
</div> </div>
</div> </div>
</div> </div>
</footer> </footer>

108
src/templates/components/github_form.jinja
View file

@ -2,28 +2,30 @@
<div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div> <div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div>
<div class="rounded-xl relative z-20 pl-8 sm:pl-10 pr-8 sm:pr-16 py-8 border-[3px] border-gray-900 bg-[#fff4da]"> <div class="rounded-xl relative z-20 pl-8 sm:pl-10 pr-8 sm:pr-16 py-8 border-[3px] border-gray-900 bg-[#fff4da]">
<img src="https://cdn.devdojo.com/images/january2023/shape-1.png" <img src="https://cdn.devdojo.com/images/january2023/shape-1.png"
class="absolute md:block hidden left-0 h-[4.5rem] w-[4.5rem] bottom-0 -translate-x-full ml-3"> class="absolute md:block hidden left-0 h-[4.5rem] w-[4.5rem] bottom-0 -translate-x-full ml-3">
<form class="flex md:flex-row flex-col w-full h-full justify-center items-stretch space-y-5 md:space-y-0 md:space-x-5" <form class="flex md:flex-row flex-col w-full h-full justify-center items-stretch space-y-5 md:space-y-0 md:space-x-5"
id="ingestForm" onsubmit="handleSubmit(event{% if is_index %}, true{% endif %})"> id="ingestForm"
onsubmit="handleSubmit(event{% if is_index %}, true{% endif %})">
<div class="relative w-full h-full"> <div class="relative w-full h-full">
<div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div> <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
<input type="text" name="input_text" id="input_text" placeholder="https://github.com/..." <input type="text"
value="{{ github_url if github_url else '' }}" name="input_text"
required id="input_text"
class="border-[3px] w-full relative z-20 border-gray-900 placeholder-gray-600 text-lg font-medium focus:outline-none py-3.5 px-6 rounded"> placeholder="https://github.com/..."
value="{{ github_url if github_url else '' }}"
required
class="border-[3px] w-full relative z-20 border-gray-900 placeholder-gray-600 text-lg font-medium focus:outline-none py-3.5 px-6 rounded">
</div> </div>
<div class="relative w-auto flex-shrink-0 h-full group"> <div class="relative w-auto flex-shrink-0 h-full group">
<div class="w-full h-full rounded bg-gray-800 translate-y-1 translate-x-1 absolute inset-0 z-10"></div> <div class="w-full h-full rounded bg-gray-800 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
<button type="submit" <button type="submit"
class="py-3.5 rounded px-6 group-hover:-translate-y-px group-hover:-translate-x-px ease-out duration-300 z-20 relative w-full border-[3px] border-gray-900 font-medium bg-[#ffc480] tracking-wide text-lg flex-shrink-0 text-gray-900"> class="py-3.5 rounded px-6 group-hover:-translate-y-px group-hover:-translate-x-px ease-out duration-300 z-20 relative w-full border-[3px] border-gray-900 font-medium bg-[#ffc480] tracking-wide text-lg flex-shrink-0 text-gray-900">
Ingest Ingest
</button> </button>
</div> </div>
<input type="hidden" name="pattern_type" value="exclude"> <input type="hidden" name="pattern_type" value="exclude">
<input type="hidden" name="pattern" value=""> <input type="hidden" name="pattern" value="">
</form> </form>
<div class="mt-4 relative z-20 flex flex-wrap gap-4 items-start"> <div class="mt-4 relative z-20 flex flex-wrap gap-4 items-start">
<!-- Pattern selector --> <!-- Pattern selector -->
<div class="w-[200px] sm:w-[250px] mr-9 mt-4"> <div class="w-[200px] sm:w-[250px] mr-9 mt-4">
@ -31,74 +33,62 @@
<div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div> <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
<div class="flex relative z-20 border-[3px] border-gray-900 rounded bg-white"> <div class="flex relative z-20 border-[3px] border-gray-900 rounded bg-white">
<div class="relative flex items-center"> <div class="relative flex items-center">
<select id="pattern_type" <select id="pattern_type"
name="pattern_type" name="pattern_type"
class="w-21 py-2 pl-2 pr-6 appearance-none bg-[#e6e8eb] focus:outline-none border-r-[3px] border-gray-900"> class="w-21 py-2 pl-2 pr-6 appearance-none bg-[#e6e8eb] focus:outline-none border-r-[3px] border-gray-900">
<option value="exclude" {% if pattern_type == 'exclude' or not pattern_type %}selected{% endif %}>Exclude</option> <option value="exclude"
{% if pattern_type == 'exclude' or not pattern_type %}selected{% endif %}>
Exclude
</option>
<option value="include" {% if pattern_type == 'include' %}selected{% endif %}>Include</option> <option value="include" {% if pattern_type == 'include' %}selected{% endif %}>Include</option>
</select> </select>
<svg class="absolute right-2 w-4 h-4 pointer-events-none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <svg class="absolute right-2 w-4 h-4 pointer-events-none"
<polyline points="6 9 12 15 18 9"/> xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round">
<polyline points="6 9 12 15 18 9" />
</svg> </svg>
</div> </div>
<input type="text" <input type="text"
id="pattern" id="pattern"
name="pattern" name="pattern"
placeholder="*.md, src/ " placeholder="*.md, src/ "
value="{{ pattern if pattern else '' }}" value="{{ pattern if pattern else '' }}"
class=" py-2 px-2 bg-[#E8F0FE] focus:outline-none w-full"> class=" py-2 px-2 bg-[#E8F0FE] focus:outline-none w-full">
</div> </div>
</div> </div>
</div> </div>
<div class="w-[200px] sm:w-[200px] mt-3"> <div class="w-[200px] sm:w-[200px] mt-3">
<label for="file_size" class="block text-gray-700 mb-1">Include files under: <span id="size_value" class="font-bold">50kb</span></label> <label for="file_size" class="block text-gray-700 mb-1">
<input type="range" Include files under: <span id="size_value" class="font-bold">50kb</span>
id="file_size" </label>
name="max_file_size" <input type="range"
min="0" id="file_size"
max="500" name="max_file_size"
required min="0"
value="{{ default_file_size }}" max="500"
class="w-full h-3 required
bg-[#FAFAFA] value="{{ default_file_size }}"
bg-no-repeat class="w-full h-3 bg-[#FAFAFA] bg-no-repeat bg-[length:50%_100%] bg-[#ebdbb7] appearance-none border-[3px] border-gray-900 rounded-sm focus:outline-none bg-gradient-to-r from-[#FE4A60] to-[#FE4A60] [&::-webkit-slider-thumb]:w-5 [&::-webkit-slider-thumb]:h-7 [&::-webkit-slider-thumb]:appearance-none [&::-webkit-slider-thumb]:bg-white [&::-webkit-slider-thumb]:rounded-sm [&::-webkit-slider-thumb]:cursor-pointer [&::-webkit-slider-thumb]:border-solid [&::-webkit-slider-thumb]:border-[3px] [&::-webkit-slider-thumb]:border-gray-900 [&::-webkit-slider-thumb]:shadow-[3px_3px_0_#000] ">
bg-[length:50%_100%]
bg-[#ebdbb7]
appearance-none
border-[3px]
border-gray-900
rounded-sm
focus:outline-none
bg-gradient-to-r from-[#FE4A60] to-[#FE4A60]
[&::-webkit-slider-thumb]:w-5
[&::-webkit-slider-thumb]:h-7
[&::-webkit-slider-thumb]:appearance-none
[&::-webkit-slider-thumb]:bg-white
[&::-webkit-slider-thumb]:rounded-sm
[&::-webkit-slider-thumb]:cursor-pointer
[&::-webkit-slider-thumb]:border-solid
[&::-webkit-slider-thumb]:border-[3px]
[&::-webkit-slider-thumb]:border-gray-900
[&::-webkit-slider-thumb]:shadow-[3px_3px_0_#000]
">
</div> </div>
</div> </div>
{% if show_examples %} {% if show_examples %}
<!-- Example repositories section --> <!-- Example repositories section -->
<div class="mt-4"> <div class="mt-4">
<p class="opacity-70 mb-1">Try these example repositories:</p> <p class="opacity-70 mb-1">Try these example repositories:</p>
<div class="flex flex-wrap gap-2"> <div class="flex flex-wrap gap-2">
{% for example in examples %} {% for example in examples %}
<button onclick="submitExample('{{ example.url }}')" <button onclick="submitExample('{{ example.url }}')"
class="px-4 py-1 bg-[#EBDBB7] hover:bg-[#FFC480] text-gray-900 rounded transition-colors duration-200 border-[3px] border-gray-900 relative hover:-translate-y-px hover:-translate-x-px"> class="px-4 py-1 bg-[#EBDBB7] hover:bg-[#FFC480] text-gray-900 rounded transition-colors duration-200 border-[3px] border-gray-900 relative hover:-translate-y-px hover:-translate-x-px">
{{ example.name }} {{ example.name }}
</button> </button>
{% endfor %} {% endfor %}
</div>
</div> </div>
</div>
{% endif %} {% endif %}
</div> </div>
</div> </div>

30
src/templates/components/navbar.jinja
View file

@ -21,7 +21,6 @@
fetchGitHubStars(); fetchGitHubStars();
</script> </script>
<header class="sticky top-0 bg-[#FFFDF8] border-b-[3px] border-gray-900 z-50"> <header class="sticky top-0 bg-[#FFFDF8] border-b-[3px] border-gray-900 z-50">
<div class="max-w-4xl mx-auto px-4"> <div class="max-w-4xl mx-auto px-4">
<div class="flex justify-between items-center h-16"> <div class="flex justify-between items-center h-16">
@ -33,24 +32,29 @@
</a> </a>
</h1> </h1>
</div> </div>
<!-- Navigation with updated styling --> <!-- Navigation with updated styling -->
<nav class="flex items-center space-x-6"> <nav class="flex items-center space-x-6">
<a href="/api" class="text-gray-900 hover:-translate-y-0.5 transition-transform">API</a> <a href="/api"
class="text-gray-900 hover:-translate-y-0.5 transition-transform">API</a>
<div class="flex items-center gap-2"> <div class="flex items-center gap-2">
<a href="https://github.com/cyclotruc/gitingest" target="_blank" rel="noopener noreferrer" <a href="https://github.com/cyclotruc/gitingest"
class="text-gray-900 hover:-translate-y-0.5 transition-transform flex items-center gap-1.5"> target="_blank"
<svg class="w-4 h-4" fill="currentColor" viewBox="0 0 24 24" aria-hidden="true"> rel="noopener noreferrer"
<path fill-rule="evenodd" class="text-gray-900 hover:-translate-y-0.5 transition-transform flex items-center gap-1.5">
d="M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z" <svg class="w-4 h-4"
clip-rule="evenodd"></path> fill="currentColor"
viewBox="0 0 24 24"
aria-hidden="true">
<path fill-rule="evenodd" d="M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z" clip-rule="evenodd">
</path>
</svg> </svg>
GitHub GitHub
</a> </a>
<div class="flex items-center text-sm text-gray-600"> <div class="flex items-center text-sm text-gray-600">
<svg class="w-4 h-4 text-[#ffc480] mr-1" fill="currentColor" viewBox="0 0 20 20"> <svg class="w-4 h-4 text-[#ffc480] mr-1"
<path fill="currentColor"
d="M9.049 2.927c.3-.921 1.603-.921 1.902 0l1.07 3.292a1 1 0 00.95.69h3.462c.969 0 1.371 1.24.588 1.81l-2.8 2.034a1 1 0 00-.364 1.118l1.07 3.292c.3.921-.755 1.688-1.54 1.118l-2.8-2.034a1 1 0 00-1.175 0l-2.8 2.034c-.784.57-1.838-.197-1.539-1.118l1.07-3.292a1 1 0 00-.364-1.118L2.98 8.72c-.783-.57-.38-1.81.588-1.81h3.461a1 1 0 00.951-.69l1.07-3.292z" /> viewBox="0 0 20 20">
<path d="M9.049 2.927c.3-.921 1.603-.921 1.902 0l1.07 3.292a1 1 0 00.95.69h3.462c.969 0 1.371 1.24.588 1.81l-2.8 2.034a1 1 0 00-.364 1.118l1.07 3.292c.3.921-.755 1.688-1.54 1.118l-2.8-2.034a1 1 0 00-1.175 0l-2.8 2.034c-.784.57-1.838-.197-1.539-1.118l1.07-3.292a1 1 0 00-.364-1.118L2.98 8.72c-.783-.57-.38-1.81.588-1.81h3.461a1 1 0 00.951-.69l1.07-3.292z" />
</svg> </svg>
<span id="github-stars">0</span> <span id="github-stars">0</span>
</div> </div>
@ -58,4 +62,4 @@
</nav> </nav>
</div> </div>
</div> </div>
</header> </header>

193
src/templates/components/result.jinja
View file

@ -1,115 +1,94 @@
{% if result %} {% if result %}
<div class="mt-10" data-results> <div class="mt-10" data-results>
<div class="relative"> <div class="relative">
<div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div> <div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div>
<div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 space-y-6"> <div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 space-y-6">
<!-- Summary and Directory Structure --> <!-- Summary and Directory Structure -->
<div class="grid grid-cols-1 md:grid-cols-12 gap-6"> <div class="grid grid-cols-1 md:grid-cols-12 gap-6">
<!-- Summary Column --> <!-- Summary Column -->
<div class="md:col-span-5"> <div class="md:col-span-5">
<div class="flex justify-between items-center mb-4 py-2"> <div class="flex justify-between items-center mb-4 py-2">
<h3 class="text-lg font-bold text-gray-900">Summary</h3> <h3 class="text-lg font-bold text-gray-900">Summary</h3>
</div> </div>
<div class="relative">
<div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
<div class="relative"> <textarea class="w-full h-[160px] p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-none focus:outline-none relative z-10"
<div readonly>{{ summary }}</textarea>
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"> </div>
</div> {% if ingest_id %}
<textarea <div class="relative mt-4 inline-block group">
class="w-full h-[160px] p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-none focus:outline-none relative z-10" <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
readonly>{{ summary }}</textarea> <a href="/download/{{ ingest_id }}"
</div> class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10">
{% if ingest_id %} <svg class="w-4 h-4 mr-2"
<div class="relative mt-4 inline-block group"> fill="none"
<div stroke="currentColor"
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"> viewBox="0 0 24 24">
</div> <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4" />
<a href="/download/{{ ingest_id }}" </svg>
class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10"> Download
<svg class="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24"> </a>
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4" />
</svg>
Download
</a>
</div>
<div class="relative mt-4 inline-block group ml-4">
<div
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
</div>
<button onclick="copyFullDigest()"
class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10">
<svg class="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
</svg>
Copy all
</button>
</div>
{% endif %}
</div> </div>
<div class="relative mt-4 inline-block group ml-4">
<!-- Directory Structure Column --> <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
<div class="md:col-span-7"> <button onclick="copyFullDigest()"
<div class="flex justify-between items-center mb-4"> class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10">
<h3 class="text-lg font-bold text-gray-900">Directory Structure</h3> <svg class="w-4 h-4 mr-2"
<div class="relative group"> fill="none"
<div stroke="currentColor"
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"> viewBox="0 0 24 24">
</div> <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
<button onclick="copyText('directory-structure')" </svg>
class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2"> Copy all
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24"> </button>
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" </div>
d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" /> {% endif %}
</svg> </div>
Copy <!-- Directory Structure Column -->
</button> <div class="md:col-span-7">
</div> <div class="flex justify-between items-center mb-4">
</div> <h3 class="text-lg font-bold text-gray-900">Directory Structure</h3>
<div class="relative"> <div class="relative group">
<div <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"> <button onclick="copyText('directory-structure')"
</div> class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2">
<textarea <svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
class="directory-structure w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10 h-[215px]" <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
readonly>{{ tree }}</textarea> </svg>
</div> Copy
</button>
</div> </div>
</div> </div>
<div class="relative">
<!-- Full Digest --> <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
<div> <textarea class="directory-structure w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10 h-[215px]"
<div class="flex justify-between items-center mb-4"> readonly>{{ tree }}</textarea>
<h3 class="text-lg font-bold text-gray-900">Files Content</h3>
<div class="relative group">
<div
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
</div>
<button onclick="copyText('result-text')"
class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2">
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
</svg>
Copy
</button>
</div>
</div>
<div class="relative">
<div
class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
</div>
<textarea
class="result-text w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10"
style="min-height: {{ '600px' if content else 'calc(100vh-800px)' }}"
readonly>{{ content }}</textarea>
</div>
</div> </div>
</div> </div>
</div> </div>
<!-- Full Digest -->
<div>
<div class="flex justify-between items-center mb-4">
<h3 class="text-lg font-bold text-gray-900">Files Content</h3>
<div class="relative group">
<div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
<button onclick="copyText('result-text')"
class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2">
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
</svg>
Copy
</button>
</div>
</div>
<div class="relative">
<div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0"></div>
<textarea class="result-text w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10"
style="min-height: {{ '600px' if content else 'calc(100vh-800px)' }}"
readonly>{{ content }}</textarea>
</div>
</div>
</div>
</div>
</div> </div>
{% endif %} {% endif %}

50
src/templates/github.jinja
View file

@ -1,39 +1,33 @@
{% extends "base.jinja" %} {% extends "base.jinja" %}
{% block content %} {% block content %}
{% if error_message %} {% if error_message %}
<div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700" id="error-message" <div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700"
data-message="{{ error_message }}"> id="error-message"
{{ error_message }} data-message="{{ error_message }}">{{ error_message }}</div>
</div> {% endif %}
{% endif %} {% with is_index=true, show_examples=false %}
{% include 'components/github_form.jinja' %}
{% with is_index=true, show_examples=false %} {% endwith %}
{% include 'components/github_form.jinja' %} {% if loading %}
{% endwith %} <div class="relative mt-10">
<div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div>
{% if loading %} <div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 flex flex-col items-center space-y-4">
<div class="relative mt-10"> <div class="loader border-8 border-[#fff4da] border-t-8 border-t-[#ffc480] rounded-full w-16 h-16 animate-spin"></div>
<div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div> <p class="text-lg font-bold text-gray-900">Loading...</p>
<div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 flex flex-col items-center space-y-4"> </div>
<div class="loader border-8 border-[#fff4da] border-t-8 border-t-[#ffc480] rounded-full w-16 h-16 animate-spin"></div> </div>
<p class="text-lg font-bold text-gray-900">Loading...</p> {% endif %}
</div> {% include 'components/result.jinja' %}
</div>
{% endif %}
{% include 'components/result.jinja' %}
{% endblock content %} {% endblock content %}
{% block extra_scripts %} {% block extra_scripts %}
<script> <script>
document.addEventListener('DOMContentLoaded', function() { document.addEventListener('DOMContentLoaded', function() {
const urlInput = document.getElementById('input_text'); const urlInput = document.getElementById('input_text');
const form = document.getElementById('ingestForm'); const form = document.getElementById('ingestForm');
if (urlInput && urlInput.value.trim() && form) { if (urlInput && urlInput.value.trim() && form) {
// Wait for stars to be loaded before submitting // Wait for stars to be loaded before submitting
waitForStars().then(() => { waitForStars().then(() => {
const submitEvent = new SubmitEvent('submit', { const submitEvent = new SubmitEvent('submit', {
cancelable: true, cancelable: true,
bubbles: true bubbles: true
}); });
@ -59,5 +53,5 @@
checkStars(); checkStars();
}); });
} }
</script> </script>
{% endblock extra_scripts %} {% endblock extra_scripts %}

104
src/templates/index.jinja
View file

@ -1,67 +1,57 @@
{% extends "base.jinja" %} {% extends "base.jinja" %}
{% block extra_head %} {% block extra_head %}
<script> <script>
function submitExample(repoName) { function submitExample(repoName) {
const input = document.getElementById('input_text'); const input = document.getElementById('input_text');
input.value = repoName; input.value = repoName;
input.focus(); input.focus();
} }
</script> </script>
{% endblock %} {% endblock %}
{% block content %} {% block content %}
<div class="mb-12"> <div class="mb-12">
<div <div class="relative w-full mx-auto flex sm:flex-row flex-col justify-center items-start sm:items-center">
class="relative w-full mx-auto flex sm:flex-row flex-col justify-center items-start sm:items-center"> <svg class="h-auto w-16 sm:w-20 md:w-24 flex-shrink-0 p-2 md:relative sm:absolute lg:absolute left-0 lg:-translate-x-full lg:ml-32 md:translate-x-10 sm:-translate-y-16 md:-translate-y-0 -translate-x-2 lg:-translate-y-10"
<svg class="h-auto w-16 sm:w-20 md:w-24 flex-shrink-0 p-2 md:relative sm:absolute lg:absolute left-0 lg:-translate-x-full lg:ml-32 md:translate-x-10 sm:-translate-y-16 md:-translate-y-0 -translate-x-2 lg:-translate-y-10" viewBox="0 0 91 98"
viewBox="0 0 91 98" fill="none" xmlns="http://www.w3.org/2000/svg"> fill="none"
<path xmlns="http://www.w3.org/2000/svg">
d="m35.878 14.162 1.333-5.369 1.933 5.183c4.47 11.982 14.036 21.085 25.828 24.467l5.42 1.555-5.209 2.16c-11.332 4.697-19.806 14.826-22.888 27.237l-1.333 5.369-1.933-5.183C34.56 57.599 24.993 48.496 13.201 45.114l-5.42-1.555 5.21-2.16c11.331-4.697 19.805-14.826 22.887-27.237Z" <path d="m35.878 14.162 1.333-5.369 1.933 5.183c4.47 11.982 14.036 21.085 25.828 24.467l5.42 1.555-5.209 2.16c-11.332 4.697-19.806 14.826-22.888 27.237l-1.333 5.369-1.933-5.183C34.56 57.599 24.993 48.496 13.201 45.114l-5.42-1.555 5.21-2.16c11.331-4.697 19.805-14.826 22.887-27.237Z" fill="#FE4A60" stroke="#000" stroke-width="3.445">
fill="#FE4A60" stroke="#000" stroke-width="3.445"></path> </path>
<path <path d="M79.653 5.729c-2.436 5.323-9.515 15.25-18.341 12.374m9.197 16.336c2.6-5.851 10.008-16.834 18.842-13.956m-9.738-15.07c-.374 3.787 1.076 12.078 9.869 14.943M70.61 34.6c.503-4.21-.69-13.346-9.49-16.214M14.922 65.967c1.338 5.677 6.372 16.756 15.808 15.659M18.21 95.832c-1.392-6.226-6.54-18.404-15.984-17.305m12.85-12.892c-.41 3.771-3.576 11.588-12.968 12.681M18.025 96c.367-4.21 3.453-12.905 12.854-14" stroke="#000" stroke-width="2.548" stroke-linecap="round">
d="M79.653 5.729c-2.436 5.323-9.515 15.25-18.341 12.374m9.197 16.336c2.6-5.851 10.008-16.834 18.842-13.956m-9.738-15.07c-.374 3.787 1.076 12.078 9.869 14.943M70.61 34.6c.503-4.21-.69-13.346-9.49-16.214M14.922 65.967c1.338 5.677 6.372 16.756 15.808 15.659M18.21 95.832c-1.392-6.226-6.54-18.404-15.984-17.305m12.85-12.892c-.41 3.771-3.576 11.588-12.968 12.681M18.025 96c.367-4.21 3.453-12.905 12.854-14" </path>
stroke="#000" stroke-width="2.548" stroke-linecap="round"></path> </svg>
</svg> <h1 class="text-4xl sm:text-5xl sm:pt-20 lg:pt-5 md:text-6xl lg:text-7xl font-bold tracking-tighter w-full inline-block text-left md:text-center relative">
<h1 Prompt-friendly
class="text-4xl sm:text-5xl sm:pt-20 lg:pt-5 md:text-6xl lg:text-7xl font-bold tracking-tighter w-full inline-block text-left md:text-center relative"> <br>
Prompt-friendly <br>codebase&nbsp; codebase&nbsp;
</h1> </h1>
<svg class="w-16 lg:w-20 h-auto lg:absolute flex-shrink-0 right-0 bottom-0 md:block hidden translate-y-10 md:translate-y-20 lg:translate-y-4 lg:-translate-x-12 -translate-x-10" <svg class="w-16 lg:w-20 h-auto lg:absolute flex-shrink-0 right-0 bottom-0 md:block hidden translate-y-10 md:translate-y-20 lg:translate-y-4 lg:-translate-x-12 -translate-x-10"
viewBox="0 0 92 80" fill="none" xmlns="http://www.w3.org/2000/svg"> viewBox="0 0 92 80"
<path fill="none"
d="m35.213 16.953.595-5.261 2.644 4.587a35.056 35.056 0 0 0 26.432 17.33l5.261.594-4.587 2.644A35.056 35.056 0 0 0 48.23 63.28l-.595 5.26-2.644-4.587a35.056 35.056 0 0 0-26.432-17.328l-5.261-.595 4.587-2.644a35.056 35.056 0 0 0 17.329-26.433Z" xmlns="http://www.w3.org/2000/svg">
fill="#5CF1A4" stroke="#000" stroke-width="2.868" class=""></path> <path d="m35.213 16.953.595-5.261 2.644 4.587a35.056 35.056 0 0 0 26.432 17.33l5.261.594-4.587 2.644A35.056 35.056 0 0 0 48.23 63.28l-.595 5.26-2.644-4.587a35.056 35.056 0 0 0-26.432-17.328l-5.261-.595 4.587-2.644a35.056 35.056 0 0 0 17.329-26.433Z" fill="#5CF1A4" stroke="#000" stroke-width="2.868" class="">
<path </path>
d="M75.062 40.108c1.07 5.255 1.072 16.52-7.472 19.54m7.422-19.682c1.836 2.965 7.643 8.14 16.187 5.121-8.544 3.02-8.207 15.23-6.971 20.957-1.97-3.343-8.044-9.274-16.588-6.254M12.054 28.012c1.34-5.22 6.126-15.4 14.554-14.369M12.035 28.162c-.274-3.487-2.93-10.719-11.358-11.75C9.104 17.443 14.013 6.262 15.414.542c.226 3.888 2.784 11.92 11.212 12.95" <path d="M75.062 40.108c1.07 5.255 1.072 16.52-7.472 19.54m7.422-19.682c1.836 2.965 7.643 8.14 16.187 5.121-8.544 3.02-8.207 15.23-6.971 20.957-1.97-3.343-8.044-9.274-16.588-6.254M12.054 28.012c1.34-5.22 6.126-15.4 14.554-14.369M12.035 28.162c-.274-3.487-2.93-10.719-11.358-11.75C9.104 17.443 14.013 6.262 15.414.542c.226 3.888 2.784 11.92 11.212 12.95" stroke="#000" stroke-width="2.319" stroke-linecap="round">
stroke="#000" stroke-width="2.319" stroke-linecap="round"></path> </path>
</svg> </svg>
</div>
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-8">
Turn any Git repository into a simple text ingest of its codebase.
</p>
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-0">
This is useful for feeding a codebase into any LLM.
</p>
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-2">
You can also replace 'hub' with 'ingest' in any Github URL
</p>
</div> </div>
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-8"> {% if error_message %}
Turn any Git repository into a simple text ingest of its codebase. <div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700"
</p> id="error-message"
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-0"> data-message="{{ error_message }}">{{ error_message }}</div>
This is useful for feeding a codebase into any LLM. {% endif %}
</p> {% with is_index=true, show_examples=true %}
<p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-2"> {% include 'components/github_form.jinja' %}
You can also replace 'hub' with 'ingest' in any Github URL {% endwith %}
</p> {% include 'components/result.jinja' %}
</div> {% endblock %}
{% if error_message %}
<div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700" id="error-message"
data-message="{{ error_message }}">
{{ error_message }}
</div>
{% endif %}
{% with is_index=true, show_examples=true %}
{% include 'components/github_form.jinja' %}
{% endwith %}
{% include 'components/result.jinja' %}
{% endblock %}