4 commits

Author	SHA1	Message	Date
Adrien Gallouët	aff6eb6e75	tokenize : drop --stdin mutual-exclusion check (#25672) ... match cli and completion, which don't enforce it	2026-07-15 18:41:51 +02:00
Adrien Gallouët	8ff8c4299d	tokenize : align usage by using common args (#25516) ... Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin, --no-bos, --no-parse-special and --show-count as common args. parse_special defaults to true for TOKENIZE to preserve the old behavior. Errors now go through LOG_ERR instead of fprintf(stderr). Signed-off-by: Adrien Gallouët <angt@huggingface.co>	2026-07-14 15:20:53 +02:00
SamareshSingh	cb8f4fa3f8	Fix locale-dependent float printing in GGUF metadata (#17331) ... * Set C locale for consistent float formatting across all binaries. * Add C locale setting to all tools binaries Add std::setlocale(LC_NUMERIC, "C") to all 16 binaries in the tools/ directory to ensure consistent floating-point formatting. * Apply suggestion from @JohannesGaessler --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>	2026-03-04 09:30:40 +01:00
Diego Devesa	1d36b3670b	llama : move end-user examples to tools directory (#13249) ... * llama : move end-user examples to tools directory --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>	2025-05-02 20:27:13 +02:00

Renamed from examples/tokenize/tokenize.cpp (Browse further)