chunking-strategy

This skill provides chunking strategies for retrieval-augmented generation (RAG) systems, vector databases, and large-document processing. It addresses a core RAG quality problem: how to split documents so that retrieval returns coherent, relevant passages. Poor chunking causes weak precision and recall, and the skill offers concrete, source-grounded guidance on chunk sizes (256-1024 tokens), overlap percentages (10-20%), and boundary detection methods rather than leaving those choices to guesswork.

It organizes chunking into five levels: fixed-size, recursive character, structure-aware (Markdown, code, tables, PDFs), semantic (embedding-based boundary detection with a similarity threshold), and advanced methods such as late chunking and contextual retrieval. It walks through an implementation pipeline (pre-process, select parameters, process and validate, evaluate and iterate) with specific tuning heuristics: reduce chunk size by 25% if precision falls below 0.7, increase overlap by 10% if recall drops below 0.6. It includes runnable Python validation snippets using sentence-transformers to measure semantic cohesion, retrieval precision, and chunk-size distribution, plus example code using LangChain's RecursiveCharacterTextSplitter and Python's `ast` module for code-aware chunking. A set of reference files covers semantic methods, advanced strategies, tools, evaluation, and visualization.

It targets ML and data engineers, RAG and search practitioners, and teams performance-tuning existing pipelines with poor retrieval quality. The declared allowed tools are Read, Write, and Bash, and the validation commands require Python libraries such as sentence-transformers, scikit-learn, and numpy.