SemanticChunker
An implementation of a document chunking strategy that splits content according to Markdown heading levels and paragraph semantics, controls token counts while retaining overlapping windows, and stores header_path in metadata to preserve contextual information.
Timeline (last 90 days)
SemanticChunker splits documents by Markdown heading hierarchy, preserves header_path metadata, and each chunk carries 50 tokens of overlap.
在文档摄入中，使用语义分块（先按Markdown标题层级切分，再按段落聚合，控制token不超过512，保留50 token重叠）能保留header_path元数据，提升检索质量。
LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser provide out-of-the-box semantic chunking implementations.
All Facts (3)
SemanticChunker splits documents by Markdown heading hierarchy, preserves header_path metadata, and each chunk carries 50 tokens of overlap.
50%Unverified在文档摄入中，使用语义分块（先按Markdown标题层级切分，再按段落聚合，控制token不超过512，保留50 token重叠）能保留header_path元数据，提升检索质量。
50%UnverifiedLangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser provide out-of-the-box semantic chunking implementations.
50%