AVRELIO
OSS · SEMANTIC-CHUNKERS 0.0.10 · MIT · PYTHON 3.9+ →
C · INGEST
$ semantic-chunkers --about

Chunk by meaning, not by length.

Semantic Chunkers is a multi-modal chunking library for intelligent chunking of text, video, and audio. It makes your AI and data processing more efficient and accurate — retrieval returns whole thoughts, not fragments.

View on GitHub →Read the docs
SEMANTIC-CHUNKERS · CHUNK LABDOC · MAMBA.PDF
CHUNKER

Our most robust method. Uses a varying similarity threshold to find dynamic, local splits — and finds a good threshold value on its own.

text only · balances accuracy and efficiency
No threshold to set — the StatisticalChunker derives a varying, local threshold from the document itself.
CHUNKS
5
AVG TOKENS
73
Split 1, tokens 39triggered by: 0.22
# Mamba: Linear-Time Sequence Modeling with Selective State Spaces # Albert Gu*1 and Tri Dao*2 1Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University
Split 2, tokens 118triggered by: token limit
# Abstract Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers’ computational inefficiency on long sequences. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities.
Split 3, tokens 72triggered by: 0.21
Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length.
Split 4, tokens 87triggered by: 0.27
# 1 Introduction Foundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an effective paradigm in modern machine learning. The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains. However, this property brings fundamental drawbacks: an inability to model anything outside of a finite window, and quadratic scaling with respect to the window length.
Split 5, tokens 47triggered by: final split
Recently, structured state space sequence models (SSMs) have emerged as a promising class of architectures for sequence modeling. These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs).
chunk.pySTATISTICAL
from semantic_router.encoders import OpenAIEncoder
from semantic_chunkers import StatisticalChunker

encoder = OpenAIEncoder(
    name="text-embedding-3-small"
)
chunker = StatisticalChunker(encoder=encoder)

chunks = chunker(docs=[content])
chunker.print(chunks[0])
§ I — PICK THE TRADE-OFF4 CHUNKERS
I
Statistical

Our most robust method. Uses a varying similarity threshold to find dynamic, local splits — and finds a good threshold value on its own.

text only · balances accuracy and efficiency
II
Consecutive

The simplest form of semantic chunking: compare each split’s embedding to the next one and cut wherever similarity drops below the threshold.

multi-modal · text, video, audio
III
Cumulative

Compares the embedding of the cumulative chunk so far against the next split. More compute, less noise — good for long, drifting documents.

text · more encoder calls
IV
Regex

No encoder at all. Splits on sentence delimiters and packs splits up to max_chunk_tokens — the cheap baseline to compare against.

no embeddings · instant
§ II — WHY SEMANTIC CHUNKING3 REASONS
I
Context-aware

Semantic chunkers build more context-aware chunks of information — boundaries land where the meaning changes, not at an arbitrary character count.

II
Multi-modal

One library for text, video, and audio. The ConsecutiveChunker works on any sequence of embeddings, whatever produced them.

III
Retrieval that retrieves

Better boundaries mean less bleed between topics, so your top-k actually contains the answer.

// MULTI-MODAL

Text, video, audio — one chunker.

Text
Sentence splits merged on meaning — the default path for RAG corpora.
Video
Frame embeddings chunked with the ConsecutiveChunker to find scene boundaries.
Audio
Segment embeddings grouped into topically coherent passages.
// API — AN ENCODER, A CHUNKER, YOUR DOCSPYTHON 3.9+
pip install -qU semantic-chunkers

from semantic_router.encoders import OpenAIEncoder
from semantic_chunkers import StatisticalChunker

encoder = OpenAIEncoder(name="text-embedding-3-small")

# the statistical chunker finds a good threshold on its own
chunker = StatisticalChunker(encoder=encoder)

chunks = chunker(docs=[content])
chunker.print(chunks[0])

# Split 1, tokens 300, triggered by: token limit
# Split 2, tokens 300, triggered by: token limit
# Split 3, tokens 112, triggered by: 0.32
GET STARTED

Better chunks in nine lines.

Better boundaries mean retrieval that stays fast and accurate. Pairs directly with Semantic Router for a deterministic mesh.

FULLY OPEN SOURCE

Transparency in code, creativity in collaboration. Semantic Chunkers uses an MIT license so you can use it however and wherever you want. Find us at github.com/aurelio-labs/semantic-chunkers.

AMERICA · EVROPA · ASIA© 2026 Aurelio Labs · MIT Licensed