chunking

NCRF++, a Neural Sequence Labeling Toolkit. Easy use to any sequence labeling tasks (e.g. NER, POS, Segmentation). It includes character LSTM/CNN, word LSTM/CNN and softmax/CRF components.

Content-Addressable Data Synchronization Tool

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

Alternative casync implementation

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

Fully neural approach for text chunking

Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

A new chunking strategy developed by ZeroEntropy for general semantic chunking using Llama-70B.

Visual document analysis studio powered by Docling — configure the extraction pipeline, inspect text, tables and bounding boxes in the browser, then chunk, embed and index into OpenSearch and Neo4j.

A package for parsing PDFs and analyzing their content using LLMs.

A TensorFlow implementation of Neural Sequence Labeling model, which is able to tackle sequence labeling tasks such as POS Tagging, Chunking, NER, Punctuation Restoration and etc.

Open-source toolkit for reliable RAG pipelines: convert PDFs to Markdown, clean documents, inspect chunks, compare chunking strategies, and enrich metadata for LLM applications.

Pure Rust PDF library for AI/RAG: structure-aware chunking, no ML, no C deps.

PDFStract - Extract, Chunking and Embedding Layer in Your RAG Pipeline - Available as CLI - WEBUI - API

🍱 Semantically create chunks from large document for passing to LLM workflows

A Python CLI to test, benchmark, and find the best RAG chunking strategy for your Markdown documents.

Deprecated/unmaintained Postgres extensions to support end-to-end Retrieval-Augmented Generation (RAG) pipelines

Upload your files and ask questions about them. Open-source RAG app with LangChain & Streamlit, built before ChatGPT had file uploads. Works with Anthropic & OpenRouter. Easy to extend. The newest release is a functioning tribute and future development ideas are in consideration.