微信扫码
与创始人交个朋友
我要投稿
文本切分五个层级:
Level 1: Character Splitting - 简单的字符长度切分
Level 2: Recursive Character Text Splitting - 通过分隔符切分,然后递归合并
Level 3: Document Specific Splitting - 针对不同文档格式切分 (PDF, Python, Markdown)
Level 4: Semantic Splitting - 语义切分
Level 5: Agentic Splitting-使用代理实现自动切分
为什么需要Recursive Character Text Splitting
按字符长度切分的问题在于,根本没有考虑到文档的结构。只是按固定数量的字符分割,这样很容易造成文档被随意切分开了,导致最后的效果不好。
递归字符文本分割器可以帮助解决这个问题。有了它,切分的过程可描述为:我们先指定一系列分隔符,这些分隔符用于分割文档,然后将分割好的小块合并,达到指定的长度。
它是切分器中最基础的做法,也是快速搭建应用流程的首选。如果在不知道选择哪个切分器的时候,这是一个不错的选择。
我们看看在langchain中的默认分隔符["\n\n", "\n", " ", ""]:
看看下面例子,在一般情况下,我们希望的切分情况,如何能实现它?
text = """
One of the most important things I didn't understand about the world when I was a child is the degree to which the returns for performance are superlinear.
Teachers and coaches implicitly told us the returns were linear. "You get out," I heard a thousand times, "what you put in." They meant well, but this is rarely true. If your product is only half as good as your competitor's, you don't get half as many customers. You get no customers, and you go out of business.
It's obviously true that the returns for performance are superlinear in business. Some think this is a flaw of capitalism, and that if we changed the rules it would stop being true. But superlinear returns for performance are a feature of the world, not an artifact of rules we've invented. We see the same pattern in fame, power, military victories, knowledge, and even benefit to humanity. In all of these, the rich get richer. [1]
"""
可以很直观地将它切分成三个段落:
如何实现
from langchain.text_splitter import RecursiveCharacterTextSplitter
构建切分器
text_splitter = RecursiveCharacterTextSplitter(chunk_size=65, chunk_overlap=0)text_splitter.create_documents([text])# chunk_size 每个片段的字符数# chunk_overlap 片段之间的重叠字符数
text_splitter = RecursiveCharacterTextSplitter(chunk_size=450, chunk_overlap=0)text_splitter.create_documents([text])
总共生成3段,这就是我们预期的切分效果。
仅仅使用字符个数作为小块内的切分依据,似乎不是一个好的方案;相同语义下,中英文、数字等字符个数差别太大了。下篇文章我们将介绍如何优化小块内的切分和计数方案。
53AI,企业落地应用大模型首选服务商
产品:大模型应用平台+智能体定制开发+落地咨询服务
承诺:先做场景POC验证,看到效果再签署服务协议。零风险落地应用大模型,已交付160+中大型企业
2024-11-13
RAGCache:让RAG系统更高效的多级动态缓存新方案
2024-11-13
Glean:企业AI搜索,估值46亿美元,ARR一年翻4倍
2024-11-12
从安装到配置,带你跑通GraphRAG
2024-11-12
蚂蚁 KAG 框架核心功能研读
2024-11-12
【RAG】浅看引入智能信息助理提升大模型处理复杂推理任务的潜力-AssisTRAG
2024-11-12
体验完百度世界2024上的iRAG,我觉得AI绘图也可以没有幻觉了。
2024-11-12
提升RAG文档效率,10种有效策略
2024-11-12
揭秘RAG:全方位解析RAG检索中的意图识别,如何助力智能问答
2024-07-18
2024-07-09
2024-05-05
2024-07-09
2024-05-19
2024-06-20
2024-07-07
2024-07-07
2024-07-08
2024-07-09
2024-11-06
2024-11-06
2024-11-05
2024-11-04
2024-10-27
2024-10-25
2024-10-21
2024-10-21