logo
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

73 lines
1.1 KiB

# Text Spliter
*author: shiyu22*
<br />
### Description
**Text spliter** is used to split text into chunk lists.
> Refer to [Recursive Characters](https://python.langchain.com/en/latest/modules/indexes/text_splitters/examples/recursive_text_splitter.html) for the operation of splitting text.
<br />
### Code Example
```Python
from towhee import pipe, ops, DataCollection
p = (
pipe.input('url')
.map('url', 'text', ops.text_loader())
.flat_map('text', 'text', ops.text_spliter())
.output('url', 'text')
)
res = p('https://github.com/towhee-io/towhee/blob/main/README.md')
DataCollection(res).show()
```
<img src="./result.png" alt="result" height="200px"/>
<br />
## Factory Constructor
Create the operator via the following factory method
***towhee.text_loader(chunk_size=300)***
**Parameters:**
***chunk_size***: int
​ The size of each chunk, defaults to 300.
<br />
### Interface
The operator split incoming the text and return chunks.
**Parameters:**
***data***: str
​ The text data.
**Return**: List[Document]
A list of the chunked document.
1 year ago