Python 
1 from langchain.text_splitter import RecursiveCharacterTextSplitter
2
3 # Function to split text with fixed-size chunks and overlap
4 def split_text_with_overlap(text, chunk_size, overlap_size):
5    # Create a text splitter with overlap
6    text_splitter = RecursiveCharacterTextSplitter(
7        chunk_size=chunk_size, 
8        chunk_overlap=overlap_size
9    )
10    
11    # Split the text
12    chunks = text_splitter.split_text(text)
13   
14    return chunks
15
16 # Example text
17 text = """Artificial Intelligence (AI) simulates human intelligence in machines for tasks like visual perception, speech recognition, and language translation. It has evolved from rule-based systems to data-driven models, enhancing performance through machine learning and deep learning."""
18
19 # Define chunk size and overlap size
20 chunk_size = 80  # 80 characters per chunk
21 overlap_size = 10  # 10 characters overlap between chunks
22
23 # Get the chunks with overlap
24 chunks = split_text_with_overlap(text, chunk_size, overlap_size)
25
26 # Print the chunks and overlaps
27 for i in range(len(chunks)):
28    print(f"Chunk {i+1}:")
29    print(chunks[i])  # Print the chunk itself
30    
31    # If there's a next chunk, print the overlap between current and next chunk
32    if i < len(chunks) - 1:
33        overlap = chunks[i][-overlap_size:]  # Get the overlap part
34        print(f"Overlap with Chunk {i+2}:")
35        print(overlap)
36    
37    print("\n" + "="*50 + "\n")
執行上述代碼后，它將生成以下輸出：
HTML 
1 Chunk 1:
2 Artificial Intelligence (AI) simulates human intelligence in machines for tasks
3 Overlap with Chunk 2:
4 for tasks
5
6 ==================================================
7
8 Chunk 2:
9 for tasks like visual perception, speech recognition, and language translation.
10 Overlap with Chunk 3:
11 anslation.
12
13 ==================================================
14
15 Chunk 3:
16 It has evolved from rule-based systems to data-driven models, enhancing
17 Overlap with Chunk 4:
18  enhancing
19
20 ==================================================
21
22 Chunk 4:
23 enhancing performance through machine learning and deep learning.

Python 
1 from langchain.text_splitter import RecursiveCharacterTextSplitter
2
3 # Function to split text into chunks using recursive chunking
4 def split_text_recursive(text, chunk_size=80):
5    # Initialize the RecursiveCharacterTextSplitter
6    text_splitter = RecursiveCharacterTextSplitter(
7        chunk_size=chunk_size,  # Maximum size of each chunk (80 words)
8        chunk_overlap=0         # No overlap between chunks
9    )
10    
11    # Split the text into chunks
12    chunks = text_splitter.split_text(text)
13    
14    return chunks
15
16 # Example text
17 text = """Artificial Intelligence (AI) simulates human intelligence in machines for tasks like visual perception, speech recognition, and language translation. It has evolved from rule-based systems to data-driven models, enhancing performance through machine learning and deep learning."""
18
19 # Split the text using recursive chunking
20 chunks = split_text_recursive(text, chunk_size=80)
21
22 # Print the resulting chunks
23 for i, chunk in enumerate(chunks):
24    print(f"Chunk {i+1}:")
25    print(chunk)
26    print("="*50)

HTML 
1 Chunk 1:
2 Artificial Intelligence (AI) simulates human intelligence in machines for tasks
3 ==================================================
4 Chunk 2:
5 like visual perception, speech recognition, and language translation. It has
6 ==================================================
7 Chunk 3:
8 evolved from rule-based systems to data-driven models, enhancing performance
9 ==================================================
10 Chunk 4:
11 through machine learning and deep learning.

Python 
1 import os
2 from langchain_experimental.text_splitter import SemanticChunker
3 from langchain_openai.embeddings import OpenAIEmbeddings
4
5 # Set the OpenAI API key as an environment variable (Replace with your actual API key)
6 os.environ["OPENAI_API_KEY"] = "replace with your actual OpenAI API key" 
7
8 # Function to split text into semantic chunks
9 def split_text_semantically(text, breakpoint_type="percentile"):
10    # Initialize the SemanticChunker with OpenAI embeddings
11    text_splitter = SemanticChunker(OpenAIEmbeddings(), breakpoint_threshold_type=breakpoint_type)
12    
13    # Create documents (chunks)
14    docs = text_splitter.create_documents([text])
15    
16    # Return the list of chunks
17    return [doc.page_content for doc in docs]
18
19 def main():
20    # Example content (State of the Union address or your own text)
21    document_content = """
22 Artificial Intelligence (AI) simulates human intelligence in machines for tasks like visual perception, speech recognition, and language translation. It has evolved from rule-based systems to data-driven models, enhancing performance through machine learning and deep learning.
23
24 Prompt Engineering involves designing input prompts to guide AI models in producing accurate and relevant responses, improving tasks such as text generation and summarization.
25    """
26    
27    # Split text using the chosen threshold type (percentile)
28    threshold_type = "percentile"
29    print(f"\nChunks using {threshold_type} threshold:")
30    chunks = split_text_semantically(document_content, breakpoint_type=threshold_type)
31    
32    # Print each chunk's content
33    for idx, chunk in enumerate(chunks):
34        print(f"Chunk {idx + 1}:\n{chunk}\n")
35        
36 if __name__ == "__main__":
37    main()

HTML 
1 Chunks using percentile threshold:
2 Chunk 1:
3 Artificial Intelligence (AI) simulates human intelligence in machines for tasks like visual perception, speech recognition, and language translation. It has evolved from rule-based systems to data-driven models, enhancing performance through machine learning and deep learning.
4
5 Chunk 2:
6 Prompt Engineering involves designing input prompts to guide AI models in producing accurate and relevant responses, improving tasks such as text generation and summarization.

Python 
1 from langchain.chat_models import ChatOpenAI
2 from langchain.prompts import PromptTemplate
3 from langchain.chains import LLMChain
4 from langchain.agents import initialize_agent, Tool, AgentType
5
6 # Initialize OpenAI chat model (replace with your API key)
7 llm = ChatOpenAI(model="gpt-3.5-turbo", api_key="replace with your actual OpenAI API key")
8
9 # Step 1: Define Chunking and Summarization Prompt Template
10 chunk_prompt_template = """
11 You are given a large piece of text. Your job is to break it into smaller parts (chunks) if necessary and summarize each chunk.
12 Once all parts are summarized, combine them into a final summary. 
13 If the text is already small enough to process at once, provide a full summary in one step. 
14 Please summarize the following text:\n{input}
15 """
16 chunk_prompt = PromptTemplate(input_variables=["input"], template=chunk_prompt_template)
17
18 # Step 2: Define Chunk Processing Tool
19 def chunk_processing_tool(query):
20    """Processes text chunks and generates summaries using the defined prompt."""
21    chunk_chain = LLMChain(llm=llm, prompt=chunk_prompt)
22    print(f"Processing chunk:\n{query}\n")  # Show the chunk being processed
23    return chunk_chain.run(input=query)
24
25 # Step 3: Define External Tool (Optional, can be used to fetch extra information if needed)
26 def external_tool(query):
27    """Simulates an external tool that could fetch additional information."""
28    return f"External response based on the query: {query}"
29
30 # Step 4: Initialize the agent with tools
31 tools = [
32    Tool(
33        name="Chunk Processing",
34        func=chunk_processing_tool,
35        description="Processes text chunks and generates summaries."
36    ),
37    Tool(
38        name="External Query",
39        func=external_tool,
40        description="Fetches additional data to enhance chunk processing."
41    )
42 ]
43
44 # Initialize the agent with defined tools and zero-shot capabilities
45 agent = initialize_agent(
46    tools=tools,
47    agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
48    llm=llm,
49    verbose=True
50 )
51
52 # Step 5: Agentic Chunk Processing Function
53 def agent_process_chunks(text):
54    """Uses the agent to process text chunks and generate a final output."""
55    # Step 1: Chunking the text into smaller, manageable sections
56    def chunk_text(text, chunk_size=500):
57        """Splits large text into smaller chunks."""
58        return [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]
59
60    chunks = chunk_text(text)
61
62    # Step 2: Process each chunk with the agent
63    chunk_results = []
64    for idx, chunk in enumerate(chunks):
65        print(f"Processing chunk {idx + 1}/{len(chunks)}...")
66        response = agent.invoke({"input": chunk})  # Process chunk using the agent
67        chunk_results.append(response['output'])  # Collect the chunk result
68
69    # Step 3: Combine the chunk results into a final output
70    final_output = "\n".join(chunk_results)
71    return final_output
72
73 # Step 6: Running the agent on an example large text input
74 if __name__ == "__main__":
75    # Example large text content
76    text_to_process = """
77    Artificial intelligence (AI) is transforming industries by enabling machines to perform tasks that
78    previously required human intelligence. From healthcare to finance, AI is driving innovation and improving
79    efficiency. For instance, in healthcare, AI algorithms assist doctors in diagnosing diseases, interpreting
80    medical images, and predicting patient outcomes. Meanwhile, in finance, AI helps detect fraud, manage
81    investments, and automate customer service.
82
83    However, the widespread adoption of AI also raises ethical concerns. Issues like privacy invasion,
84    algorithmic bias, and the potential loss of jobs due to automation are significant challenges. Experts
85    argue that it's essential to develop AI responsibly to ensure that it benefits society as a whole.
86    Proper regulations, transparency, and accountability can help address these issues, ensuring that AI
87    technologies are used for the greater good.
88
89    Beyond individual industries, AI is also impacting the global economy. Nations are investing heavily
90    in AI research and development to maintain a competitive edge. This technological race could redefine
91    global power dynamics, with countries that excel in AI leading the way in economic and military strength.
92    Despite the potential for AI to contribute positively to society, its development and application require
93    careful consideration of ethical, legal, and societal implications.
94    """
95
96    # Process the text and print the final result
97    final_result = agent_process_chunks(text_to_process)
98    print("\nFinal Output:\n", final_result)

HTML 
1 Processing chunk 1/3...
2
3
4 > Entering new AgentExecutor chain...
5 I should use Chunk Processing to extract the key information from the text provided.
6 Action: Chunk Processing
7 Action Input: Artificial intelligence (AI) is transforming industries by enabling machines to perform tasks that previously required human intelligence. From healthcare to finance, AI is driving innovation and improving efficiency. For instance, in healthcare, AI algorithms assist doctors in diagnosing diseases, interpreting medical images, and predicting patient outcomes. Meanwhile, in finance, AI helps detect fraud, manage investments, and automate customer service.Processing chunk:
8 Artificial intelligence (AI) is transforming industries by enabling machines to perform tasks that previously required human intelligence. From healthcare to finance, AI is driving innovation and improving efficiency. For instance, in healthcare, AI algorithms assist doctors in diagnosing diseases, interpreting medical images, and predicting patient outcomes. Meanwhile, in finance, AI helps detect fraud, manage investments, and automate customer service.
9
10 Observation: Artificial intelligence (AI) is revolutionizing various industries by allowing machines to complete tasks that once needed human intelligence. In healthcare, AI algorithms aid doctors in diagnosing illnesses, analyzing medical images, and forecasting patient results. In finance, AI is used to identify fraud, oversee investments, and streamline customer service. AI is playing a vital role in enhancing efficiency and driving innovation across different sectors.
11 Thought:I need more specific information about the impact of AI in different industries.
12 Action: External Query
13 Action Input: Impact of artificial intelligence in healthcare
14 Observation: External response based on the query: Impact of artificial intelligence in healthcare
15 Thought:I should now look for information on the impact of AI in finance.
16 Action: External Query
17 Action Input: Impact of artificial intelligence in finance
18 Observation: External response based on the query: Impact of artificial intelligence in finance
19 Thought:I now have a better understanding of how AI is impacting healthcare and finance.
20 Final Answer: Artificial intelligence is revolutionizing industries like healthcare and finance by enhancing efficiency, driving innovation, and enabling machines to perform tasks that previously required human intelligence. In healthcare, AI aids in diagnosing diseases, interpreting medical images, and predicting patient outcomes, while in finance, it helps detect fraud, manage investments, and automate customer service.
21
22 > Finished chain.
23 Processing chunk 2/3...
24
25 > Entering new AgentExecutor chain...
26 This question is discussing ethical concerns related to the widespread adoption of AI and the need to develop AI responsibly.
27 Action: Chunk Processing
28 Action Input: The text providedProcessing chunk:
29 The text provided
30
31 Observation: I'm sorry, but you haven't provided any text to be summarized. Could you please provide the text so I can assist you with summarizing it?
32 Thought:I need to provide the text for chunk processing to summarize.
33 Action: External Query
34 Action Input: Retrieve the text related to the ethical concerns of AI adoption and responsible development
35 Observation: External response based on the query: Retrieve the text related to the ethical concerns of AI adoption and responsible development
36 Thought:Now that I have the text related to ethical concerns of AI adoption and responsible development, I can move forward with chunk processing.
37 Action: Chunk Processing
38 Action Input: The retrieved textProcessing chunk:
39 The retrieved text
40
41 Observation: I'm sorry, but it seems like you have not provided any text for me to summarize. Could you please provide the text you would like me to summarize? Thank you!
42 Thought:I need to ensure that the text related to ethical concerns of AI adoption and responsible development is provided for chunk processing to generate a summary.
43 Action: External Query
44 Action Input: Retrieve the text related to the ethical concerns of AI adoption and responsible development
45 Observation: External response based on the query: Retrieve the text related to the ethical concerns of AI adoption and responsible development
46 Thought:Now that I have the text related to ethical concerns of AI adoption and responsible development, I can proceed with chunk processing to generate a summary.
47 Action: Chunk Processing
48 Action Input: The retrieved textProcessing chunk:
49 The retrieved text
50
51 Observation: I'm sorry, but you haven't provided any text to be summarized. Can you please provide the text so I can help you with the summarization?
52 Thought:I need to make sure that the text related to ethical concerns of AI adoption and responsible development is entered for chunk processing to summarize.
53 Action: Chunk Processing
54 Action Input: Text related to ethical concerns of AI adoption and responsible developmentProcessing chunk:
55 Text related to ethical concerns of AI adoption and responsible development
56
57 Observation: The text discusses the ethical concerns surrounding the adoption of artificial intelligence (AI) and the importance of responsible development. It highlights issues such as bias in AI algorithms, privacy violations, and the potential for autonomous AI systems to make harmful decisions. The text emphasizes the need for transparency, accountability, and ethical guidelines to ensure that AI technologies are developed and deployed in a responsible manner.
58 Thought:The text provides information on ethical concerns related to AI adoption and responsible development, emphasizing the need for regulation, transparency, and accountability. 
59 Final Answer: The text discusses the ethical concerns surrounding the adoption of artificial intelligence (AI) and the importance of responsible development.
60
61 > Finished chain.
62 Processing chunk 3/3...
63
64 > Entering new AgentExecutor chain...
65 This question seems to be about the impact of AI on the global economy and the potential implications.
66 Action: Chunk Processing
67 Action Input: The text providedProcessing chunk:
68 The text provided
69
70 Observation: I'm sorry, but you did not provide any text for me to summarize. Please provide the text that you would like me to summarize.
71 Thought:I need to provide the text for Chunk Processing to summarize.
72 Action: External Query
73 Action Input: Fetch the text about the impact of AI on the global economy and its implications.
74 Observation: External response based on the query: Fetch the text about the impact of AI on the global economy and its implications.
75 Thought:Now that I have the text about the impact of AI on the global economy and its implications, I can proceed with Chunk Processing.
76 Action: Chunk Processing
77 Action Input: The text about the impact of AI on the global economy and its implications.Processing chunk:
78 The text about the impact of AI on the global economy and its implications.
79
80 Observation: The text discusses the significant impact that artificial intelligence (AI) is having on the global economy. It highlights how AI is revolutionizing industries by increasing productivity, reducing costs, and creating new job opportunities. However, there are concerns about job displacement and the need for retraining workers to adapt to the changing landscape. Overall, AI is reshaping the economy and prompting a shift in the way businesses operate.
81 Thought:Based on the summary generated by Chunk Processing, the impact of AI on the global economy seems to be significant, with both positive and negative implications.
82 Final Answer: The impact of AI on the global economy is significant, revolutionizing industries, increasing productivity, reducing costs, creating new job opportunities, but also raising concerns about job displacement and the need for worker retraining.
83
84 > Finished chain.
85
86 Final Output:
87  Artificial intelligence is revolutionizing industries like healthcare and finance by enhancing efficiency, driving innovation, and enabling machines to perform tasks that previously required human intelligence. In healthcare, AI aids in diagnosing diseases, interpreting medical images, and predicting patient outcomes, while in finance, it helps detect fraud, manage investments, and automate customer service.
88 The text discusses the ethical concerns surrounding the adoption of artificial intelligence (AI) and the importance of responsible development.
89 The impact of AI on the global economy is significant, revolutionizing industries, increasing productivity, reducing costs, creating new job opportunities, but also raising concerns about job displacement and the need for worker retraining.