基于NebulaGraph的知识图谱增强的大语言模型

For each chat interaction:

• first generate a standalone question from conversation context and last message, then
• query the query engine with the condensed question for a response.

1
2
3
4
5

from llama_index.chat_engine import CondenseQuestionChatEngine
self.custom_chat_engine = CondenseQuestionChatEngine.from_defaults(
	query_engine=custom_query_engine,
	service_context=service_context,	
)

1

chat_engine = index.as_chat_engine(chat_mode="condense_question", verbose=True)

此模式没有sources

For each chat interaction, the agent enter a ReAct loop:

• first decide whether to use the query engine tool and come up with appropriate input
• (optional) use the query engine tool and observe its output
• decide whether to repeat or give final response

1
2
3
4
5
6
7
8

# NOTE: lazy import
from llama_index.agent import OpenAIAgent, ReActAgent
from llama_index.tools.query_engine import QueryEngineTool
# convert query engine to tool
service_context = llama_index.global_service_context
llm = service_context.llm
query_engine_tool = QueryEngineTool.from_defaults(query_engine=custom_query_engine)
custom_chat_engine = ReActAgent.from_tools(tools=[query_engine_tool], llm=llm, verbose=True,)

1

chat_engine = index.as_chat_engine(chat_mode="react", verbose=True)

使用的LLM是ChatGPT才能使用此模式，此模式需要需要人工指定是否需要使用query_engine_tool（即是否查询文档向量）

1
2
3
4
5
6
7
8

# NOTE: lazy import
from llama_index.agent import OpenAIAgent, ReActAgent
from llama_index.tools.query_engine import QueryEngineTool
# convert query engine to tool
service_context = llama_index.global_service_context
llm = service_context.llm
query_engine_tool = QueryEngineTool.from_defaults(query_engine=custom_query_engine)
chat_engine = index.as_chat_engine(tools=[query_engine_tool], chat_mode="openai", verbose=True)

1

chat_engine = index.as_chat_engine(chat_mode="openai", verbose=True)

此模式会判断，如果 LLM是ChatGPT，并且model是function接口（text-davinci-003不是function接口，而gpt-3.5-turbo是function接口），那么会使用OpenAI Agent Mode，否则使用ReAct Agent Mode

1
2
3
4
5
6
7
8
9
10
11
12
13

# BEST 模式 如果使用的是openai 那么将不会使用文档索引器
# NOTE: lazy import
from llama_index.agent import OpenAIAgent, ReActAgent
from llama_index.tools.query_engine import QueryEngineTool
# convert query engine to tool
query_engine_tool = QueryEngineTool.from_defaults(query_engine=custom_query_engine)
# get LLM
service_context = llama_index.global_service_context
llm = service_context.llm
if isinstance(llm, OpenAI) and is_function_calling_model(llm.model):
    self.custom_chat_engine = OpenAIAgent.from_tools(tools=[query_engine_tool], llm=llm, verbose=True,)
else:
    self.custom_chat_engine = ReActAgent.from_tools(tools=[query_engine_tool], llm=llm, verbose=True,)

1

chat_engine = index.as_chat_engine(chat_mode="best", verbose=True)

compact (default): similar to refine but compact (concatenate) the chunks beforehand, resulting in less LLM calls.

Details: stuff as many text (concatenated/packed from the retrieved chunks) that can fit within the context window (considering the maximum prompt size between text_qa_template and refine_template). If the text is too long to fit in one prompt, it is splitted in as many parts as needed (using a TokenTextSplitter and thus allowing some overlap between text chunks).

Each text part is considered a “chunk” and is sent to the refine synthesizer.

In short, it is like refine, but with less LLM calls.

tree_summarize: Query the LLM using the text_qa_template prompt as many times as needed so that all concatenated chunks have been queried, resulting in as many answers that are themselves recursively used as chunks in a tree_summarize LLM call and so on, until there’s only one chunk left, and thus only one final answer.

Details: concatenate the chunks as much as possible to fit within the context window using the text_qa_template prompt, and split them if needed (again with a TokenTextSplitter and some text overlap). Then, query each resulting chunk/split against text_qa_template (there is no refine query !) and get as many answers.

If there is only one answer (because there was only one chunk), then it’s the final answer.

If there are more than one answer, these themselves are considered as chunks and sent recursively to the tree_summarize process (concatenated/splitted-to-fit/queried).

Good for summarization purposes.

simple_summarize: Truncates all text chunks to fit into a single LLM prompt. Good for quick summarization purposes, but may lose detail due to truncation.

no_text: Only runs the retriever to fetch the nodes that would have been sent to the LLM, without actually sending them. Then can be inspected by checking response.source_nodes.

accumulate: Given a set of text chunks and the query, apply the query to each text chunk while accumulating the responses into an array. Returns a concatenated string of all responses. Good for when you need to run the same query separately against each text chunk.

compact_accumulate: The same as accumulate, but will “compact” each LLM prompt similar to compact, and run the same query against each text chunk.

使用过text-davinci-003, text-davinci-002,gpt-3.5-turbo模型,目前使用gpt-3.5-turbo

免费版限速情况：

码字ing

对接其他公司封装的ChatGPT API

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

def get_api_response(prompt: str):
    url = 'https://api-tinystar.meinenghua.com/api/v1/chat/completions'
    params = {
        'questions': prompt,
        'prompt': "你是智能助手",
        'modelOptions': {
            'maxTokens': 512,
            'topP': 1
        },
        'disableSensitiveWordDetection': True
    }
    params = json.dumps(params)
    headers = {
        'Accept-Charset': 'utf-8',
        'Content-Type': 'application/json',
        'App-Key': '',
        'App-Secret': ''
        }
    # 用bytes函数转换为字节
    data = bytes(params, 'utf8')
    req = urllib.request.Request(url=url, data=data, headers=headers, method='POST')
    res_str = urllib.request.urlopen(req).read().decode('utf-8')
    res_dict = json.loads(res_str)
    resposne = res_dict['response']
    print(f'---call api resposne: {resposne}')

    return resposne

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# set context window size
context_window = 2048
# set number of output tokens
num_output = 256

class OurLLM(CustomLLM):

    @property
    def metadata(self) -> LLMMetadata:
        """Get LLM metadata."""
        return LLMMetadata(
            # context_window=context_window,
            # num_output=num_output,
            model_name="tinystar"
        )

    @llm_completion_callback()
    def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
        prompt_length = len(prompt)
        # response = pipeline(prompt, max_new_tokens=num_output)[0]["generated_text"]
        # # only return newly generated tokens
        # text = response[prompt_length:]
        text = get_api_response(prompt=prompt)

        return CompletionResponse(text=text)
    
    @llm_completion_callback()
    def stream_complete(self, prompt: str, **kwargs: Any) -> CompletionResponseGen:
        raise NotImplementedError()

1
2
3
4
5
6
7
8
9

llm = OurLLM()
# use chatgpt
# from llama_index.llms import OpenAI
# llm = OpenAI(model='gpt-3.5-turbo', max_tokens=512)
service_context = ServiceContext.from_defaults(
    llm=llm, 
    embed_model="local",
    chunk_size=512,
)

使用方式: llama-index==0.8.3 local模式 默认该模型

llama-index==0.8.3 后续计划使用该模型，中文支持更加友好

使用方式: llama-index==0.8.3

sentence-transformers中的模型

使用方式: llama-index==0.7.17 local模式 默认该模型

1
2
3
4
5

service_context = ServiceContext.from_defaults(
                llm=OpenAI(model="gpt-3.5-turbo", max_tokens=512), 
                chunk_size=512,
                embed_model="local"
                )

sentence-transformers中的模型

使用方式: llama-index==0.7.17

1
2
3
4
5
6

embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2"))
service_context = ServiceContext.from_defaults(
                llm=OpenAI(model="gpt-3.5-turbo", max_tokens=512), 
                chunk_size=512,
                embed_model=embed_model
                )

施工中

多语言版本

基于深度学习的开源中文知识图谱抽取框架

https://github.com/zjunlp/DeepKE

DeepKE-LLM是基于大语言模型知识图谱抽取框架

https://github.com/zjunlp/DeepKE/blob/main/example/llm/README_CN.md