[Ollama] GraphRAG Community Support for running Ollama #345
Labels
community_support
Issue handled by community members
good first issue
Good for newcomers
oss_llm
OSS LLM related issue
Comments
|
Embeddings are not working with Ollama... I was able to get things working with Ollama for the entities and openai for embeddings. |
|
Working config can be found here: #339 (comment) |
|
Ollama works as expected GRAPHRAG_API_KEY=123
GRAPHRAG_API_BASE=http://172.17.0.1:11434/v1
# GRAPHRAG_LLM_MODEL=llama3:instruct
GRAPHRAG_LLM_MODEL=codestral
GRAPHRAG_LLM_THREAD_COUNT=4
GRAPHRAG_LLM_CONCURRENT_REQUESTS=8
GRAPHRAG_LLM_MAX_TOKENS=2048
GRAPHRAG_EMBEDDING_API_BASE=http://172.17.0.1:11435/v1
GRAPHRAG_EMBEDDING_MODEL=mxbai-embed-large
API shapes
OAIJSON.stringify({
object: "list",
data: [
...results.map((r, i) => ({
object: "embedding",
index: i,
embedding: r.embedding,
})),
],
model,
usage: {
prompt_tokens: 0,
total_tokens: 0,
},
})OllamaJSON.stringify({
model,
prompt: input,
}) |
Sorry for what might be obvious... but how do you run this proxy? When I run ollama serve it only listen on the default port and not on 11435 What do you use to run this proxy? |
|
@bmaltais, no worries! 11435 is a proxy server written in JS/Node to specifically map request/response between OAI and Ollama formats, I didn't list the whole code as it's pretty much from the Node docs |
This is what I was afraid of ;-) I guess I will wait for something to be built by someone. I don't understand enough about node.js to build this. |
Can you please explain how did you do this..for embeddings api.. |
|
It works with ollama embedding by changing the file in /opt/anaconda3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/openai_embeddings_llm.py with from typing_extensions import Unpack from graphrag.llm.base import BaseLLM from .openai_configuration import OpenAIConfiguration import ollama class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]): |
Can you please provide the complete /opt/anaconda3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/openai_embeddings_llm.py replacement code and also the settings file. |
|
@SpaceLearner Does it work when you try to query? I adapted your code to work with langchain, it create the embeddings... but when I try to do a local query I get an error. This is my embeddings version: This the error: I suspect the query embeddings code also need to be modified... |
hack the file with the fellowing contents(tips: only fix # Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License
"""OpenAI Embedding model implementation."""
import asyncio
from collections.abc import Callable
from typing import Any
import numpy as np
import tiktoken
from tenacity import (
AsyncRetrying,
RetryError,
Retrying,
retry_if_exception_type,
stop_after_attempt,
wait_exponential_jitter,
)
from graphrag.query.llm.base import BaseTextEmbedding
from graphrag.query.llm.oai.base import OpenAILLMImpl
from graphrag.query.llm.oai.typing import (
OPENAI_RETRY_ERROR_TYPES,
OpenaiApiType,
)
from graphrag.query.llm.text_utils import chunk_text
from graphrag.query.progress import StatusReporter
from langchain_community.embeddings import OllamaEmbeddings
class OpenAIEmbedding(BaseTextEmbedding, OpenAILLMImpl):
"""Wrapper for OpenAI Embedding models."""
def __init__(
self,
api_key: str | None = None,
azure_ad_token_provider: Callable | None = None,
model: str = "text-embedding-3-small",
deployment_name: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
api_type: OpenaiApiType = OpenaiApiType.OpenAI,
organization: str | None = None,
encoding_name: str = "cl100k_base",
max_tokens: int = 8191,
max_retries: int = 10,
request_timeout: float = 180.0,
retry_error_types: tuple[type[BaseException]] = OPENAI_RETRY_ERROR_TYPES, # type: ignore
reporter: StatusReporter | None = None,
):
OpenAILLMImpl.__init__(
self=self,
api_key=api_key,
azure_ad_token_provider=azure_ad_token_provider,
deployment_name=deployment_name,
api_base=api_base,
api_version=api_version,
api_type=api_type, # type: ignore
organization=organization,
max_retries=max_retries,
request_timeout=request_timeout,
reporter=reporter,
)
self.model = model
self.encoding_name = encoding_name
self.max_tokens = max_tokens
self.token_encoder = tiktoken.get_encoding(self.encoding_name)
self.retry_error_types = retry_error_types
def embed(self, text: str, **kwargs: Any) -> list[float]:
"""
Embed text using OpenAI Embedding's sync function.
For text longer than max_tokens, chunk texts into max_tokens, embed each chunk, then combine using weighted average.
Please refer to: https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb
"""
token_chunks = chunk_text(
text=text, token_encoder=self.token_encoder, max_tokens=self.max_tokens
)
chunk_embeddings = []
chunk_lens = []
for chunk in token_chunks:
try:
embedding, chunk_len = self._embed_with_retry(chunk, **kwargs)
chunk_embeddings.append(embedding)
chunk_lens.append(chunk_len)
# TODO: catch a more specific exception
except Exception as e: # noqa BLE001
self._reporter.error(
message="Error embedding chunk",
details={self.__class__.__name__: str(e)},
)
continue
chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)
chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)
return chunk_embeddings.tolist()
async def aembed(self, text: str, **kwargs: Any) -> list[float]:
"""
Embed text using OpenAI Embedding's async function.
For text longer than max_tokens, chunk texts into max_tokens, embed each chunk, then combine using weighted average.
"""
token_chunks = chunk_text(
text=text, token_encoder=self.token_encoder, max_tokens=self.max_tokens
)
chunk_embeddings = []
chunk_lens = []
embedding_results = await asyncio.gather(*[
self._aembed_with_retry(chunk, **kwargs) for chunk in token_chunks
])
embedding_results = [result for result in embedding_results if result[0]]
chunk_embeddings = [result[0] for result in embedding_results]
chunk_lens = [result[1] for result in embedding_results]
chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens) # type: ignore
chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)
return chunk_embeddings.tolist()
def _embed_with_retry(
self, text: str | tuple, **kwargs: Any
) -> tuple[list[float], int]:
try:
retryer = Retrying(
stop=stop_after_attempt(self.max_retries),
wait=wait_exponential_jitter(max=10),
reraise=True,
retry=retry_if_exception_type(self.retry_error_types),
)
for attempt in retryer:
with attempt:
embedding = (
OllamaEmbeddings(
model=self.model,
).embed_query(text)
or []
)
return (embedding, len(text))
except RetryError as e:
self._reporter.error(
message="Error at embed_with_retry()",
details={self.__class__.__name__: str(e)},
)
return ([], 0)
else:
# TODO: why not just throw in this case?
return ([], 0)
async def _aembed_with_retry(
self, text: str | tuple, **kwargs: Any
) -> tuple[list[float], int]:
try:
retryer = AsyncRetrying(
stop=stop_after_attempt(self.max_retries),
wait=wait_exponential_jitter(max=10),
reraise=True,
retry=retry_if_exception_type(self.retry_error_types),
)
async for attempt in retryer:
with attempt:
embedding = (
await OllamaEmbeddings(
model=self.model,
).embed_query(text) or [] )
return (embedding, len(text))
except RetryError as e:
self._reporter.error(
message="Error at embed_with_retry()",
details={self.__class__.__name__: str(e)},
)
return ([], 0)
else:
# TODO: why not just throw in this case?
return ([], 0) |
|
It seems I have it working now. It returns nothings if I set llm to llama3, but works ok when switching to mistral. |
|
To change the openai request format to the one supported by ollama, setting only requires the base_url parameter, for example, api_base: http://localhost:8000/v1 |
|
@gdhua, your prompt-fu failed you, this proxy server doesn't transform embeddings API between OAI/Ollama formats. @bmaltais, here's the final version of the proxy I ended up using. There was another issue with the fact GraphRAG sends raw token IDs into the embeddings API, rather than non-tokenised raw text. Proxy server for OpenAI <-> Ollama embeddings
import os
import sys
import json
import logging
import asyncio
from aiohttp import web
import aiohttp
import tiktoken
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
config = {
"proxy_port": int(os.environ.get("PROXY_PORT", 11435)),
"api_url": os.environ.get("OLLAMA_ENDPOINT"),
"tiktoken_encoding": "cl100k_base"
}
encoding = tiktoken.get_encoding(config["tiktoken_encoding"])
async def handle_embeddings(request):
try:
body = await request.json()
model = body["model"]
input_data = body["input"]
print(f"/v1/embeddings handler {str(input_data)[:100]}")
if isinstance(input_data, str):
input_data = [input_data]
results = await asyncio.gather(*[fetch_embeddings(model, i) for i in input_data])
response_data = {
"object": "list",
"data": [
{
"object": "embedding",
"index": i,
"embedding": r["embedding"]
} for i, r in enumerate(results)
],
"model": model,
"usage": {
"prompt_tokens": 0,
"total_tokens": 0
}
}
return web.json_response(response_data)
except Exception as e:
print(f"Error: {str(e)}")
return web.Response(status=500)
async def fetch_embeddings(model, input_text):
if isinstance(input_text, int):
input_text = encoding.decode([input_text])
# If array of ints - decode the logits with tiktoken
if isinstance(input_text, list):
input_text = encoding.decode(input_text)
if not isinstance(input_text, str):
raise ValueError(f"Input is not a string: {input_text}")
async with aiohttp.ClientSession() as session:
async with session.post(
f"{config['api_url']}/api/embeddings",
headers={"Content-Type": "application/json"},
json={"model": model, "prompt": input_text}
) as response:
text = await response.text()
json_data = json.loads(text)
print(f"Embeddings: {input_text[:50]}... -> {text[:50]}...")
return json_data
def main():
print('Starting embeddings proxy...')
if not config["api_url"]:
raise ValueError("OLLAMA_ENDPOINT environment variable is required")
app = web.Application()
app.router.add_post("/v1/embeddings", handle_embeddings)
web.run_app(app, port=config["proxy_port"], host="0.0.0.0")
if __name__ == "__main__":
main()A few caveats:
|
|
@xiaoquisme , errors when using However, a disclaimer is that my llama3 sometimes even forgets to (where gpt rarely does) answer in the structure of json at all for queries like "Can you give me a joke for people read about this". I think this may only be fixed by improving the prompts or using a more "obedient" model. |
|
this worked for me. |
This was referenced Jul 9, 2024
|
I'm making this thread as our official discussion place for Ollama setup and troubleshooting. |
AlonsoGuevara
changed the title
This was referenced Jul 9, 2024
AlonsoGuevara
added
good first issue
|
this is a temp hacked solution for ollama |
4 tasks
Thanks. For anyone who don't use langchain and just want to use ollama's embedding model, you can make these changes and it will work for global query answering:
And yes, when doing local query there will still be an error concerning another function in this same .py file. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
is there a working example for using Ollama? Or is it not supposed to work? Did try, but without any success.
Thanks in advance
The text was updated successfully, but these errors were encountered: