SSEclient

[rodrigo-pedro]

I can't seem to be able to get streaming to work with sseclient in python. I must be doing something wrong.

Here is my server side:

@app.post("/", response_model=CompletionResponse)
async def completion(request: CompletionRequest = Body(...)):
    llm = OpenAI(
        openai_api_base=api_base, openai_api_key=api_key, streaming=True, verbose=True
    )

    template = "You are a helpful assistant.\
        \n\n### Instruction:\n{question}\n\n### Response:\n"

    prompt = PromptTemplate(template=template, input_variables=["question"])

    llm_chain = LLMChain(prompt=prompt, llm=llm)

    return StreamingResponse.from_chain(
        llm_chain, request.prompt, media_type="text/event-stream"
    )

And here is the client:

prompt = input("Enter prompt: ")
print("Assistant:", end=" ")

url = "http://localhost:7000/"
response = requests.post(url, stream=True, json={"prompt": prompt}, headers={"accept": "text/event-stream", "Content-Type": "application/json"})

sseClient = sseclient.SSEClient(response)

for event in sseClient.events():
    print("received event")

What's happening is that I don't seem to be receiving events.

It works fine if I use curl like so

curl -N -X POST \     
-H "Accept: text/event-stream" -H "Content-Type: application/json" \
-d '{"prompt": "Give me list of text splitters available with code samples" }' \
http://localhost:7000

[rodrigo-pedro]

Nevermind i just realized you have a gradio example. It works with iter_content

[ajndkr]

@rodrigo-pedro i will update the docs to add an example for client side code.