How to use the API?

[Brackmann]

Hello there!
I am trying to integrate freedomgpt with python through an API for my students. When I access the url "http://localhost:8889/" I get a "Cannot GET /" message. Where do I find more information about it and how to send prompts?
Thanks in advance.

[johnem-msft]

Did you ever get an answer to this?
I'm running FreedomGPT on a Windows Destkop ... I see XMRIG running in the background (fgptminer.exe) which is mining FNT tokens ...
I also see two instances of server.exe ...
Checking Test-NetConnection 127.0.0.1 -Port 8889 I don't see anything there ... (TCPSuccess=False) ... same for nmap ...

Checking:
netstat -anob | Select-String server.exe -Context (2,2)
I do see:
[server.exe]
TCP 127.0.0.1:8887 0.0.0.0:0 LISTENING 25288
[server.exe]
TCP 127.0.0.1:8887 0.0.0.0:0 LISTENING 22788

Hence ... I see something here on 8887 ... will check against that ...

If you came across some other standalone LLM you liked please also let us know :)

EDIT: Looks like this is llama.cpp:
Llama.cpp has a low-level API that is a direct ctypes binding to the C API provided by llama.cpp1. The entire API can be found in llama_cpp/llama_cpp.py and should mirror llama.h. There is also an API wrapper around llama.cpp called gpt-llama.cpp2. It runs a local API server that simulates OpenAI's API GPT endpoints but uses local llama-based models to process requests. llama-cpp-ui is a web API and frontend UI for llama.cpp written in C++

[johnem-msft]

This ended up working fine - this is PS5 - I know Invoke-WebRequest and curl exist differently now in PS7

/## PS5 Invoke-WebRequest to llama.cpp ##
/## API ##
$apiKey = "MY_API_KEY"

/## ASK ##
$ask = 'What percentage of the eath is covered in water?'
$headers=@{}
$headers.Add("Authorization", "Bearer $apiKey")
$headers.Add("Content-Type", "application/json")
$body = @{"prompt" = "$ask"} | ConvertTo-Json

$curl = InvokeWebRequest -Uri "http://127.0.0.1:8887/completion" -Method Post -Headers $headers -Body $body
($curl.Content | ConvertFrom-Json).content

This would be even more straight forward w. standard curl ... anyways ...

NOTE: http://127.0.0.1:8887 is a nice little webui too

[johnem-msft]

EDIT: After all this ... I'm not seeing a good way to make llama.cpp listen on an external port ... so ... thought I would mention that here.

There are some files (for the Windows Version):
/AppData/Local/FreedomGPT/app-3.0.4/resources/app/main/port.js & index.js ... there are some mentions of "localhost" in index.js ... but ... it's not as straight forward as changing those values from "localhost" to 0.0.0.0 or otherwise ... hmm ...

If this weren't all pre-packaged you could do so via:
--host HOST ip address to listen (default: 127.0.0.1)
(env: LLAMA_ARG_HOST)
--port PORT port to listen (default: 8080)
(env: LLAMA_ARG_PORT)
https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md

Seeing some other chatter about using nginx as a reverse proxy ...

EDIT2:
Find server.exe - end it in taskman (if Windows) and then run:
C:\Users\ADMIN\AppData\Local\FreedomGPT\app-3.0.4\resources/models/windows/llama/server -m C:\Users\ADMIN\AppData\Local\FreedomGPT\app-3.0.4\dolphin-2.9-llama3-8b.Q8_0.gguf -c 2048 --port 8887 --chat-template chatml --host 0.0.0.0 #fixed - now answers from all addresses