How It Works¶
This page explains what the Provider Client does on your machine and how it talks to Flockadoo. For install steps, see Installation. For engine and model selection, see the setup wizard.
Overview¶
The client runs alongside the local LLM engine the wizard installed (vLLM, vLLM Metal, or llama.cpp). It:
- Reports health (GPU and models) to Flockadoo on a schedule.
- Registers models and initial pricing for your flock.
- Accepts inference requests from Flockadoo and forwards them to your local LLM server.
- Opens an outbound connection so Flockadoo can reach you without open firewall ports. It reconnects if the connection drops.
On a Team Flock, setup joins this machine to a private network with winglink. The master uses the provider API key. A worker uses a join- code. Setup leaves auto_start off, prints the vLLM command, and does not start the engine. The master starts first. Workers join within 60 minutes of the master coming up. A worker does not serve HTTP. It joins the master. The flock stays offline after they join until vLLM finishes loading and the next health report lands. The status page shows Flock, Nodes, Winglink IP, and Peers. See Team Flocks.
If auto_start is on (the wizard turns this on when it finds the engine binary on a single-node machine), the client starts that engine when llm_url is down. It uses the serve flags already in config.yaml (including whether tool calling is on). It will not start a second copy if the port is already in use, and it will not pull fresh catalog flags at runtime. Re-run setup after you change model or catalog flags.
Startup checks¶
- The engine starts.
- The performance benchmark runs (vLLM only). This is the only phase where the client does not take requests.
- You are online. The provider can receive ordinary inference requests.
- Capability checks run in the background, only when the GPU is idle. Each one is tool calling, forced tool calling, JSON output, JSON schema, logprobs, or (for a model that thinks) reasoning.
A busy GPU can take hours to finish the checks. That is fine. Ordinary requests keep flowing.
The console and the local dashboard show a Capabilities block. The header counts checks that have finished, including ones that came back unverified. For example, testing - 2/5 checked (still serving requests). Each line is one of:
| State | Meaning |
|---|---|
waiting |
Not started. Requests are still served. |
testing |
This check is running. |
supported |
The engine passed the check. |
unsupported |
The engine answered, and the model cannot do this. It is not a broken install. |
unverified |
The check timed out or the engine errored. It retries while the GPU is idle. |
Capability checks¶
After the engine is ready, the client checks five features one at a time, only while no request is in flight: tools, forced tools, JSON object, JSON schema, and logprobs. Session Status stays online during these checks. A check that times out stays unverified and is retried while the GPU is idle.
Each check uses one request slot while it runs, and checks never run at the same time as the performance benchmark. On llama.cpp, which runs one request at a time, new requests go to other providers while a check runs.
Info
Your provider stays online and can earn from ordinary inference requests while capability checks run. These checks determine whether you can also receive requests requiring additional features.
The results also appear on the flock page in the dashboard. See Engine checks.
Health monitoring¶
Every 3 minutes, the client sends a health report to Flockadoo. It also sends one when an inference slot fills or frees, after a 10 second cooldown so a burst of requests does not flood the platform. The report includes:
- GPU type, memory, and utilization (on Linux and Windows with NVIDIA).
- Which models your local server exposes.
- Which engine serves them (vLLM, vLLM Metal, or llama.cpp) and its version, as the running engine reports it.
- A GPU busy flag. See When the client is busy below.
Flockadoo detects flock country from the public IP of your connection. You do not send location from the client. See Flock location.
You can also check status locally anytime (these URLs stay on your machine):
- Open http://localhost:8080/ in a browser (or whatever port you configured) for a live status page.
- GET /api/status — JSON snapshot of that page.
- GET /api/health — full health snapshot, including the GPU busy flag.
- GET /api/busy —
{"busy": true}or{"busy": false}. This is the slot check only. It does not readnvidia-smi.
When the client is busy¶
The client keeps two busy signals. Each one answers a different question.
| Signal | Busy when |
|---|---|
New request (GET /api/busy, and 503) |
Every inference slot is taken. The default is 1 slot (max_concurrent_inference). NVIDIA utilization is not used. |
Health report GPU is_busy |
On NVIDIA, utilization is above 20%, or an inference slot is full. On macOS, only when a slot is full. |
A different session that arrives while a slot is full gets 503 {"error":"GPU is busy"}. Flockadoo can try another provider. A follow-up for the same session waits up to 90 seconds (session_queue_wait_seconds) for that slot instead of 503.
Utilization above 20% sets the GPU busy flag in the health report. That update waits for the 3 minute report. A request already on the machine is rejected when a slot is full. Another program using the GPU can set the flag. After your own request finishes and the slot is free, the next request is accepted while nvidia-smi is still high for a moment.
See Configuration.
Model pricing and registration¶
At startup, the client discovers models on your local LLM server, looks up default prices from Flockadoo, and registers each model for that flock.
Ongoing, each health cycle checks for new models (for example after you re-run setup with a different approved model). New models are registered automatically — you do not need to restart the client.
Initial prices come from platform averages. You set your own prices per flock in the dashboard — see Model pricing.
Approved marketplace models must match a platform approved model build. The public catalog’s engines field lists which client (vLLM, vLLM Metal, llama.cpp) may serve that row. The client verifies the weight fingerprint against platform records before registration, health reporting, and inference.
The client reads the served model id and finds weights in the HuggingFace hub cache using the approved hf_ref. Optional hf_hub_cache or model_path only if your layout is non-standard.
Inference requests¶
When Flockadoo sends a request to your flock:
- Authentication — Valid requests include a signed header; missing or invalid requests get 401 Unauthorized.
- Busy — If every inference slot is full, and this is not a same-session follow-up, the client responds with 503
{"error":"GPU is busy"}so Flockadoo can try another provider. Same-session follow-ups can wait for the slot instead. - Proxy — Valid requests are forwarded to your local LLM server. The client supports POST /v1/chat/completions, POST /v1/completions, and POST /v1/embeddings.
Team Flocks¶
Inference still arrives over the outbound connection to Flockadoo. Machines in one Team Flock also join a private network so they can run one model together. Other flocks are not on that network. See Team Flock networking.
Connection to Flockadoo¶
On startup the client opens an outbound connection to Flockadoo. If the connection drops, the client reconnects on its own. You do not configure a public URL.
On shutdown, the client closes the connection.