HPE PCAI LLM Gateway — Admin

Profile editor

Presets are applied to every key created with this profile.

Cascade model order (first with budget headroom serves)

Model routing

MCP grants mode

Enrollment memory

Agents offered

Default budgets

Default MCP grants

Register an MCP server

Register a model endpoint pool

Presented endpoints

Agents

A named tool surface, selectable as a model: a bundle of MCP servers — optionally narrowed to specific tools on each — plus an optional system prompt. A key sees the intersection of its grants and the agent's servers/tools — an agent can only ever HIDE tools, never add one.

Description

Rides /v1/models, so a client can pick an agent without reading this page.

MCP servers and tools

System prompt

Prepended to every model call of an agent turn, after the client's own system message. Never merged into it, never written into the stored conversation.

Create a guardrail

A content policy evaluated around model calls: pre_call before the upstream call, post_call on the response.

Config (JSON)

Agent traces

One card per relay request: model iterations, tool calls and wall-clock totals.

Admin audit trail

One immutable row per admin mutation — keys, budgets, models, MCP, profiles, credentials, enrollment. Append-only, never pruned, secret values redacted. Data-plane calls live in the usage ledger.

SSO self-mint default profile

Every key a user mints via Sign in with SSO gets this tenant's profile. On mint and on every rotate, two things happen: the profile's budgets, MCP grants and rate-limit presets are stamped onto the key row, and the user row is bound to the profile when it has none. Scopes stay the fixed least-privilege grant; profiles never widen them.

Mint an enrollment code

Self-service key issuance — the code is the credential.

The plaintext code is returned once at mint time (only its SHA-256 hash is stored). Redeem at the chat console via Have an invite code?. Budget (tokens and/or USD per period) is the enrollment's team envelope — a HARD cap on the whole team; per-key allocations must fit inside it. With overflow ON (default) the unallocated remainder is a shared pool keys may draw on. Leave BOTH budget fields blank for an UNCAPPED team — the envelope then says "no cap" on the team console while the per-key allocations (profile presets, or what the leader hands out) still apply; there is no way to add a cap afterwards except by approving a top-up, so set it here if the team should be bounded.

Off-peak billing

Requests bill at a reduced multiplier while GPU demand is low.

GPU avg (last)	—
Last check	never
PromQL	—
MCP server / tool	—
Thresholds	—
Cadence	—
Last error	—

Cache utilization — all models

POD MEMORY is the pod's working set against its memory limit; RAM (cached) is the HiCache host prefix-cache pool (tokens); DISK is the L3 volume occupancy (or activity when only counters exist). Aggregated over the cache-bearing endpoints. GPU stays on the per-model cards.

Per-model utilization

One row per model with its own PromQL. This is the evaluator state the discount reads — a model with no query configured has no row.

Model groups

A group is a named set of equivalent registered models. A request naming the group picks a member by LIVE load, the conversation is pinned to that member, and the selected member bills and carries its ACL — the group never bypasses budgets or allowed_models. Members are one level: a member may not itself be a group. While enabled, a group is advertised in /v1/models and usable as a direct model id or a fallback-chain entry.

Group	Type	Members	Policy	Description	Created

Create a model group

Save upserts by name: an existing group is replaced WHOLE — members not listed are dropped. A row's edit button loads the group here.

Would pick now (live preview)

The same power-of-two choice a real request would get: scores are the member's minimum request queue across its endpoints (no data = unknown, neither herded nor starved). A preview never pins or mutates anything.

Session memory distiller

Quiet conversations are distilled into each key's RAG memory dataset.

Summarizer model	Registered models (the Models tab's pools). Empty = the caller's routing.
Scheduling	off-peak only Scheduled distills run only while the summarizer model's off-peak evaluator reports low demand (global evaluator when the model has none).
Runtime enabled	enabled The env flag PCAI_LLM_DISTILLER_ENABLED must also be true — it owns the route mount.
Quiet window	Minutes since the last conversation write before the sweep distills it.
Output cap	Tokens. 0 = the env default.

Endpoint throughput and load

From the inference servers' own Prometheus metrics. Rates are averaged over the window shown below; a pod with no traffic in that window reads idle, not broken.

Pool	Endpoint	Prompt t/s	Decode t/s	Running	Queue	KV%	Cache hit	Prefill share

Model routing — how a routed request picks its model

Runtime enabled	enabled Also requires the env flag PCAI_LLM_ROUTING_ENABLED, which owns the route mount.
Decision mode	off — heuristics only shadow — record the pick, heuristics serve active — serve the pick above the floor off — the heuristics pick the model. shadow — the model is asked and its pick recorded, the heuristics still serve. active — the model's pick serves when it clears the confidence floor and lands inside the candidate window.
Decision model	The pool asked to choose. Empty = heuristics only.
Confidence floor	Below this the model is treated as abstaining and the heuristics serve.
Switch margin	A challenger must beat the current model by this much before the router pays a switch. A switch re-prefills the whole context on the new model. 0 = the first pick wins, 1 = never switch.
Decision timeout (ms)	On timeout the heuristics serve, so a hung endpoint cannot park a request.
Route the default model	also route requests that name auto Off: auto keeps the profile-order cascade. routing and routing__<model> always route.
Sticky routing	never downgrade a conversation A thread may escalate, but never falls below the model that already served it.
Sticky idle reset	Seconds of inactivity before a conversation forgets its floor. 0 = never.

Model cards — what each model is for

Tick a model to offer it to the router. Its description becomes that option's text in the routing question, so this is the only place the router learns what a model is good at. Live load, cost and image support are appended to each option automatically; a {{token}} in a description replaces the matching one.

Tick every model to mean no restriction (any chat model is routable). Ticking none is refused: on the wire an empty set means no restriction, so an empty form would save the opposite of "route to nothing".

Decision prompt — the policy every routed request is judged by

inject current GPU load into the prompt

Routing engages only when a request names routing or routing__<model> (or auto, when enabled above), and the calling key's profile routing policy is shadow or active.

Live statistics — routing counters from this gateway process ↻ refresh