​

Runtime Security Reverse-Proxy

• Native routes, OpenAI, Anthropic (Claude), Google Gemini, Google Vertex AI, AWS Bedrock.
• OpenAI-compatible, Groq, DeepSeek, Perplexity, Mistral AI, OpenRouter, Cerebras.
• Self-hosted via OpenAI-compatible, Ollama, llama.cpp, vLLM, on-prem TGI, or anything else exposing an OpenAI-shaped API.

​

1. Why a Reverse-Proxy?

• Applications that already go through an OpenAI or Anthropic SDK and want zero code changes beyond base_url.
• Third-party tools (LangChain, LlamaIndex, raw curl, Cursor, etc.) where the wire protocol is the only contract Antidote can rely on.
• Teams that want a hard guarantee that every call went through the firewall, enforced at the network boundary instead of relying on each developer to remember to call the scan endpoints.

​

2. Endpoints

• backend/app/api/runtime_security_proxy.py, FastAPI routes, auth, header filtering, error shaping.
• backend/app/runtime_security/proxy.py, pure payload-walk helpers (scan_openai_chat_request, scan_anthropic_response, …) and forward_upstream().
• backend/tests/test_runtime_security_proxy.py, unit tests for the payload-walk helpers against a FakeScanner.

​

3. Base URL Swap in Client SDKs

​

OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-antidote-host>/api/runtime-security/proxy/openai/v1",
    api_key="sk-...your-openai-key...",           # forwarded upstream
    default_headers={
        "X-API-Key": "ak_...your-antidote-api-key...",   # antidote auth
        "X-Antidote-App-Id": "app_...",                  # which App owns this traffic
    },
)

client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

​

OpenAI TypeScript SDK

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://<your-antidote-host>/api/runtime-security/proxy/openai/v1",
  apiKey: process.env.OPENAI_API_KEY,
  defaultHeaders: {
    "X-API-Key": process.env.ANTIDOTE_API_KEY!,
    "X-Antidote-App-Id": process.env.ANTIDOTE_APP_ID!,
  },
});

​

Anthropic Python SDK

from anthropic import Anthropic

client = Anthropic(
    base_url="https://<your-antidote-host>/api/runtime-security/proxy/anthropic",
    api_key="sk-ant-...your-anthropic-key...",    # forwarded upstream as x-api-key
    default_headers={
        "X-Antidote-Key": "ak_...your-antidote-api-key...",  # antidote auth
        "X-Antidote-App-Id": "app_...",                      # which App owns this traffic
    },
)

client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)

Why two different antidote-auth headers? OpenAI puts its upstream key in Authorization, Anthropic puts its upstream key in x-api-key. To avoid colliding with either, OpenAI callers send the antidote credential as X-API-Key (free for OpenAI) and Anthropic callers send it as X-Antidote-Key (free for Anthropic). Authorization is never consumed by the proxy routes, it is forwarded verbatim.

​

OpenAI-compatible providers (Groq, DeepSeek, Perplexity, Mistral, OpenRouter, Cerebras)

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-antidote-host>/api/runtime-security/proxy/openai-compatible/v1",
    api_key="gsk_...your-groq-key...",
    default_headers={
        "X-API-Key": "ak_...your-antidote-api-key...",
        "X-Antidote-App-Id": "app_...",
        "X-Antidote-Upstream-Provider": "groq",   # groq|deepseek|perplexity|mistral|openrouter|cerebras
    },
)

​

Self-hosted Ollama, llama.cpp, vLLM, TGI

# server env
RUNTIME_SECURITY_OPENAI_COMPAT_EXTRA_BASES=http://ollama.internal:11434/v1,http://llamacpp.internal:8080

client = OpenAI(
    base_url="https://<your-antidote-host>/api/runtime-security/proxy/openai-compatible/v1",
    api_key="not-needed-for-ollama",
    default_headers={
        "X-API-Key": "ak_...",
        "X-Antidote-App-Id": "app_...",
        "X-Antidote-Upstream-Base": "http://ollama.internal:11434/v1",
    },
)

Ollama’s OpenAI-compatible endpoint is http://<host>:11434/v1; llama.cpp’s server exposes http://<host>:8080/v1 by default.

​

Google Gemini (Google AI Studio)

import google.generativeai as genai

genai.configure(
    api_key="...your-google-api-key...",   # forwarded upstream via x-goog-api-key / ?key=
    transport="rest",
    client_options={
        "api_endpoint": "https://<your-antidote-host>/api/runtime-security/proxy/gemini",
    },
)
# OR pass the key as a header, the proxy forwards both query-string `?key=` and `x-goog-api-key`.

import requests

requests.post(
    "https://<your-antidote-host>/api/runtime-security/proxy/gemini/v1beta/models/gemini-1.5-pro:generateContent",
    headers={
        "X-API-Key": "ak_...your-antidote-api-key...",
        "X-Antidote-App-Id": "app_...",
        "x-goog-api-key": "...your-google-api-key...",
    },
    json={"contents": [{"role": "user", "parts": [{"text": "Hello"}]}]},
)

​

Google Vertex AI

from google.cloud import aiplatform_v1
# Or via raw HTTP:
import requests

token = ...  # from `gcloud auth print-access-token` or a service-account flow

requests.post(
    "https://<your-antidote-host>/api/runtime-security/proxy/vertex/v1/projects/"
    "my-proj/locations/us-central1/publishers/google/models/gemini-1.5-pro:generateContent",
    headers={
        "X-API-Key": "ak_...",
        "X-Antidote-App-Id": "app_...",
        "Authorization": f"Bearer {token}",
    },
    json={"contents": [{"role": "user", "parts": [{"text": "Hello"}]}]},
)

​

AWS Bedrock (Converse API)

import requests, os

requests.post(
    "https://<your-antidote-host>/api/runtime-security/proxy/bedrock/"
    "model/anthropic.claude-3-5-sonnet-20241022-v2:0/converse",
    headers={
        "X-API-Key": "ak_...your-antidote-api-key...",
        "X-Antidote-App-Id": "app_...",
        "X-Antidote-AWS-Access-Key-Id": os.environ["AWS_ACCESS_KEY_ID"],
        "X-Antidote-AWS-Secret-Access-Key": os.environ["AWS_SECRET_ACCESS_KEY"],
        "X-Antidote-AWS-Session-Token": os.environ.get("AWS_SESSION_TOKEN", ""),
        "X-Antidote-AWS-Region": "us-east-1",
    },
    json={
        "messages": [{"role": "user", "content": [{"text": "Hello"}]}],
        "system": [{"text": "You are a helpful assistant."}],
        "inferenceConfig": {"maxTokens": 1024},
    },
)

Streaming. /bedrock/model/{m}/converse-stream is not yet implemented, Bedrock streams use the AWS event-stream binary framing which requires a custom parser. Use the non-streaming /converse route for now.

​

4. Authentication Model

1. X-API-Key: ak_..., the normal antidote API key header. OpenAI clients should use this.
2. X-Antidote-Key: ak_... | Bearer <jwt>, fallback for clients that already use x-api-key for the upstream provider (i.e. Anthropic).
3. antidote_session cookie, lets the in-app dashboard exercise the proxy without provisioning an API key.

​

App attribution

1. Returns HTTP 400 APP_ID_REQUIRED / APP_NOT_FOUND for a missing or unknown App.
2. Returns HTTP 423 APP_DISABLED when app.status == "disabled", HTTP 410 APP_ARCHIVED when archived or soft-deleted.
3. Verifies X-Antidote-App-Token against the App’s signed token table when require_signed_token=true (APP_TOKEN_REQUIRED / APP_TOKEN_INVALID).
4. Enforces max_events_per_hour / max_events_per_day and returns HTTP 429 APP_QUOTA_EXCEEDED with Retry-After when hit.
5. Loads the App’s current_config_version and stores the ResolvedAppContext on request.state.runtime_security_app. The scanner picks up thresholds, detectors, custom phrases, custom PII rules, tool policy, and routing.forbidden_providers from this context, so two Apps in the same workspace can run with very different security postures against the same shared firewall.

Strict attribution mode. Setting ANTIDOTE_REQUIRE_APP_ATTRIBUTION=1 makes the scan endpoints reject calls without an App-Id too. The proxy already requires it regardless of this flag.

​

5. Request & Response Flow

1. Authenticate (see §4) and load the current RuntimeSecurityConfig via _load_settings(db). If cfg.enabled == False, return HTTP 503, the proxy refuses to pass traffic through when the firewall is disabled, so operators get a loud signal instead of silent bypass.
2. Parse the JSON body and reject stream=true with HTTP 400. See §9 for rationale.
3. Input scan. Walk the payload’s user-supplied text fields and call RuntimeSecurityScanner.scan(text, direction="input") on each one. On the first block verdict the route short-circuits with a provider-shaped error (§7) and persists a block event. On a redact verdict the helper rewrites the matching field in place so the forwarded request carries the sanitised copy.
4. Log the input verdict (cfg.log_events) via the same _persist_event() used by /scan/input, stamped with source_app="proxy:openai" / source_app="proxy:anthropic" so the dashboard can segment proxy traffic from direct scan traffic.
5. Forward upstream via httpx.AsyncClient.post(). Headers are filtered by _filter_inbound_headers(), hop-by-hop and antidote auth headers are stripped, and provider-specific headers are kept (Authorization for OpenAI, x-api-key / anthropic-version for Anthropic).
6. Pass through upstream errors untouched. If the upstream returns a non-2xx, the route surfaces the upstream body and status code verbatim so SDKs can parse the provider’s native error shape. We do not rewrite these.
7. Output scan. Walk the response body (choices[].message for OpenAI, content[] for Anthropic) and scan each text block with direction="output". Same block / redact semantics as the request phase.
8. Log the output verdict and return the (possibly redacted) body to the caller with the upstream status code.

​

6. Supported Payload Shapes

​

OpenAI Chat Completions (/v1/chat/completions)

• Role filter. Only user, system, and developer messages are scanned on input. Assistant turns are the model’s own previous output; they get scanned when they come back through direction="output", never here. This avoids false-positive input blocks when a suspicious string appears in a historical assistant turn.
• Content shapes. Both the legacy string form (content: "...") and the current array-of-parts form (content: [{type: "text", text: "..."}]) are supported. Non-text parts (image_url, audio, tool_use, …) pass through untouched.
• Redaction. The first matching text part is rewritten in place; other parts (images, tool calls) are preserved. For string content the entire field is replaced with the scanner’s redacted text.

​

OpenAI Legacy Completions (/v1/completions)

​

Anthropic Messages (/v1/messages)

• body["system"], string form and list-of-text-blocks form.
• body["messages"][] where role == "user", with both string and array content.
• Assistant turns are skipped for the same reason as OpenAI.

​

Google Gemini / Vertex AI (:generateContent and :streamGenerateContent)

• body["systemInstruction"]["parts"][].text is scanned and redacted in place (string and object form both supported).
• body["contents"][] is filtered to user turns. The role can be absent or "user"; "model" turns are skipped (they’re prior model output and will be scanned when they come back as response).
• parts[] with text are scanned; non-text parts (inlineData, fileData, functionCall) pass through untouched.
• Streaming uses SSE with alt=sse query-string. The proxy parses candidates[].content.parts[].text deltas, scans the accumulating buffer with the same windowed cadence as OpenAI / Anthropic, and emits a finishReason: SAFETY terminal event when the scanner blocks.

​

AWS Bedrock Converse (/model/{modelId}/converse)

• body["system"][].text, list-of-text-blocks form (the only form Bedrock Converse accepts).
• body["messages"][] where role == "user", scanning content[].text blocks. Non-text content (image, document, toolUse, toolResult) passes through untouched. Assistant turns are skipped.

​

Verdict Folding

​

7. Error Surface

​

OpenAI errors

{
  "error": {
    "message": "Blocked by Antidote Runtime Security: prompt_injection:score=0.92",
    "type": "antidote_runtime_security",
    "param": null,
    "code": "antidote_blocked"
  }
}

​

Anthropic errors

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "Blocked by Antidote Runtime Security: ..."
  }
}

​

Google Gemini / Vertex errors

{
  "error": {
    "code": 400,
    "status": "antidote_blocked",
    "message": "Blocked by Antidote Runtime Security: …"
  }
}

​

AWS Bedrock errors

{
  "message": "Blocked by Antidote Runtime Security: …",
  "type": "antidote_blocked"
}

​

Upstream error passthrough

​

8. Configuration

​

9. MVP Limitations

1. Streaming coverage is partial. OpenAI (stream=true), Anthropic (stream=true), all OpenAI-compatible providers, Google Gemini (:streamGenerateContent), and Vertex AI (:streamGenerateContent) all stream through the proxy with windowed scanning and a provider-shaped block chunk on policy hit. AWS Bedrock streaming (/converse-stream) is not yet implemented, Bedrock uses the AWS event-stream binary framing which needs a custom parser. The legacy OpenAI /v1/completions route also rejects streaming.
2. Non-text modalities pass through unscanned. Image inputs, audio, tool-call arguments, and function-call results are not scanned for PII or injection. Text-only fields of multimodal messages are still scanned. Vision/audio scanning will ship as a separate firewall feature.
3. No per-route rate limiting beyond the existing scan endpoints. The proxy uses the same runtime_security.scan permission and the shared rate limits that apply to the rest of the Runtime Security surface. Per-tenant upstream quota enforcement is out of scope here; the upstream provider is the source of truth for that.
4. Tool/function-call arguments are not scanned or redacted. This matches the scan endpoints’ current behaviour. A malicious prompt injection that routes through tool args would be caught at the surface prose, not the structured arg values.

​

10. Observability

• direction: "input" or "output"
• verdict: "allow" / "redact" / "block"
• provider: "openai" or "anthropic"
• source_app: "proxy:openai" or "proxy:anthropic", use this to segment proxy traffic from direct scan-endpoint traffic in the dashboard filters.
• model: the model name from the request body (input phase) or the upstream response (output phase).
• injection_score / injection_label, pii_count, pii_categories, text_length, blocked_reason as usual.
• metadata_: {phase, location}, location points at the blocked field (e.g. "messages[2].user", "choices[0].message", "system") to speed up root-cause analysis.

​

11. Testing

• Clean pass-through for all three payload shapes.
• In-place redaction for string content and array-of-parts content.
• Block short-circuit raises ProxyBlocked with the expected location string.
• Assistant turns are excluded from input scans (both OpenAI and Anthropic).
• Array-of-parts content preserves non-text parts (images, tool_use).
• Verdict folding returns the worst verdict across multiple fields.
• Anthropic system string and list-of-text-blocks shapes.
• Legacy OpenAI completions string-prompt and list-prompt handling.

python3 -m pytest backend/tests/test_runtime_security_proxy.py -q

​

12. Dedicated Runtime Security Container

​

Image layout

• POST /api/runtime-security/scan/input
• POST /api/runtime-security/scan/output
• GET /api/runtime-security/analytics
• GET /api/runtime-security/events
• GET /api/runtime-security/config
• PUT /api/runtime-security/config
• GET /api/runtime-security/health
• POST /api/runtime-security/proxy/openai/v1/chat/completions
• POST /api/runtime-security/proxy/openai/v1/completions
• POST /api/runtime-security/proxy/anthropic/v1/messages
• POST /api/runtime-security/proxy/openai-compatible/v1/chat/completions
• POST /api/runtime-security/proxy/openai-compatible/v1/completions
• POST /api/runtime-security/proxy/gemini/v1beta/models/{model}:{action}
• POST /api/runtime-security/proxy/vertex/v1/projects/.../models/{model}:{action}
• POST /api/runtime-security/proxy/bedrock/model/{modelId}/converse
• GET /healthz, container liveness
• GET /readyz, container readiness (returns ready once the injection model is warm)

​

Why a separate container?

1. Blast-radius isolation. A bug in the dataset or healing stack can crash or OOM the main API without ever touching the firewall. If the proxy is what your production LLM traffic depends on, it shouldn’t share a process with batch scan workers.
2. Right-sized scaling. The proxy runs on the critical path of every LLM call. The main API does not. You want to scale them on different signals (QPS for the proxy; dataset-count and worker-queue depth for the main API).
3. Smaller attack surface. The firewall image has no upload endpoints, no admin surface, no healing routes, nothing to exploit beyond the two wire-protocol shapes. Pen-testing the firewall becomes a much smaller scope.
4. Faster image rebuilds and pulls. The runtime-security image does not pull ultralytics/torchvision/pandas/scikit-image. Cold pulls are minutes faster in CI and autoscalers.

​

Shared database

​

Docker Compose service

runtime_security:
  build:
    context: ./backend
    dockerfile: Dockerfile.runtime-security
  environment:
    DATABASE_URL: ${DATABASE_URL}
    JWT_SECRET: ${JWT_SECRET}
    RUNTIME_SECURITY_INJECTION_MODEL: ${RUNTIME_SECURITY_INJECTION_MODEL:-deepset/deberta-v3-base-injection}
    RUNTIME_SECURITY_OPENAI_BASE_URL: ${RUNTIME_SECURITY_OPENAI_BASE_URL:-https://api.openai.com}
    RUNTIME_SECURITY_ANTHROPIC_BASE_URL: ${RUNTIME_SECURITY_ANTHROPIC_BASE_URL:-https://api.anthropic.com}
    RUNTIME_SECURITY_GEMINI_BASE_URL: ${RUNTIME_SECURITY_GEMINI_BASE_URL:-https://generativelanguage.googleapis.com}
    RUNTIME_SECURITY_OPENAI_COMPAT_ALLOWLIST: ${RUNTIME_SECURITY_OPENAI_COMPAT_ALLOWLIST:-}
    RUNTIME_SECURITY_OPENAI_COMPAT_EXTRA_BASES: ${RUNTIME_SECURITY_OPENAI_COMPAT_EXTRA_BASES:-}
    # Bedrock signing, optional; can also be provided per-request via X-Antidote-AWS-* headers.
    AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}
    AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-}
    AWS_REGION: ${AWS_REGION:-}
    # ... see compose file for the full environment
  ports: ["8088:8088"]
  depends_on: [db]
  volumes:
    - runtime_security_models:/models   # persist HF model cache
  healthcheck:
    test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8088/healthz"]

docker compose up -d db runtime_security

http://<host>:8088/api/runtime-security/proxy/openai/v1
http://<host>:8088/api/runtime-security/proxy/anthropic

​

Environment variables

​

Model prefetch at build time

• Mount a pre-populated /models volume at runtime, or
• Ship a custom model via RUNTIME_SECURITY_INJECTION_MODEL pointing at a local path, or
• Set RUNTIME_SECURITY_DISABLE_INJECTION_MODEL=1 to run in phrase-heuristic-only mode.

​

Health and readiness

• GET /healthz, always returns 200 {"status": "ok"}. Use this for liveness probes.
• GET /readyz, calls warmup() on the injection model and returns ready if the classifier loaded or degraded if the container is running on heuristics only. Use this for readiness probes so traffic doesn’t arrive before the classifier is loaded.
• Kubernetes / ECS task definitions should set readinessProbe.httpGet.path = /readyz and livenessProbe.httpGet.path = /healthz.

​

Scaling guidance

• The proxy is I/O-bound on upstream requests and CPU-bound on scanner inference. Two workers per container is a reasonable starting point; scale horizontally before scaling --workers past ~4 because the HF pipeline holds its own thread-pool internally.
• Model load is the slow step. Keep the container warm; aggressive scale‑to‑zero forces a fresh model load on every wake.
• Pin CPU limits high enough that DeBERTa inference doesn’t starve on a shared host.

​

13. Deployment Notes

• No new database migration. The proxy reuses the existing runtime_security_events table.
• No new permission. Proxy routes enforce the existing runtime_security.scan permission already used by /scan/input.
• TLS termination. The proxy must sit behind TLS, any plaintext deployment would leak both the client’s upstream provider key and the tenant’s prompt data. The existing ALB/NGINX termination used by the rest of the API is sufficient.
• Egress. The backend process must be able to reach each upstream provider you want to enable: api.openai.com, api.anthropic.com, generativelanguage.googleapis.com, {region}-aiplatform.googleapis.com, bedrock-runtime.{region}.amazonaws.com, and any OpenAI-compatible hosts you allowlist. In air-gapped deployments, point the RUNTIME_SECURITY_*_BASE_URL env vars at your internal gateway and register self-hosted bases via RUNTIME_SECURITY_OPENAI_COMPAT_EXTRA_BASES.
• Timeouts. Default RUNTIME_SECURITY_PROXY_TIMEOUT=60 seconds. Raise it for long-context requests that your upstream honors (e.g. Claude 200k-context calls).