Monitoring LLM Applications with New Relic AI Monitoring

AI features need observability just like any other production system

Shipping a feature backed by a large language model changes what you need to monitor. A traditional service has latency, throughput, and errors. An AI feature adds token cost, model latency that varies wildly by prompt, and a quality dimension that has no HTTP status code. New Relic AI Monitoring is built for exactly this, and here is how I approach instrumenting an LLM app.

Why LLM Apps Break Differently

A normal API call either works or returns an error. An LLM call can succeed at the HTTP layer and still fail the user — a confident wrong answer, a truncated response, or a three-second wait that blows your latency budget. Your observability has to capture three new axes:

  • Cost — tokens consumed per request, and the money that represents
  • Latency — model response time, which depends on prompt and output length
  • Quality — was the response useful, safe, and on-topic

What AI Monitoring Captures

New Relic AI Monitoring instruments the popular model SDKs and captures each interaction as structured telemetry: the model name, prompt and completion token counts, latency, and the request and response content. It ties these into the wider APM trace so an LLM call shows up inside the transaction that made it.

Every model call becomes a span you can trace, cost, and compare

Track Cost Before Finance Does

Token cost is the metric that surprises teams. A single verbose prompt template deployed to production can multiply your bill overnight. Query it directly:

SELECT sum(token_count) FROM LlmCompletion
FACET model TIMESERIES SINCE 1 day ago

Alert when daily token volume jumps beyond a threshold — that is usually a prompt regression or a retry storm, not real demand.

Watch Latency by Model

Different models and different prompt sizes give very different latencies. Compare them so you can route cheap or fast where quality allows:

SELECT average(response_time), percentile(response_time, 95)
FROM LlmCompletion FACET model SINCE 6 hours ago

The Quality Problem

Quality has no built-in metric, so you must create signals:

  1. User feedback — capture thumbs up or down and send it as a custom event
  2. Guardrail hits — count how often a safety filter or validator rejects a response
  3. Fallback rate — how often you fall back to a default because the model output failed validation
  4. Empty or truncated responses — a strong smell of a broken prompt or token limit

Record these as custom attributes on the same events so you can slice quality by model, prompt version, and user segment.

Close the Loop

The goal is a dashboard where cost, latency, and quality sit side by side, faceted by prompt version. When you ship a new prompt, you should see immediately whether it got cheaper, faster, or better — and whether improving one dimension quietly wrecked another.

An LLM feature with no observability is a slot machine you are paying to run in production. Instrument cost, latency, and quality, or you are flying blind on all three.

What to Learn Next

  • Custom events for user feedback and guardrail signals
  • Distributed tracing so a model call is visible inside its full request
  • Anomaly alerts on token spend and fallback rate

Arivanandhan Chitheshwaran