Validate Vikingbot metrics with real conversations

Use this guide after setting up Prometheus and Grafana to check the complete path:

Chat → persisted bot session → feedback / follow-up → /metrics → Prometheus → Grafana

Prerequisites

• Start OpenViking with server.observability.metrics.enabled=true and openviking-server --with-bot --port 30300.
• Confirm /bot/v1/health returns HTTP 200 and /metrics returns Prometheus exposition text.
• Confirm Prometheus scrapes OpenViking and Grafana uses that Prometheus data source.

The localhost setup in the previous guide uses OpenViking at http://127.0.0.1:30300, Prometheus at http://127.0.0.1:30909, and Grafana at http://127.0.0.1:13000. If authentication is enabled, include the credentials required by your deployment in each request; see Authentication.

How to interpret the metrics

Feedback metrics are snapshot gauges, including names ending in _total. FeedbackCollector aggregates persisted metadata.feedback_events and metadata.response_outcomes from <bot_data_path>/sessions/*.jsonl. Its default refresh TTL is 30 seconds, so repeated scrapes can reuse a snapshot. Allow a collector refresh and a subsequent Prometheus scrape before comparing values.

Use current values or a short time window, rather than counter-style rate(). Select valid="1"; valid="0" is a fallback snapshot after collection failure and is unsuitable for acceptance checks. See the Prometheus metric types for the gauge/counter distinction.

Use a fresh session ID for each scenario. Totals cover all persisted sessions, so other traffic can affect the result. Proportions may barely move when historical sample counts are large. Outcome totals can decrease when an existing response is reclassified.

Record a baseline

Run these queries in Prometheus or Grafana Explore's table view:

openviking_service_readiness{valid="1"}
openviking_component_health{valid="1"}
openviking_queue_pending
openviking_vikingdb_collection_vectors{valid="1"}
openviking_model_usage_available{valid="1"}

Run each expression separately. Readiness should be 1; healthy components normally report 1. Queue depth should be small or zero. Collection vector counts and model-usage availability depend on the configured services and need not change during a bot conversation.

Record the feedback snapshot too:

{__name__=~"openviking_feedback.*",valid="1"}

1. Chat and confirm persistence

curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "realcase-01-chat",
    "user_id": "metrics-validation-user",
    "message": "Describe OpenViking in one sentence."
  }'

Expect HTTP 200 with session_id, response_id, and a nonempty message. Save the response ID for feedback.

After refresh, openviking_feedback_responses_total{valid="1"} should normally increase by one. A new session file also increases openviking_feedback_sessions_scanned_total{valid="1"}. Ordinary chat alone does not create an explicit feedback event.

2. Submit positive feedback

Send another chat using session_id: "realcase-02-thumb-up", then replace <response_id> below with that chat's response ID:

curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "realcase-02-thumb-up",
    "response_id": "<response_id>",
    "feedback_type": "thumb_up",
    "feedback_text": "helpful"
  }'

Expect accepted: true. For a response with no previous feedback, these gauges normally increase by one:

openviking_feedback_events_total{valid="1"}
openviking_feedback_thumb_up_total{valid="1"}
openviking_feedback_responses_with_feedback_total{valid="1"}
openviking_feedback_positive_outcomes_total{valid="1"}

coverage, thumbs_up_rate, and one_turn_resolution_rate can also change. The implementation includes both resolved and positive_feedback in one-turn resolution. Check totals first; aggregate proportions depend on all previous responses and outcomes.

3. Submit negative feedback

Chat with a fresh session, realcase-03-thumb-down, then send the same feedback request with that session's response ID, feedback_type: "thumb_down", and feedback_text: "not helpful".

Expect the following totals to increase by one after refresh:

openviking_feedback_events_total{valid="1"}
openviking_feedback_thumb_down_total{valid="1"}
openviking_feedback_negative_outcomes_total{valid="1"}

openviking_feedback_thumbs_down_rate and openviking_feedback_negative_feedback_rate may change too. Positive-feedback and reask totals should not increase from this action alone.

4. Follow up within ten minutes

Chat using realcase-04-reask. Without submitting feedback, send a second message to the same session within ten minutes of the assistant response:

curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "realcase-04-reask",
    "user_id": "metrics-validation-user",
    "message": "Please explain that more concretely."
  }'

The previous assistant response is classified as reasked; this classification uses timing, not the semantic similarity of the questions.

openviking_feedback_reasked_outcomes_total{valid="1"}
openviking_feedback_reask_rate{valid="1"}

The reasked total normally increases by one. The second assistant response also increases responses_total. Explicit feedback totals should stay unchanged.

5. Follow up after ten minutes

Chat using realcase-05-followup-no-feedback, submit no feedback, and wait more than ten minutes after the assistant response. Send another message to the same session.

openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}

The previous response should now be classified as follow_up_without_feedback, increasing this total by one. A follow-up at or before the ten-minute boundary instead qualifies as reasked. Explicit feedback totals stay unchanged.

6. Understand the resolved outcome

A response with no subsequent user message is classified as resolved when the outcome evaluator runs. The metrics collector only reads stored outcomes; it does not run an inactivity timer or create a resolved outcome after ten minutes.

openviking_feedback_resolved_outcomes_total{valid="1"}

A new chat followed by silence is therefore not a deterministic acceptance test for this metric. Inspect persisted metadata.response_outcomes if a resolved sample is expected. Use the positive, negative, and follow-up scenarios above as the repeatable acceptance path.

7. Check channel-level metrics

This scenario requires an enabled bot_api channel whose ID is demo. Chat through /bot/v1/chat/channel:

curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat/channel" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "realcase-07-channel-demo",
    "user_id": "metrics-validation-user",
    "channel_id": "demo",
    "message": "Describe OpenViking briefly."
  }'

Submit feedback with the returned response ID and the same channel_id:

curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "realcase-07-channel-demo",
    "response_id": "<response_id>",
    "channel_id": "demo",
    "feedback_type": "thumb_down",
    "feedback_text": "not helpful"
  }'

Check the target channel:

openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"}
openviking_feedback_channel_negative_outcomes_total{channel="bot_api__demo",valid="1"}
openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"}

The channel's event and negative-outcome totals should increase. Other channels should not change because of this request. A missing configured channel returns 404; plain /bot/v1/chat does not exercise this channel route.

Acceptance and troubleshooting

For a minimal check, require readiness 1, a persisted chat response, increased positive and negative feedback totals, and an increased reasked total after a prompt follow-up. Validate channel totals separately if your deployment uses channels.

Symptom	Check
Chat succeeds but feedback totals stay unchanged	Submit explicit feedback; ordinary chat does not create feedback events.
A recent event is missing	Allow the 30-second collector TTL, background refresh, and the next Prometheus scrape; then inspect session persistence.
Totals change but proportions barely move	Historical samples dilute a single event. Compare current totals in table view.
Only valid="0" is present	Inspect collector errors and access to the bot session directory.
bot_api__demo is missing	Check the enabled channel ID, /chat/channel endpoint, and matching feedback channel_id.

Related documentation

• Observability and troubleshooting
• Prometheus and Grafana setup
• Metrics concepts
• Metrics API