Networking Development Engineering
Networking Development Engineering
  • HOME
  • SOLUTIONS
  • USE CASES
    • From question to ticket
    • Continuous Analysis
    • Inventory Sync
    • Device Onboarding
    • Silent Packet Loss
  • CONTACT US
  • BLOG
  • Más
    • HOME
    • SOLUTIONS
    • USE CASES
      • From question to ticket
      • Continuous Analysis
      • Inventory Sync
      • Device Onboarding
      • Silent Packet Loss
    • CONTACT US
    • BLOG
  • Iniciar sesión

  • Mi cuenta
  • Iniciaste sesión como:

  • filler@godaddy.com


  • Mi cuenta
  • Cierra sesión

Iniciaste sesión como:

filler@godaddy.com

  • HOME
  • SOLUTIONS
  • USE CASES
    • From question to ticket
    • Continuous Analysis
    • Inventory Sync
    • Device Onboarding
    • Silent Packet Loss
  • CONTACT US
  • BLOG

Cuenta

  • Mi cuenta
  • Cierra sesión

  • Iniciar sesión
  • Mi cuenta
Grafana annotation by FlowWeaver: silent packet loss 14.8% from srl1 to srl2, both ports clean
Use case · NetBox + Prometheus + Grafana MCP

Silent Packet Loss Between Two Clean Ports

Both ports report zero errors, yet 14.8% of the packets never arrive. One read-only FlowWeaver workflow finds the link, applies fixed rules and pins the verdict on the Grafana graph.

Lab: three Nokia SR Linux nodes in containerlab, with packet loss injected on purpose. No production or customer data.

View on GitHub

The Problem

Grafana: sent and received packets drift apart while port errors stay at zero on both ends

Both ports look healthy

Packets are being lost between two devices, but every error counter on both ports reads zero, so each device looks healthy on its own.

The answer is split across three tools

NetBox knows what is connected to what. Prometheus knows what each end sent and received. Grafana is where the operator is looking. Cross-checking them by hand, one interface at a time, is where the time goes.

The Environment

Network

Telemetry
Telemetry

Three Nokia SR Linux nodes in containerlab, in a triangle, with live UDP traffic. The fault: packet loss injected on srl1 ethernet-1/1, toward srl2, with containerlab netem.

Telemetry

Telemetry
Telemetry

gNMIc collects the interface counters into Prometheus, and Grafana shows them.

Source of truth

Source of truth
Source of truth

NetBox with the devices, their interfaces and the cables between them.

FlowWeaver

Source of truth
Source of truth

Self-hosted, with its NetBox integration, a Grafana MCP server and an email channel.

What We Asked FlowWeaver

The request, in plain English

"Create a workflow called "Silent packet loss check" that diagnoses traffic lost between two devices when both ports look healthy. Inputs: device and interface, default srl1 and ethernet-1/1. 1. Ask NetBox which device and interface are on the other end of the cable. 2. Using the Grafana MCP query_prometheus tool on the prometheus datasource, get for both ends the last 5 minutes of packets sent and received in both directions, plus port errors: FCS errors, input discards, output errors and output discards (do not use in_error_packets). 3. Compare both ends and give one verdict from fixed rules: silent loss in one direction (8 percent or more with clean ports), errors on one end, errors on both ends, healthy link, cable missing in NetBox, telemetry missing, not enough traffic, and an inconclusive fallback. 4. Create a Grafana annotation with the verdict on the dashboard silent-loss, panel 1, at the time of the check. 5. Email the verdict with the figures to [our ops mailbox]. If any step fails, send a failure email. The workflow must not change anything on the network. Create it in draft."

What FlowWeaver built

It took a few follow-up messages to get the plan right. FlowWeaver answered with a plan and built nothing until it was confirmed. The result: a draft of 12 nodes and five new read-only Python snippets, checked in simulation with 0 issues.

The Workflow

Ten steps, each one visible on the canvas. The far end of the cable comes from NetBox, the numbers from Prometheus, and the verdict from fixed rules, not from the AI. 

netbox-lookup

build-queries · telemetry-near/far
build-queries · telemetry-near/far

Find the far end

Reads the cable peer of the interface in NetBox. No cable: the workflow takes the "cable missing" branch instead of guessing.

build-queries · telemetry-near/far

build-queries · telemetry-near/far
build-queries · telemetry-near/far

Read both ends

Two Grafana MCP query_prometheus calls fetch six SR Linux counters per end, raw, over the last 5 minutes. 

compute-verdict

build-queries · telemetry-near/far
finalize-verdict · annotate

Apply fixed rules

Turns counters into packets per second and returns exactly one verdict, with every figure behind it.

finalize-verdict · annotate

finalize-verdict · annotate
finalize-verdict · annotate

Pin it on the graph

Writes the annotation text and places it on the sent vs received panel, at the moment of the diagnosis.

send-email · notify-failure

finalize-verdict · annotate
send-email · notify-failure

Tell the team

Emails the verdict and figures. If any step fails, a separate email names the step and links to the run.

Logic You Can Read

The compute-verdict rules

FlowWeaver shows the logic of each node as a diagram, so the rules can be checked without reading code. They run in this order.

Telemetry missing / not enough traffic

No samples at all is "telemetry missing". Less than 1 packet/s sent in both directions is "not enough traffic".

Errors on one end or both

Errors or discards on a port. The problem is visible on the ports, so it is not silent.

Silent packet loss

Both ports clean and 8% or more of what one end sent never reached the other. The 8% threshold leaves room for sampling noise: in the lab, timing alone shows 2-5% of difference between the two ends.

Healthy link

Both ports clean and the loss stays below 8%.

Cable missing in NetBox

A separate branch, cable-missing-verdict, says so with the reason, and the result still reaches Grafana and the mailbox.

What It Can and Can't Touch

No device is touched

The workflow never opens SSH and never changes a configuration. It reads NetBox and Prometheus.

One write, and it goes to Grafana

The annotation is the only change it makes, plus the email.

The MCP server is scoped

It exposes search, datasource reads, Prometheus queries and annotations. Datasource creation and changes are blocked at the MCP proxy.

Honest note

Grafana OSS has no annotation-only role, so the service account behind the MCP server is an Editor. The limit comes from which tools the MCP server exposes, not from the Grafana role.

How It Gets to Production

Workflow simulation: Ready To Run, 12 nodes, with what simulation does not cover

Simulation first

The whole graph is checked before it can run. Simulation does not test live reachability, and it says so.

A QA run

Promoted from draft to QA and run there, against the devices that allow QA.

A named approver

Production needs a passing QA run from the last 48 hours and an approver's name, different from the person promoting.

Deterministic run

The AI built the workflow. The engine runs it. No LLM decides anything at run time.

Result

Run data: silent packet loss verdict, all error counters at zero and 14.8% loss from srl1 to srl2

One verdict

The Production run completed all 12 steps in 8.7 seconds: "srl1/ethernet-1/1 -> srl2/ethernet-1/1 is losing 14.8% of packets with both ports reporting clean counters."

14.8% lost, one direction only

srl1 sent 371.3 packets/s and srl2 received 316.3. The other direction showed no loss.

Zero errors on both ends

FCS errors, discards and output errors stayed at zero. The verdict was pinned on the Grafana panel and emailed with the figures.

An independent check

The same loss, computed directly in Prometheus, reads 14.8% at the time of the run.

Limits

It tells you which link and which direction, not what sits in the middle of the link (a patch panel, a media converter or a carrier circuit). It is built for a point-to-point link with a cable in NetBox.

The Evidence

Each step logged

Run 64d1de93 completed its 12 steps in 8.7 s, each one logged with its timing. The failure branches were skipped.

Pinned on the graph

Grafana annotation at 16:39:56: silent packet loss 14.8%, both ports clean

The verdict lands as an annotation on the Grafana panel at 16:39:56: 14.8% lost from srl1 to srl2, both ports clean.

Checked in Prometheus

Checked in Prometheus
Grafana Explore: srl1 to srl2 loss computed in Prometheus, near 15%, 14.8 at run time

The loss queried on its own in Grafana Explore holds near 15% and reads 14.8 at the time of the run.

Try It Yourself

Open source

What you need
Open source

FlowWeaver is open source (Apache 2.0) and self-hosted, with your own LLM.

Quick Start

What you need
Open source

Install it and follow the Quick Start at github.com/Isaza-Labs/flow-weaver.

View on GitHub

What you need

What you need
What you need

NetBox with your cables, Prometheus with interface counters, and a Grafana MCP server.

Still Chasing Loss One Interface at a Time?

If you have telemetry and NetBox, request a demo and we'll build this check on your stack.

Request a demo



Copyright © 2026 NetworkingDev - All Rights Reserved.
Nashira and FlowWeaver are products of Isaza Labs LLC. 

Este sitio web utiliza cookies

Usamos cookies para analizar el tráfico del sitio web y optimizar tu experiencia en el sitio. Al aceptar nuestro uso de cookies, tus datos se agruparán con los datos de todos los demás usuarios.

RechazarAceptar