OpenFax case record
Prepared as a factual service-continuity summary
June 29 – July 1 production outage

A routing-control outage, with signs the private configuration may have been unstable earlier.

OpenFax’s production disruption was driven by a Telnyx-side routing condition on the private Direct Connect environment. OpenFax could change its own side of signaling, but could not control Telnyx’s return path because the VXC did not appear anywhere in the online account.

Primary focus
Production outage
The main issue is the interruption to service continuity and the limits on customer recovery during the event.
Control constraint
VXC not visible
The VXC connection did not appear in the online account, leaving no customer portal controls for temporary routing relief.
Current Status (July 6, 2026)
Persistence of failure
The VXC management interface remains invisible in the portal today, confirming the administrative blockage was not a transient incident-only state.
Objective quality fingerprint
MOS collapse, Jun 29
Call-quality (MOS) scores held at 4.44–4.49 for eleven straight months, collapsed to ~1.0 from June 29 ≈11:00 UTC, and snapped back to baseline at July 1 ≈13:00 UTC — exactly at the regional cutover.
What was originally configured
Private NAT + private media
This private-IP NAT configuration was prescribed by Telnyx during VXC installation and deployment testing.
What OpenFax observed
Sudden, sharp onset
Quality metrics (MOS, PDD, timeout rates) were flat for eleven months, then collapsed abruptly on June 29 — a precisely datable failure, not gradual degradation.
What still failed
No media in outage
During the outage, signaling could improve on public while media still was not being received over the private return path.
What restored service
Regional standup
A separate public-internet deployment restored service before the original private-link issue was resolved.

Broader commercial impact may extend beyond the outage window

OpenFax’s concern is not limited to one outage interval. If private-path signaling was intermittently misconfigured for a prolonged period, that could have affected overall carrier performance, prior surcharge billing, and incomplete fax delivery. It also raises the possibility that historical calls may have run into OpenFax’s call-limit timer when signaling did not terminate correctly, adding meaningful cost to monthly invoices while media may not have been received from all carriers.

Carrier performance consideration Incomplete fax delivery and signaling inconsistency may have been symptoms of the same private VXC configuration issue.
Invoice impact consideration Incorrect prior surcharge billing may need to be reviewed in light of possible signaling failures and call-extension behavior.
Configuration origin The private NAT signaling design at issue was prescribed by Telnyx during installation and deployment testing of the VXC environment.

The outage, in one clear storyline

This version puts the outage first, keeps billing in the background, and explains the issue as a service-control problem with a simple progression from normal state to constrained recovery to restored operations.

Chapter 1

Normal production design

OpenFax operated through a redundant Equinix Direct Connect design, with an active/standby arrangement. Telnyx propagated the private NAT over BGP, and production media was expected to travel over the VXC path.

  • Private NAT presented through the Telnyx-provided VXC environment
  • SIP and media were originally aligned to the private deployment
  • This deployment pattern was prescribed by Telnyx during installation and deployment testing
Chapter 2

OpenFax narrowed the failure

Before the total outage, OpenFax was already seeing many calls with random signaling misses. During the outage, media was not received at all. That combination raised the possibility that the private-IP NAT signaling path had been inconsistent for a longer period before the full production break.

  • Random signaling was not received on many calls before the total outage
  • During the outage window, no media was being received
  • Short call durations and long-extended calls may both fit a longer-running private NAT inconsistency
Chapter 3

Recovery required a new region

With no customer-side portal control over the private path, no visible VXC object in the online account, and no timely response to test disabling the VXC over roughly two days despite repeated email and phone escalation, OpenFax had to restore operations by standing up infrastructure in another region over public internet. That recovered service before the original private path was resolved.

  • OpenFax could adjust signaling behavior on its own side, but not the Telnyx return path
  • Repeated support attempts by email and phone did not produce timely VXC disablement testing
  • Recovery came from bypassing the affected private environment entirely

Private-path control map

The key issue was not whether OpenFax could change its own edge. It was that OpenFax had no control over the Telnyx-side return path while the VXC was absent from the online account.

OpenFax edge Own BGP side
SBC / SIP proxy controls
Before incident
Private NAT over VXC Media over private path
Telnyx private side BGP propagation
Return-path routing
OpenFax during incident Signaling switched public
What OpenFax proved
SIP improved on public Media not received
Telnyx still controlled Private return path
Portal visibility
Online account Customer portal controls
Administrative reality
VXC not present No temporary route control
Only visible customer action Delete Direct Connect
cancel circuits / re-order required
Why the portal point matters

The VXC did not appear in the online account, so OpenFax could not simply disable or reroute the private connectivity from a normal customer control surface.

Why the MOS fingerprint matters

Eleven months of flat 4.44–4.49 MOS followed by an instant collapse to ~1.0 is an objective, independently verifiable measure that audio stopped flowing at a precise hour — and that it recovered the moment OpenFax cut over away from the Telnyx private path.

Recovery controls available during the incident

A simple view of what OpenFax could act on versus what remained blocked by missing portal visibility and Telnyx-side routing control.

Outage timeline with public status-page context

The left side tracks the OpenFax production story. The right side lists contemporaneous public entries from status.telnyx.com/history during the June 29 to July 4 window. These public entries provide timing context, but do not by themselves prove causation.

OpenFax production narrative

June 29, ≈11:00 UTC Objective onset: call-quality (MOS) collapses

Average MOS had been rock-steady at 4.44–4.49 every month since August 2025. From roughly 11:00 UTC on June 29 it drops to ~1.0–1.5 — audio effectively unusable — the earliest bad reading in the CDR data.

June 30 Production outage begins on the private environment

Private-path behavior fails in a way OpenFax cannot fully correct from its own side. The VXC remains absent from the online account.

June 30 (diagnostic sequence) OpenFax isolates the failure step by step

10:29 ICT — identifies 100.79.0.23 as the VXC NAT address Telnyx's re-INVITE was being sent to. 12:04 ICT — advertises its public IP instead; signaling timeout fixed, but a media (RTP) gap is exposed. 13:57 ICT — with rtpengine deployed as a media relay, the final signature is isolated: T.30 sends DCS correctly but the far end never returns CFR, with zero packet loss on OpenFax's own media path — ruling out OpenFax infrastructure.

June 30 → July 1 Public signaling improves, but media still does not return

Under the Direct Connect deployment, OpenFax moves signaling to public. That narrows the problem, but media is still not being received because return-path control remains outside customer reach.

July 1 Operational recovery by standing up another region

OpenFax restores services over a public-internet deployment in a different region, while the original private-link condition remains unresolved. Average MOS snaps back to the 4.44–4.49 baseline at ≈13:00 UTC — instantly, and exactly at the cutover. Abandoned-call rates stay elevated (30–35%) for roughly four more hours as backlog/retry traffic clears, normalizing by July 2.

July 6 VXC portal visibility remains absent

As of Monday, July 6, 2026, the VXC connection and associated configuration controls under https://portal.telnyx.com/#/networking remain unavailable in the OpenFax account, demonstrating the persistence of the control-plane failure.

Public Telnyx status-page anchors

June 29 Voice services update – all regions

Public history shows multiple maintenance-completed items, including voice-service and programmable-voice entries on June 29.

July 1 SIP trunking signaling – US/CA regions

The status page shows a maintenance-completed signaling entry on July 1, alongside programmable-voice maintenance for the same day.

July 2 Additional voice / signaling maintenance entries

Public history lists multiple maintenance-completed entries on July 2, including voice services update – all regions and SIP trunking signaling region updates.

July 4 Mission Control portal SIP debugger issue resolved

A July 4 public portal-related entry appears in the same broader status window. It is included here strictly as timing context, not as a claim of direct linkage to the OpenFax outage.

Technical appendix

The detailed material remains available, but it is moved below the main storyline so the page reads like an executive landing site first and a technical record second.

Operational takeaway
Why the outage points back to missing customer control
  • OpenFax could alter its own side of signaling and validate that signaling behavior improved.
  • Media still failed, which isolated the remaining issue to the private return path and Telnyx-side control.
  • Because the VXC did not appear in the online account, OpenFax had no normal self-service path to temporarily disable or bypass the private route from the portal.
  • The only visible customer action would have been deletion of the Direct Connect, which would cancel circuits and require re-ordering by Telnyx.
What the CDR quality data shows about onset
A sudden, precisely datable failure — not gradual degradation
  • Monthly average MOS, Quality %, and PDD were all flat from August 2025 through June 26, 2026 — no drift, no gradual decline.
  • Monthly rates of MEDIA_TIMEOUT / RECOVERY_ON_TIMER_EXPIRE hangup causes were also flat (0.03–0.16%, no trend).
  • The failure appears abruptly at June 29 ≈11:00 UTC and ends abruptly at July 1 ≈13:00 UTC — a sharp-onset event with a clean fingerprint, coinciding with the publicly posted maintenance window.
  • The private NAT configuration at issue was prescribed by Telnyx during VXC installation and deployment testing.
Status-page context used on this page
Public entries included from June 29 to July 4
  • June 29: Voice Services Update - All Regions; Voice services update - all regions; Scheduled maintenance for Programmable Voice - Voice API / Programmable Voice - TeXML EU/US/APAC.
  • July 1: SIP Trunking Signaling - US/CA regions; Scheduled maintenance for Programmable Voice - Voice API / Programmable Voice - TeXML EU/US/APAC/UAE.
  • July 2: Voice services update - all regions; SIP Trunking Signaling - AU/Asia regions; SIP Trunking Signaling - CA region; additional public maintenance-completed entries.
  • July 4: Issues with Mission Control Portal SIP Debugger — resolved; SIP Trunking Signaling - US region.

These entries are shown as contemporaneous public anchors only. They are not presented as a standalone causal determination.

Carrier performance and invoice impact context
Broader effects potentially tied to VXC configuration

The outage remains the main narrative, but OpenFax also asks that prior carrier performance, incomplete fax delivery, and earlier surcharge billing be reviewed in the context of the Telnyx-prescribed VXC configuration.

  • OpenFax is considering whether prior carrier-side performance issues and incomplete fax delivery were affected by the same private signaling inconsistency.
  • Random signaling differences between VXC and public routing may have caused historical calls to run into OpenFax’s call-limit timer, adding significant monthly invoice cost.
  • OpenFax is also considering whether previous surcharge billing was incorrect if calls were not terminating cleanly because signaling was not consistently received.
  • Detailed surcharge and ledger analysis can remain available as supporting material without becoming the headline of the page.
View full Dossier & Evidence →