Troubleshooting

The dashboard is green but no television receives anything

Every channel shows as running. Bitrate figures look plausible. Not one television has a picture. This is the most common fault on a new installation by a wide margin, and it is almost always the multicast route.

Likely cause

The route for multicast traffic points at the wrong network interface, or at an interface that has no carrier — nothing plugged in, or plugged into a dead port. Packets are produced correctly and handed to an interface that leads nowhere.

Why this is invisible from inside the process

Sending a UDP datagram to an unroutable multicast destination succeeds. The socket call returns without error. There is no ICMP unreachable to observe, no connection to fail, no retry to count. Consequently every counter inside the streaming process reads perfectly healthy while nothing leaves the building.

No streaming engine can detect this, and no amount of restarting changes it. It has to be checked against the host routing table from outside the process, which is what Castmux does.

Binding the output socket to a source address does not solve it. The kernel still consults the routing table to choose the egress interface; the source address only sets what appears in the packet header.

How to confirm

Two commands on the appliance. Ask the kernel where a datagram for the channel’s group would actually go, and then look at which interfaces have a carrier:

# Substitute the destination group of an affected channel
ip route get 239.10.0.1

# Which interfaces are actually up, and which are merely configured
ip -brief link

The first command names the interface that will be used. If it is not the interface facing the televisions, that is your fault found. If it is the right interface but ip -brief link shows that interface DOWN or with no carrier, that is also your fault found.

Link-local addresses will mislead you

An interface commonly carries both a link-local address in 169.254.0.0/16 and its real address, and ip -brief often lists the link-local first. If you identify interfaces by the first address printed, you will conclude the television-facing NIC is a different one than it is. Read the whole line.

Fix

Point the entire multicast range at the interface facing the televisions, and make the change survive a reboot — a hand-typed ip route command is lost the next time the appliance restarts, which is how this fault returns weeks later with nobody remembering the first time. Installation covers the supplied routing unit, which re-evaluates the route periodically so it follows the cable if the link moves.

Fix the range, never individual groups

Adding a host route for one group — a /32 for 239.10.0.1 — makes that one channel work and leaves 224.0.0.0/4 still broken. Every channel added afterwards is silently unroutable, and it will not be obvious why, because the channels that were working still work.

That exact mistake has produced two separate multi-hour incidents. One presented as a container restarting every fifteen minutes for hours; the other as a single channel churning every thirty seconds. Both were investigated as engine and watchdog bugs. Neither was. Route the range.

Castmux checks the routing table from outside the process and will report a destination as unroutable when it is. It deliberately does not restart the channel for this condition: restarting cannot fix a routing fault, and a restart loop would bury the one log line that names the real cause. See Monitoring.

Pixelation and stuttering on one channel

The picture breaks up in blocks, or freezes and jumps, on one channel while others are fine.

Most likely: two senders on the same group

Two processes transmitting to the same multicast destination. The televisions receive both streams interleaved, and the result is continuous corruption. The usual origin of this is a migration: an older service still running from a previous install, or a system unit that was disabled but not stopped, alongside the new deployment. Both senders look completely healthy from their own point of view, because each is producing a valid stream.

Confirm with the duplicate-sender alert, which is raised when Castmux observes traffic to one of its destinations that it did not send, and by looking on the appliance and on the network for another process transmitting to the same group and port. Fix by stopping the older sender properly — disabling a unit does not stop something else from starting it, so check scheduled jobs as well.

Also consider

• Source-side loss. The origin is delivering a damaged stream and you are faithfully forwarding it. To see this honestly, run the channel on the pass-through engine: it preserves the original continuity counters, so downstream analysis shows the loss. The remux engine re-stamps continuity counters, which makes the output look structurally clean and hides where the damage came from. See Streaming engines.
• An oversubscribed multiplex, if the channel is carried on DVB-C. If the combined bitrate of the services exceeds what the carrier can hold, everything in that multiplex degrades, not only the channel somebody reported. See DVB-C multiplexing.
• Switches without IGMP snooping. Without snooping, every multicast stream is flooded to every port. A dozen channels then arrive at every television and at every other device on the segment, and access ports saturate. The symptom is usually pixelation on many channels at once, worsening as you add channels.

A channel restarts repeatedly

Uptime never grows beyond a minute or two. Start by reading that channel’s log in the interface: the reason for each restart is recorded, and it distinguishes most of the causes below without any further investigation.

Source unreachable or intermittent

The origin drops the connection, or is only sometimes available. Castmux backs off progressively rather than hammering it, and the backoff resets after a sustained healthy run. Check the origin before changing anything on the appliance.

The origin publishes an IPv6 address the appliance cannot reach

Extremely common, rarely suspected. See Everything is slow to start below — it causes both symptoms.

SRT latency set too low

The configured latency must cover the round-trip time of the link with room for retransmission. Set too low, recovery fails and the session drops repeatedly.

A watchdog keyed on a metric that does not exist for the engine

External monitoring alerting on frames-per-second or playback speed will restart healthy pass-through channels forever, because those metrics do not exist for that engine. See the warning in Monitoring.

An oversubscribed uplink at startup

If every channel starts at once, each opens its origin connection simultaneously and the uplink cannot serve them all; several time out, restart, and try again together. The startup stagger exists specifically to prevent this. If it has been reduced or disabled, restore it.

A channel will not start at all

Check these in order. The list is ordered by how often each turns out to be the answer, not by how interesting it is.

1. Is the channel active? A channel that has been deactivated is not scheduled at all.
2. Is it on-demand with no viewers? This is the most common false alarm on the page. An on-demand channel with nobody watching is supposed not to be running. Confirm by opening its preview, which registers as a viewer and starts it. See On-demand streaming.
3. Is the source reachable from the appliance? Not from your laptop — from the appliance. Different DNS, different routing, different firewall rules, possibly a different VLAN. A source that plays perfectly in VLC on your machine tells you the origin is alive, and nothing about whether the appliance can reach it.
4. Is the destination already in use by another channel? Two channels configured with the same group and port is the same duplicate-sender fault as above, created locally.
5. On DVB-C, could the channel be probed? A service that could not be probed cannot be assigned to a multiplex. See DVB-C multiplexing.

Everything is slow to start, and fetches take seconds

Channels take an unreasonable time to come up, playlist fetches take seconds, and the slowness is inconsistent — sometimes the same request is fast.

Cause

The origin publishes an IPv6 AAAA record and the appliance has no working IPv6 route. Address selection prefers IPv6, so every HTTP request tries IPv6 first and only then falls back to IPv4. The cost of that fallback depends entirely on how the IPv6 attempt fails:

How the IPv6 attempt fails	Cost per request
Immediate refusal	A few hundred milliseconds. Irritating, survivable.
Blackholed route	A full connect timeout, with nothing to abort it early. This is where multi-second stalls come from.

Paid on every playlist and every segment request, this is enough to make a channel miss its timing and restart. In one case it produced a channel restarting 765 times in a single day.

This one gets misdiagnosed, so measure it properly

The same incident was initially attributed to a slow origin on the basis of one timed fetch that took 10.8 seconds. Repeat runs of the identical fetch took 0.59 seconds. A single sample of a fault that only sometimes hits a timeout will tell you whatever you already believed. Run the fetch ten times and look at the distribution.

Fix

Configure the appliance to prefer IPv4 over IPv6 in address selection. Do it in both places: on the host and inside the container, which carries its own copy of the address-selection policy. Fixing only the host leaves the streaming process — the one making the requests — unchanged, and the symptom does not move at all, which is usually read as the diagnosis being wrong.

The channel plays but there is no sound, or the wrong language

Picture is correct, audio is missing, or only one language is available where the source carries several. Three distinct causes:

• The remux engine forwards a single audio track. Its default stream selection takes one video and one audio stream and drops the rest. If the site needs multiple languages, run the channel on the pass-through engine, which carries every audio PID along with the original programme map and its language descriptors, so the television can offer the choice. See Streaming engines.
• A demuxed HLS ladder. Where the origin publishes audio as separate renditions rather than muxed into the video variants, selecting a single video variant would put silent video on air. The resolution-profile guard detects this and keeps the master URL for exactly that reason, and says so on the channel. If you have overridden the source URL by hand to point at a video variant of such a ladder, that is where the audio went. See Resolution profiles.
• The source genuinely carries one track. No engine setting creates an audio track the input does not contain. Confirm against the origin before pursuing anything else.

A switch in the interface disagrees with the badge next to it

A toggle shows one state while the status rendered beside it, from the same underlying value, shows the other.

This is browser behaviour, not application state. On a soft reload or a back-navigation, browsers restore form controls to the state they held before, and ignore the state the server sent in the page. The badge is plain text and reflects the server; the switch is a control and reflects your browser’s memory of the previous page.

Force a hard reload and the two will agree. The reason this shows up on a product like this at all is that channel identifiers are reused after a delete: create a channel, delete it, create another, and the new one occupies the old identifier — so the browser cheerfully restores the deleted channel’s toggle onto its replacement.

Televisions lost their channel numbers after a line-up change

On DVB-C, televisions store channels against the service identities carried in the multiplex. Changing the order of slots changes those identities, so the televisions no longer recognise what they had stored and must be re-tuned — across the whole building, which is precisely the job nobody wants.

Until service identity can be pinned independently of slot order roadmap, avoid reordering a live multiplex. Append new services rather than inserting them, and plan the running order before the estate is tuned rather than after. See DVB-C multiplexing.

What to send to support

The following turns a multi-day exchange into a single reply. Gather it before writing the message, not in response to being asked.

• The channel name, its source URL and its destination group and port.
• The per-channel log for that channel, from the interface.
• The output of ip route get <group> for the affected destination.
• The output of ip -brief link.
• The configuration file, with credentials removed.
• Whether the fault affects one channel or all of them — this alone separates a source problem from a network problem.
• Whether anything changed on the network. A switch replaced, a VLAN altered, a cable moved, a firmware update. The answer is more often yes than people expect.