Secretly, I like it when things break, sort of

Not that I am a glutton for punishment or anything, but I enjoy troubleshooting. There is a certain satisfaction in the analytical aspect of tracking down a problem and fixing it, hopefully in a permanent fashion.  Figuring out where a problem is requires a good bit of detective work;

  • Examining the clues; what happened before the failure, what are the fault indications, are there any external factors
  • Round up the usual suspects; a good maintenance log is helpful here to track re-occurring failures.  If the failure cannot be attributed to an external source (such as a power surge or lightning storm), what was the last thing that was changed or worked on?
  • Following the trail back to the origin; Often the first failed part found is a symptom, not the actual problem.  It takes some skill in reading schematics and making sense of a failure to trace it back to the real problem.

It can sometimes be exciting, like turning on the 25 KV high voltage supply and having big blue flashes issue forth from the top of the transmitter.  Sometimes it can be quite frustrating, like when the station owners refuse to spend money to fix a problem.  Sometimes it can be dull, like fixing the same problem over and over again because of the previously stated money problem.  It’s also disheartening when the problem was caused by the stupid DJ spilling soda in the console.  Not that all DJs are stupid, just the ones that spill things into consoles.

The challenge of finding the root cause can often be enlightening.  I have often discovered unrelated problems waiting in the wings while investigating the why of an outage.  It is great to fix those things before they burn the house down, but this approach often goes unnoticed by the ownership or management.  Lately, for some reason, an ounce of prevention goes unnoticed or unappreciated.

There is quite a bit of science to troubleshooting, but there is some combination of personal traits that make a good troubleshooter.   These are:

  • Inquiring or curious disposition.  It is fairly easy to get to the first failed module or part.  Discovering the reasons behind the failure and or getting down to the component level takes a good deal more effort.
  • Patience.  This goes with the second part above, it takes some stick-to-it-tive-ness to trace out the not readily apparent problem.
  • Good analytical skills.  Often failures generate a cause-and-effect scenario.  The effects can be startlingly distractive and mask the causes and the underlying problem.
  • Ability to view the large picture.  This is critical to discover outside influences and other issues that are indirectly connected to the system or issue at hand.
  • Ability to analyze the system design.  This requires background training and experience to look at a circuit diagram and discover non-error-tolerant systems.  Sometimes these systems can be modified for better fault tolerance.

Poorly designed equipment is the bane of the broadcast engineer.  Equipment manufacturers can sometimes fail to follow two key principles: KISS and maintainability.  KISS stands for Keep It Simple, Stupid.  There is no better design criteria than the KISS principle.  Adding layers of complexity increases the failure expectations.  Maintenance can be something as simple as cleaning or changing air filters.  Making maintenance tasks difficult almost ensures that they will not be done.

Bathtub design curve
Bathtub design curve

Eventually, all things wear out.  It also takes some large-picture skills to know when it is time to replace equipment and that can vary greatly from situation to situation.

Wireless LAN bridges and STLs

Wireless LAN technology has been around for quite a while.  Point-to-point 2.4 and 5.8 GHz spread spectrum hardware has also been around for some time.  These systems operate in the ISM bands at relatively low power levels and are license free.  There are several different makes and models, however, they all have some similar specifications.  Most have DS-1 (T-1) or ethernet 10 base T or 100 base T 802.3 architecture.  Some are field configurable for either format.

The advantage of using 802.3 ethernet is the availability of ethernet sound cards and the possibility of making inexpensive audio to ethernet A/D converter.  However, if a station is currently using telephone company DS-1 service, then they likely have the audio to DS-1 multiplexers on hand.

Axxcelera AB Full Access outdoor unit
Axxcelera AB Full Access outdoor unit

I have used the Axxcelera point to point system as an inter-city relay for a satellite downlink.  Axxcelera is owned by Moseley, which has a long history in the STL business.  The point-to-point system has an indoor unit, which has the RJ-45 ports, and an outdoor unit, which has the RF section and an integrated antenna.  The two units are connected via Cat 5 cable (be sure to use UV resistant cable) through a power injection port.  The newer units do not need the power injection port.  The system gain is about +46 dBm and the claimed effective range is 20 miles.  I’d suspect it to be somewhat less than that with the integrated antenna.  There is also an N connector for an external antenna, making the outdoor unit a MMA (Mast Mounted Amplifier).

The indoor unit came configured with four 10 base T ports, which we reconfigured for DS-1 service.  We connected a QEI CATLINK T-1 mux with several 7.5 KHz audio channels and one control channel connected to a broadcast tool DSC 3264 (Starguide satellite receiver channel controller) that allowed the station to change channels on the fly.  It took some doing, but in the end, the system worked well.  The path was about 1/2 mile through downtown buildings, it was line of sight but did not have full Fresnel clearance.  I never heard of any dropouts.

The other system that I have used is the ADTRAN TRACER.  This system is different in that it does not have an integrated antenna, an external antenna is required.  We installed this over an eight-mile path using two six-foot grid parabolic dishes (Radio Waves G6-2.4NF) on 2.4 GHz.  The primary configuration is a rack-mounted indoor unit with either four or eight 802.3, E-1, or T-1 ports.  These ports are not field configurable.  The antenna connector is a type N.  There is also a split configuration available; an indoor unit with the T-1 ports and an outdoor MMA.  This setup is best used where transmission line lengths would create prohibitive losses.  MMA’s are not my first choice in these systems, there are too many things that can go wrong when active components are mounted high above ground level.

This system replaced two Telephone company T-1 lines and is used as an STL for two stations and an inter-city relay for a satellite downlink site.  It also extends the office LAN (802.3 ethernet) to the transmitter site where a backup audiovault server lives.  This is accomplished through a T-1 port using an ethernet to T-1 bridge.

ADTRAN Tracer 6000 series microwave radio
ADTRAN Tracer 6000 series microwave radio

ADTRAN also created a path analyser spreadsheet.

The license-free aspect of these system makes them easy to deploy.  There are several frequency plans available and the paths are fairly robust.  In highly congested areas, however, interference may become an issue.  Of course, because they are unlicensed, frequency coordination would be a real problem.  Axxcelera has the ExcelFlex which is a unit requiring a license that can run in any frequency band from 6-38 GHz

Temperature Sensor for Burk Remote Control

As a part of the re-wiring of a transmitter site, I realized that the site needed a temperature sensor.  I believe it is important for any remote transmitter site to have a temperature sensor, too many things can go wrong at a transmitter site.

I recall one incident at WGNA-FM in Albany, NY where a ventilation fan failed on a hot summer day.  The Harris HT-10 transmitter stayed on the air while the temperature climbed through 160 degrees inside the building, finally shutting down when the solid-state driver module failed.  This site had a temperature sensor and a live operator taking transmitter readings every three hours.  They faithfully logged the temperature increase in three-hour increments but didn’t call anyone until the station went off.  When I arrived there, it was so hot inside the building that I couldn’t touch a metal surface.

For the present-day project, I looked at the Burk BTU-4, which seemed a little pricey.  After doing a bit of quick research, I found the National Semiconductor LM34 a good alternative.  This unit puts out 10 mv/degree F, which can be directly calibrated to a metering input of an ACR-16 using the Lin scale.   I checked the accuracy by using a piece of melting ice, it was right on.  The design and implementation are very simple:

LM34 temperature sensor
LM34 temperature sensor

The LM34 is available in TO-46 or TO-92 case.  I bypassed the V+ with a 0.01uf capacitor.  It should be mounted in a small box with the case of the LM34 mounted on a metal cover or heat sink.  The TO-46 version, the case is ground.  It can be run on any voltage from +5 to +30 VDC, I used +5 because I had a little wall wart with the right voltage.  The wire is any type of balanced audio wire like Belden 8459, etc.  The LM34 cost 4 dollars, the rest of the parts were laying around.  It took about 15 minutes to create.

The site is air-conditioned, therefore I set the remote control to alarm if the temperature goes above 85 degrees F inside the building.

Transmitter trips main breaker

Received a call last night, after a particularly bad thunderstorm, that WGHQ in Kingston, NY was off the air.  Earlier in the day, the transmitter had tripped the main breaker after a thunderstorm.  I arrived at the transmitter site and found the breaker tripped again.  Once the breaker was reset, the transmitter came back on and ran without any overload indications.  The transmitter is a 10-year-old Nautel ND-5.

WGHQ Nautel ND-5 transmitter
WGHQ Nautel ND-5 transmitter

I was thinking of breaker fatigue as the breaker is the original 1960 breaker installed when the building was built.  I reset the breaker and turned the power output down to 3 KW, thinking the reduced load might not trip the breaker until we could get a replacement.  The transmitter was on the air running as I was about to lock up and go home when I heard, but more felt through the floor, a THUMP! There I stood and watched the transmitter go dark.

At least it happened when I was there looking at it.  Because of the lightning, I was thinking of something in the output network.  I reset the breaker and once again, no faults, and the transmitter came back on.  Strange.  Obviously some sort of power supply issue.  Here are the clues:

  1. The B- voltage was right where it should be at 72 volts.
  2. All other readings, reflected power, forward power, and power supply current are normal before and after the breaker trip
  3. No fault lights
  4. The service panel breaker, which was tripping, is rated for 70 amps, and the transmitter front panel breaker which did not trip, is 50 amps.

The Nautel factory rep was thinking either breaker fatigue or the big transformer in the base of the transmitter had gone bad.  According to him, no one had ever heard of a transformer going bad in these transmitters, which makes a certain amount of sense.  Unlike a tube transmitter, which steps the B+ voltage up several times, these transmitters reduce the B- voltage by about 2/3rds or so.  With a step-up situation, a surge would be multiplied many times and could very easily punch a hole in the transformer’s secondary winding insulation.  I have, in fact, experienced this on at least two occasions.

That leaves the wiring between the transmitter and the service panel.  I double-checked the panel breaker with my volt meter to ensure that the voltage was indeed off.  Then I removed each phase from the connection lugs in the transmitter and tested the wire to ground with my Fluke 77 DVM.  Sure enough, two of the phases showed resistance of 1.2 and 1.7 MΩ to ground where it should have been infinite.  Further, when I took the cover off of the service panel, I found a dead mouse.  Unfortunately, I didn’t have any #4 THHN, and all the home improvement stores were closed by that time, so it had to wait until morning.

The thunderstorm seems to be a coincidence.

After we pulled the wire out of the conduit, we found this:

mouse chewed feces encrusted electrical cable
Mouse chewed feces encrusted electrical cable

It is a little hard to see, but that shiny spot is copper.  The cable jacket is chewed back quite a ways and the entire thing is encrusted in mouse feces and urine.  I love to work on stuff like this.  LOVE IT!  Hantavirus, here we come!  That reminds me, I need to get some of those blue latex exam gloves and throw them in the truck…  Anyway, far back in the conduit running through the concrete floor where it bends to go up to the service panel, the mice apparently had a nest.  They got into the conduit under the transmitter, where it transitioned from 3-inch rigid to 1 1/4-inch flexible metal without the benefit of a junction box or proper fitting.

We pulled new copper conductors in and installed a proper junction/transition between the 3-inch and 1 1/4-inch conduit.  The service panel was also missing several knockouts of various sizes, which were sealed with knockout seals.  The transmitter was back on the air at full power about 16 hours after it went off.  Unfortunately, the station has no back up transmitter, so they were off for that period of time.  Perhaps now they will look into a backup transmitter or at least an exterminator, but probably not.