Bench Work

Almost every broadcast engineer has to do some type of bench work. While I enjoy a certain amount of bench work, it is not my strong suit.  I suppose if I had to do it more often, I would become more proficient.  Truth be told, I would rather be at a transmitter site than sitting work chair studying schematic diagrams. It is becoming increasingly difficult to make repairs in the field due to surface mount components.  The company I work for has a repair and rework shop where almost anything can be repaired.  There is one bench tech, who is pretty proficient with power supplies and RF amplifiers among other things.  There is a complete set of test equipment including several Tektronix spectrum analyzers and oscilloscopes.

Likely the most versatile piece of equipment is the IFR 1500 service monitor.

IFR 1500 communications service monitor
IFR 1500 communications service monitor

The bench itself is fairly large:

Shop work bench
Shop work bench

There is also a good stock of spare equipment that can be rented out while repairs are being made:

Shop spare equipment
Shop spare equipment

Repair work includes by is not limited to:

  • RF repairs; Moseley STL systems, Marti STL and RPU systems, TFT STL systems, most exciters, IPA modules, etc
  • Transmitter repairs and retuning
  • Mechanical devices like transmission line dehydrators, transfer switches, etc
  • Switching and linear power supplies
  • Uninteruptable power supplies
  • Remote control equipment; Gentner VRC-2000, Burk ARC16, Moseley MRC-1600
  • Audio Processing; All Orban equipment, Symetrix, Valley, DBX
  • Audio equipment; Amplifiers, consoles, reel to reel machines, cassette decks, CD players, DAT machines, etc

I am sure there are many other things that I am leaving out.

Troubleshooting

Good troubleshooters are becoming rare these days.  To some, the idea of working through a problem, finding and then fixing an issue seems like a time-consuming, wasteful evolution.  More often than not, it is easier to replace the entire assembly with a new one, throwing the old one away.  This is especially true with computer components.  The other option is to send a module or assembly back to the factory for repair.  Truth be told, often that is a good course of action when a fully equipped repair bench is not available.  Surface mount technology can be difficult to repair in the field, as can many RF components.

Being able to troubleshoot components and assemblies is still a valuable skill.  Finding and identifying trouble is a good skill no matter what it is used for.  I find analytical troubleshooting skills to be good life skills to have.  I think my in-laws are occasionally amazed when I walk into a situation and point to something and say: There it is, fix that.

Coil burned out on 40 amp RF contactor
Coil burned out on 40 amp RF contactor

Many times, however, there is no smoking gun. Those situations require a bit of investigative work. The first step in troubleshooting is developing a history:

  • Has this failed before
  • Is there a history of failures
  • Has it been worked on recently
  • Is it new
  • Has it been installed properly
  • It is old
  • Has it been affected by some outside force like lightning or a power surge

This is where good maintenance records or maintenance logs come in handy.  Recently, I have found many places that lack any type of maintenance documents, which means the repair history is unknown.  This makes it difficult to find a good starting point and can greatly increase the amount of time required to troubleshoot a problem.

Once the pertinent history is gathered, it can be organized and analyzed for clues.  For example, if something has been worked on recently, that is a good place to start. If something has a past history of failures, that is a good place to start.  Newly installed equipment is subject to early failures under warranty due to component failures.  Old equipment may just be plumb-worn out.  Improperly installed equipment can exhibit all kinds of bizarre failure modes.   That information coupled with known symptoms would indicate a good starting point for troubleshooting the problem.

If no good starting point can be discerned, then the next step is to recreate the failure.  This usually means turning the thing back on to see what it does.  Chances are good that whatever the problem is, it will still be there.  Once a good set of symptoms have been identified, then it is time to start working at one end of the problem unit once the failed component is isolated.

Oftentimes, equipment manuals will have troubleshooting guides.  These can greatly speed up the process for large, complicated things like transmitters, generators, and so on.  There is also the tried and true troubleshooting chart:

Generic transmitter power supply trouble shooting chart
Generic transmitter power supply trouble shooting chart

This is an example of a troubleshooting chart for a transmitter power supply.  Many equipment manuals will have this type of information in the maintenance sections.

It is also important to note that when working on high-voltage systems, it is necessary to have two persons on-site at all times.

Good troubleshooting skills have many applications.

RoHS and Electronics Reliability

ROHS stands for Restriction of Hazardous Substances Directive. It is a mainly European effort to reduce lead (Pb), Mercury (Hg), Cadmium (Cd), Hexavalent Chromium (Cr+6), Polybrominated Biphenyl (PBB), Polybrominated diphenyl ethers (PBDE) and Acrylamide in electronics and consumer goods.

The main effort appears to be in the reduction of lead in circuit boards and solder.  Generally speaking, the reduction of pollutants is a good thing.  Lead is toxic, especially to young children. Mercury is a potent neurotoxin.  Those other elements and chemicals don’t sound good either.

There are all sorts of green logos and other nice-looking things attached to products that meet the standard.

Typical ROHS label
Typical ROHS label

I feel better, don’t you?

Now for the other side of ROHS.  According to Lead Free Electronics Reliability (large .pdf) by Dr. Andrew Kostic, the effort had been hugely expensive with very limited results:

A huge (~ $14B annual revenue) semiconductor manufacturer estimated the annual worldwide Pb reduction per 1,000,000,000 integrated circuits was only equivalent to ~100 automobile batteries.

Wow!  That is simply amazing on the face of it.  Over the years, I have probably found and carted at least 10 old car batteries to the recycling center for a few dollars each.  According to the Kostic paper:

(Computer chip manufacturer) Intel’s efforts to remove lead from its chips have so far cost the company more than $100 million and there is no clear end in sight to the project’s mounting costs

Wouldn’t $100m be better spent on other, more pressing pollution issues?  Fukushima, springs to mind.

Further, the replacement metals used in electronics have some problems of their own.  They may be better for the environment, however, they lack testing and are

Not optimized for high reliability, severe stress, long life applications

Further, replacing parts in legacy equipment using ROHS parts and solder may present problems with bonds between dissimilar metals.  Thus, making field repairs, or any repairs impossible.

Many of the newer solders and circuit boards use Tin (Sn) as the finishing metal.  There is a problem with tin, known as Tin whiskers.  This was first noted at the Bell Labs in 1947.  Small hairs grow out of the surface of the metal, acting as short circuits, and at higher (above 6 GHz) frequency RF, antennas.   This happens with other metals such as Zinc, Silver, and Gold.

Silver Sulfide Whiskers on circuit board
Silver Sulfide Whiskers on circuit board

As you can probably deduce, this can have certain detrimental effects on the performance of the circuits in question.  I can imagine all sorts of strange behavior from controllers and other bits and parts of equipment.

I don’t know how prevalent this is in Europe where the directive has been in effect for 6 years or so.  It would be interesting to find out.  I also wonder how many US manufacturers are adopting RoHS as the de facto standard in order to do business in Europe.

How much is prevention worth?

I sometimes get the distinct impression that the corner office doesn’t understand what it takes to keep a radio station on the air and in good repair.  It is most often the problems or “issues” that tend to get the most attention.  The things that are working well tend to get ignored. After all, how often do you hear a news report about the airliner that landed safely?

Lightning strike TV tower
Lightning strike, TV tower

When lightning strikes the tower and knocks the transmitter off the air causing major damage and expensive repairs, that is a problem.  When lightning strikes the tower and nothing happens, no problem.  What is the difference between those two situations?

Grounding strap, FM transmitter site
Grounding strap, FM transmitter site

If the generator starts and runs during every power outage and has done so for the last five years straight, it is obviously a reliable unit, does it need all that maintenance?

Caterpillar 75 KW diesel GENSET
Caterpillar 75 KW diesel GENSET

Money spent on preventing undesirable outcomes can be difficult to quantify as disasters and events that do not happen are ill defined.   It is difficult to quantify the “amount saved” on something that didn’t or won’t occur.  Using past situations is good start, but that only covers a fraction of possible outcomes.  In order to invest money wisely, one has to look at the probabilities.  If there is an unlimited budget, then the probability exercise should be minimal, however, there is very seldom an unlimited budget.

For example, how much does a back up STL system cost vs the risk of being off the air while the main STL system is being repaired?  How often do failures occur, when are they likely to occur and for how long are all good questions.  Is there an alternative to a full backup like an IP CODEC?  Such a solution would cover all aspects of the STL system including antennas, transmission line, transmitters and receivers.

There are certain FM stations north of here that have neither RADOMES or antenna heaters.  Once every two years or so, the antenna ices up and the transmitter folds back due to VSWR.  How much of an impact to listeners notice when this happens?  If it happened more often, say two to three times a year, would it be wise to invest in some type of deicing equipment?

What is the ownership and management opinion on off air conditions?  I have often heard tell “Oh, its only the AM, we don’t mind if it goes off the air.”  That is, until it actually goes off the air, then it is a big problem.

Based on my and others experiences, these are the things that will happen at an average transmitter site:

  • The electricity will go off at least once per year for several hours.
  • The main transmitter will fail at least once every two years.
  • Lightning will strike the tower at least once per year.
  • The STL system will fail, at unknown intervals.

At studio sites, these things will occur:

  • The file server will crash depending on the operating system
  • The telephone lines and or T-1 service, internet service, ISDN etc will go out
  • The electric power will go out for several hours
  • The satellite dish will fail once every two to three years
  • If there is a tower, it will get struck by lightning

Other site specific things can occur like floods, blizzards, earthquakes, fire, etc.

Money spent on backup systems for those items is good insurance.  Not only will the station stay on the air, the on call engineer’s phone will ring less often, which, if you are the on call engineer, should make you happy.

If a full backup is not available, a second transmitter for example, having a good stock of spare parts on hand can mean the difference between an early evening and an all nighter.   Keeping good maintenance logs and well documented repair records can point out trends and give a good basis for ordering spare parts.

Repair trends are important.  If the same part seems to be going bad over and over, it is time to dig deeper and find the cause of failure.

The old adage “An ounce of prevention is worth a pound of cure,” still holds true.