Troubleshooting

Good troubleshooters are becoming rare these days.  To some, the idea of working through a problem, finding and then fixing an issue seems like a time-consuming, wasteful evolution.  More often than not, it is easier to replace the entire assembly with a new one, throwing the old one away.  This is especially true with computer components.  The other option is to send a module or assembly back to the factory for repair.  Truth be told, often that is a good course of action when a fully equipped repair bench is not available.  Surface mount technology can be difficult to repair in the field, as can many RF components.

Being able to troubleshoot components and assemblies is still a valuable skill.  Finding and identifying trouble is a good skill no matter what it is used for.  I find analytical troubleshooting skills to be good life skills to have.  I think my in-laws are occasionally amazed when I walk into a situation and point to something and say: There it is, fix that.

Coil burned out on 40 amp RF contactor
Coil burned out on 40 amp RF contactor

Many times, however, there is no smoking gun. Those situations require a bit of investigative work. The first step in troubleshooting is developing a history:

  • Has this failed before
  • Is there a history of failures
  • Has it been worked on recently
  • Is it new
  • Has it been installed properly
  • It is old
  • Has it been affected by some outside force like lightning or a power surge

This is where good maintenance records or maintenance logs come in handy.  Recently, I have found many places that lack any type of maintenance documents, which means the repair history is unknown.  This makes it difficult to find a good starting point and can greatly increase the amount of time required to troubleshoot a problem.

Once the pertinent history is gathered, it can be organized and analyzed for clues.  For example, if something has been worked on recently, that is a good place to start. If something has a past history of failures, that is a good place to start.  Newly installed equipment is subject to early failures under warranty due to component failures.  Old equipment may just be plumb-worn out.  Improperly installed equipment can exhibit all kinds of bizarre failure modes.   That information coupled with known symptoms would indicate a good starting point for troubleshooting the problem.

If no good starting point can be discerned, then the next step is to recreate the failure.  This usually means turning the thing back on to see what it does.  Chances are good that whatever the problem is, it will still be there.  Once a good set of symptoms have been identified, then it is time to start working at one end of the problem unit once the failed component is isolated.

Oftentimes, equipment manuals will have troubleshooting guides.  These can greatly speed up the process for large, complicated things like transmitters, generators, and so on.  There is also the tried and true troubleshooting chart:

Generic transmitter power supply trouble shooting chart
Generic transmitter power supply trouble shooting chart

This is an example of a troubleshooting chart for a transmitter power supply.  Many equipment manuals will have this type of information in the maintenance sections.

It is also important to note that when working on high-voltage systems, it is necessary to have two persons on-site at all times.

Good troubleshooting skills have many applications.

RoHS and Electronics Reliability

ROHS stands for Restriction of Hazardous Substances Directive. It is a mainly European effort to reduce lead (Pb), Mercury (Hg), Cadmium (Cd), Hexavalent Chromium (Cr+6), Polybrominated Biphenyl (PBB), Polybrominated diphenyl ethers (PBDE) and Acrylamide in electronics and consumer goods.

The main effort appears to be in the reduction of lead in circuit boards and solder.  Generally speaking, the reduction of pollutants is a good thing.  Lead is toxic, especially to young children. Mercury is a potent neurotoxin.  Those other elements and chemicals don’t sound good either.

There are all sorts of green logos and other nice-looking things attached to products that meet the standard.

Typical ROHS label
Typical ROHS label

I feel better, don’t you?

Now for the other side of ROHS.  According to Lead Free Electronics Reliability (large .pdf) by Dr. Andrew Kostic, the effort had been hugely expensive with very limited results:

A huge (~ $14B annual revenue) semiconductor manufacturer estimated the annual worldwide Pb reduction per 1,000,000,000 integrated circuits was only equivalent to ~100 automobile batteries.

Wow!  That is simply amazing on the face of it.  Over the years, I have probably found and carted at least 10 old car batteries to the recycling center for a few dollars each.  According to the Kostic paper:

(Computer chip manufacturer) Intel’s efforts to remove lead from its chips have so far cost the company more than $100 million and there is no clear end in sight to the project’s mounting costs

Wouldn’t $100m be better spent on other, more pressing pollution issues?  Fukushima, springs to mind.

Further, the replacement metals used in electronics have some problems of their own.  They may be better for the environment, however, they lack testing and are

Not optimized for high reliability, severe stress, long life applications

Further, replacing parts in legacy equipment using ROHS parts and solder may present problems with bonds between dissimilar metals.  Thus, making field repairs, or any repairs impossible.

Many of the newer solders and circuit boards use Tin (Sn) as the finishing metal.  There is a problem with tin, known as Tin whiskers.  This was first noted at the Bell Labs in 1947.  Small hairs grow out of the surface of the metal, acting as short circuits, and at higher (above 6 GHz) frequency RF, antennas.   This happens with other metals such as Zinc, Silver, and Gold.

Silver Sulfide Whiskers on circuit board
Silver Sulfide Whiskers on circuit board

As you can probably deduce, this can have certain detrimental effects on the performance of the circuits in question.  I can imagine all sorts of strange behavior from controllers and other bits and parts of equipment.

I don’t know how prevalent this is in Europe where the directive has been in effect for 6 years or so.  It would be interesting to find out.  I also wonder how many US manufacturers are adopting RoHS as the de facto standard in order to do business in Europe.

Troubles at the Tower

3 tower AM directional array
3 tower AM directional array

Troubles at the AM tower; I don’t know why, it won’t switch power.
Over the phone I can tell, the program director’s day is not going very well.
Press the “day” button but there is no kerchunk, the directional coupler shows the load is junk.
Out into the big field, I go to find the problem quickly and fix it just so.
The wind is cold, the snow is deep, I think of the contract terms I must keep.
Reaching the tuning house, take out the keys, lock, do not be frozen, please.
Once inside, there I find, no big surprise, the mice have been a working this pre-sunrise.
A nest they have build in a most inconvenient place, in the back of the phasor wiring chase.
Oh, the wires they have chewed, the circuit’s destroyed, all for the lack of mousetraps deployed.
As I reach in to clean out the mess, the smell of mouse makes me gag, I confess.
The fuses are blown, the contactor is jammed, perhaps, if I am lucky, I can move it by hand.
A large screwdriver strategically employed, I pry up slowly, further damage to avoid.
The bar thunks up, the contacts engage, the transmitter is ready to apply amperage.
Call on the cell phone, tell them it’s fixed, stand back and watch the base current meter, transfixed.
Then; Up it goes! Wonderful radio frequency current flows!
I clean up, lock the door, lock the gate, carrying bad news the owner will hate.
The damage is grave, the repair bill is steep, if a good relationship with the FCC you desire to keep.
Business is off, the accounts are low, is this really necessary, he wants to know.
The terms of the license are your obligation to keep, getting caught out of tolerance will not be cheap.
Looking forlorn, the owner says in disgust, it is only the AM, but fix it if you must.
Happy as a lark, with a song in my heart, I dig though the manual and order the part.
Time to go home, eat breakfast, brush teeth, take a shower. I have another client to see before the noon hour.

40 amp RF contactor
40 amp RF contactor

Dedicated to all those who have been there, done that and the breed of RF men and broadcast engineers who are slowly fading away.

Generators and mice

Never a good mix, unfortunately, it usually turns out bad for the mice and sometimes the equipment.  This is an Onan GGMA 20 KW propane generator installed in a rural area, not that the location matters that much.  Mice will find what they perceive as a safe secure spot to hold up for the winter.

Onan GGMA20 propane generator
Onan GGMA20 propane generator

Unfortunately, the mice decided that the generator cooling fan was a good place to make a nest.  It probably was until the generator started, then the mice had a quick lesson in centripetal force.

Mice and generator
Mice and generator

This will require some additional maintenance in the springtime when I change to oil.  By that time, the carcasses should be mostly dried out and easier to deal with.

Onan generator mice
Onan generator mice

The mice are generally a nuisance, getting into ATU’s, transmitters, electrical panels, spare parts boxes, etc.  Once in place, they begin to breed and reproduce.  The gestational period for a mouse is 21 days, which means populations rapidly increase creating further problems. If left alone, mice will chew through electrical insulation, control wires, cardboard boxes, packing material, and so on.  They tend to carry diseases like hantavirus and bubonic plague.

I don’t usually agree to using poison to get rid of pests, it tends to linger in the environment and accumulate up the food chain.  However, judicious use of some type of poison is usually the only way to effectively get rid of a mouse infestation.

Wherever possible, make sure that all openings and holes into equipment and buildings are sealed up.  Do not kill snakes and other predators, who will assist in keeping the mice in check.  Employ traps and wear gloves when removing dead mice and mouse parts.  Beware of fleas.