I have just finished putting back together this PRE BMX III console.
PRE BMXIII analog audio console, reassembled
We basically ripped the guts out of this unit and in doing so, I was reminded of how well these things are built. The PRE BMX series consoles were truly wonders of audio engineering. It is a testament to their ruggedness and serviceability that so many of these units are still in use twenty to thirty years after they were manufactured.
This console suffered some pretty bad water damage to the backplane:
PRE BMXIII module backplane
Which was replace, along with many switches and buttons. The Mic2, Mic3 and CD1 modules seemed to have taken most of the damage, there were several logic ICs and IC sockets that needed to be replaced on those modules. Of course, this was not inexpensive; the parts were somewhere north of $3K plus about 40 man hours of labor… that adds up fast.
The good news, I think that the studio was back in service last night.
Or whatever those guys are called these days:
Especially when that sink is located on the second floor, above the studio on the first floor. ‘Tis but a small thing really, one of those little details, but in light of the sink also being clogged, it becomes very significant. That, coupled with the fact that the building is uninhabited at night and disaster is afoot.
The water was running slowly all night…
Wet ceiling tiles
It filled up the sink. It ran across the floor. It soaked the carpet. It seeped into the sub floor and out of the ceiling on the first floor and then into this nice Pacific Recorders BMX III console.
Pacific Recorders BMX III console, draining
Pacific Recorders BMX III console, draining/drying
You know that burning electronic/plastic smell? Yeah, that’s it, mixed with stale funky water, wet wood and a nondescript mildewy odor; that is what the room smells like. Very pleasing. The furniture below the console was soaked too:
Studio furniture after water damage
Some of the input module edge connectors; they didn’t fair so well:
PRE BMXIII burned edge connectors
The backplane for the power supply buss has to be replaced and these switches with the water bubbles in them, they have to go too:
Pacific Recorders BMXIII buss select switch full of water
We dried out the furniture with an industrial strength hair dryer. By three PM we had unsoldered all of the bad parts and cleaned off the modules and the console back plane.
Parts for repairs are on order from Mooretronix. I doubt this will be repaired before next Tuesday.
Somebody came in and was all “awww, this sucks bla bla bla.” Well, maybe, but I get paid by the hour and frankly, there are much worse things that I could be doing…
This is a picture of a surge module taken from an LEA series type surge suppressor:
LEA 600 volt MOV module
Looks like it took a pretty significant power hit, enough to explode several MOV’s. This site is at the end of a long transmission line that stretches across an entire county. Over the years, the station has made many complaints to the utility company about the quality of their power and the frequency of interruptions encountered at this transmitter site. Occasionally, something will happen. Often times it is the figurative shoulder shrug.
That is why we installed the surge suppressor.
This is a universal truism that can also be expressed as “Murphy’s Law.” I don’t rightly know how Murphy received credit for this, however, I chalk it up to either the luck of the Irish or the gift of self promotion. Either way, that principle was demonstrated again with a 950 MHz STL link between Mt. Beacon and Peekskill, NY for WHUD.
I had noticed, while doing some transmitter maintenance, the receive signal strength of the STL had dropped from 300 µV to 30 µV. That is an alarming development. Therefore, we scheduled a tower crew for the next day, not wanting to go off the air over the coming holiday, which would be a sure bet otherwise. Upon arrival, the tower crew noticed a strange thing in the STL transmission line at the base of the tower, which looked like some type of a splice. Truth be told, I have been associated with this station since 1999, and had never noticed the splice before. This STL system was installed in 1998, when the station’s studio moved from Peekskill to Beacon. I can say, of all the things that have gone wrong over the years, this STL system was always very reliable. Regardless of that, I quick check with a spectrum analyzer showed a 3 dB return loss at 137 feet (41.75 m), exactly the distance from the transmitter room to the base of the tower.
3 dB return loss, distance to fault 137 feet
A 3 dB return loss coincides exactly with the drop in received signal strength at the other end of the path. Thus, the tower crew took apart the splice and water poured out of it. I would estimate at least 4-6 ounces of water (180 ml), perhaps more.
7/8 coax cable splice connector, opened up
We then began to take in the details:
- The 7/8 coax coming out of the building was Cablewave FLC78-50J
- The 7/8 coax going up the tower was Andrew LDF4-50A
- The splice connector was Andrew L45Z
- The center conductor threaded connector did not fit properly into the Cablewave cable, it was too loose.
- The cable was chaffing on a tower leg, about 50 feet above the splice because it was not properly secured to the tower
- The 7/8 splice connector was missing an O ring on the backnut of the Cablewave cable
Thus, water ingress causes the high return loss. Problems with this system began immediately after Hurricane Irene, the end of last August. We were able to make a temporary fix using two type N connectors of the proper manufacture for each type of cable. The radio station returned to air just before noon, about 45 minutes after turn off. After the repair, the return loss dropped to about 20 dB, which is good.
The permanent fix is for the entire run of cable from the transmitter room to the STL antenna to be replaced. That type of line splice should have never been used on a 950 MHz STL, and it was certainly wrong to mix cable types with an Andrew connector. Those little details will always manifest themselves eventually.
This picture reminded me of something that happened early on in my radio career:
WDCD three tower array, Albany, NY
This is another view, looking across from the roof of the transmitter building before the former studio building was removed:
WFLY STL antenna, circa 1992
The story dates back to 1990 or so. In the second picture, one can see two Scala PR-950U Paraflector antennas. These are the STL and TSL antennas for WFLY. They are on wooden utility poles because of the WPTR 1540 KHz antenna system is behind the poles, out of the picture to the left. As you can see in the second picture, these poles were immediately behind the studio building, known as the “Gold Studio, ” the name itself being pure propaganda.
Also, in the second picture you can see behind the poles, a pair of poplar trees. The reason for the second, taller pole was because across the street, out of the picture to the right, there was a stand of poplar trees which were growing up into the path of the WFLY STL system.
When this was noticed, then General Manager, John Kelly, tactfully approached the property owner and asked if the radio station could cut the “popular” trees down. Of course, the property owner wanted much money to do this. There was many telephone calls and discussions on how to kill the “popular” trees and other, not so ethical solutions to this growing problem. Finally, it was decided that it would be simple and less expensive to install the taller utility pole.
Thus, Northeast Towers found the utility pole and came to install it. In this area of Albany, the soil is a sandy loam, which required much hand digging and back bracing in the hole before they placed the pole in the ground. As it is a seventy foot pole, a good 12 feet was placed in the ground and the hole was back filled with concrete. That is why the pole still stands today.
Naturally, all of this work is taking place on the hottest day of the year. Also, it stands to reason, the guy in the hole doing the manual labor is the oldest, most out of shape person on the crew. After lots of grunting and swearing, our man comes out of the hole looking whiter than the driven snow and sweating profusely. He kind of staggered into the back door of the building and collapsed on the floor just inside the back door. At this point, he was in full cardiac arrest. The promotions director, who’s office was closest to the door, called the ambulance.
Fortunately, the board operator on WPTR was an EMT with the local fire department. After his pager went off, he ran out to his car, got his EMT bag and arrived on scene within seconds. He was able to start CPR quickly. In the mean time, a crowd had gathered out in the hallway. John (the General Manager), hearing the commotion, storms out of his office and down the hallway. He gets to the edge of the crowd and yells:
“WHAT ARE YOU PEOPLE DOING HERE? DON’T YOU HAVE JOBS TO DO? AND WHAT IS THAT GUY DOING LAYING ON THE FLOOR?”
The good news is, the guy survived, thanks in no small part to the quick action of the board operator.
Anyway, tales of radio when it was fun.
Class Charlie fire in the transmitter room electrical panel. Away fire party from repair locker forward. Set condition ZEBRA throughout the ship, this is not a drill.
Or something like that. If you were driving around Albany, NY this afternoon and noticed WDCD-FM was off the air, this is the reason why.
WDCD AM/FM 480 volt 3 phase AC main distribution panel
A little after noon time, the 480 volt main distribution panel at WDCD AM/FM caught fire, taking the FM station off the air.
WDCD conference room clock, time of power outage noted
According to this clock, it happened at 12:19 pm, when there was a loud bang and the lights in the studio flickered several times, followed by the building fire alarm going off. Thankfully, a quick response by the station staff and the Town of Colonie fire department limited the damage to the interior of the distribution panel. Other than the dry chemical fire extinguisher residue all over the place, the building is none the worse for wear.
WDCD distribution panel burned parts
The 480 Volt three phase electrical distribution panel was installed in 1947 when the original building was constructed. The power company cut the power to the building and an electrician was able to re-route the distribution for the dry step down transformers that power the studios and equipment racks. The original 480 volt service was installed due to the 50 KW AM transmitter for WPTR (WDCD-AM). Currently, WDCD-AM is silent, pending programming decisions by the owner, Crawford Broadcasting.
WDCD burned electrical distribution panel parts
So, we spent the late afternoon vacuuming the NextGen computers and UPS out, wiping down the equipment and making sure to clean out the power supplies and other nooks and crannies. Then, we powered everything back up, one at a time and to our pleasant surprise, all came back up without error. Total off air time for the FM station was about 6 hours.
This incident happened a few years ago. I thought I had lost the pictures of the disaster, but I found them this morning on my thumb drive. Hooray! This occurred one morning just before Christmas after the area received a snow/ice/rain storm. The gutters on the old ATT long lines building clogged with ice and the water on the roof built up. Unfortunately, the transmitter was installed directly below a disused exhaust stack for the former backup generators.
I received the off air call from the morning show while I was driving to the office. I diverted and went to the transmitter site and found water pouring into the top of the main transmitter.
WBPM transmitter room flood
Thus, water ran down directly into the top of the QEI FMQ-3500 transmitter (transmitter was upgraded to 6 KW). Unfortunately, high voltage and dirty stack water do not mix. The combination of sooty, iron laden water and the B+ damaged much of the transmitter circuits beyond repair. The main transmitter is on the right, the backup transmitter is on the left.
I inspected the backup transmitter, also a QEI FMQ-3500, and it seemed to me that no water made it into the unit. I rigged the tarp to ensure that none did, which was a very pleasing bit of work, what with the cold, smelly, dirty diesel water dripping on my head and running down my neck and back.
Top of WBPM QEI FMQ-3500 transmitter
The 1 5/8 coax switch was also damaged:
WBPM 1 5/8 coax switch
As was the remote control in the equipment rack:
WBPM Gentner remote control
Fortunately, the backup transmitter ran, although I pressed the plate on button with a dry wooden stick while standing on a dry, non-conducting ladder. Even so, I still felt a little trepidation holding that stick.
WBPM Saugerties, NY Nautel V-7.5 transmitter
It took almost a year, but finally the insurance company for the building owner came through, and a new Nautel V-7.5 transmitter was installed. I believe this is the last V transmitter Nautel made. We moved the transmitter location across the room, not under the old generator stacks. We also removed the generator stacks and patched up the roof with hydraulic cement and roofing tar. By the way, that yellow color should look familiar to anyone who ever worked inside of a Bell Telephone System building.
PIROD tower company has been around for a while, thus there are likely many of these tower light controllers out in the field. They perform a vital service in controlling and monitoring tower lights at remote transmitter sites maintaining a safe operating environment for aircraft and compliance with FCC rules.
PIROD was sold to Valmont in 2004, Valmont no longer manufactures or supports the product. All is not lost, however, as XCEL Tower Controls does support it and parts are still available through them.
These units were fairly rugged, had good surge suppression on the incoming AC lines and are designed for easy access to service parts.
PIROD PRLCA tower light controller, WRKI Brookfield, CT
This particular controller is being installed at WRKI in Brookfield, CT. We are adding toroid cores to the tower lighting circuits coming off of the tower because the last controller has been mostly destroyed by lightning. It is a tall tower, on top of a tall hill, thus it gets struck by lightning many times over the course of a year.
WRKI tower, Brookfield, CT
The block diagram looks like this:
PIROD PRLCA block diagram
Click for higher resolution.
The basic schematic looks like this:
PRCLA tower light controller schematic
Click for higher resolution.
The entire manual can be found here, (medium sized .pdf) courtesy of John Brickley of EXEL tower controllers.
Good audio clip below of the WIYY’s (Rock 98, Baltimore) console melting down during the morning show. These things happen from time to time. I often found, when it happened at one the stations I was working for, nobody would know anything about it. Nope. Just stopped working. What? No, I don’t know anything about the coffee dripping out of the bottom of the console.
Rock 98, (WIYY) Baltimore, coffee spilled into console courtesy of the Baltimore Sun.
If that link doesn’t work, try this one (7MB .mp3 file).
Pictures and stuff at their facebook page.
Off the air for twenty minutes during morning drive. I wonder what kind of console it was? From the pictures on facebook, it looks like Wheatstone stuff. Ouch! That’s going to leave a mark.
I tried to enforce a no eating no drinking in the studio rule. Most of the time I was successful, however, there were various incidents over the years. The worst was the morning show spilling “distilled water” in the console, but not saying anything about it. Months later, the air monitor stopped muting when the main mic was turned on. Nearly caused the guy who did the spilling to loose his hearing. Karma.
At another station, someone spilled soda on all the remote mic on/off/cough switches for the guest positions. That prompted an early morning phone call, which the morning show producer yelled at me and told me I must be at the station in five minutes (I lived about 25 minutes away at the time). Ha! I took my sweet time getting there. The soda cooked all the +5VDC regulators on the guest microphone modules, thus, for the next several days, all the morning show DJ’s had to share one microphone.
Old time radio guys will tell you, do not mess with the engineer.
Good troubleshooters are becoming rare these days. To some, the idea of working through a problem, finding and then fixing an issue seems like a time consuming, wasteful evolution. More often than not, it is easier to replace the entire assembly with a new one, throwing the old one away. This is especially true with computer components. The other option is to send a module or assembly back to the factory for repair. Truth be told, often that is a good course of action when a fully equipped repair bench is not available. Surface mount technology can be difficult to repair in the field, as can many RF components.
Being able to trouble shoot components and assemblies is still a valuable skill. Finding and identifying trouble is a good skill no matter what it is used for. I find analytical troubleshooting skills to be good life skill to have. I think my in-laws are occasionally amazed when I walk into a situation and point to something and say: There it is, fix that.
Coil burned out on 40 amp RF contactor
Many times, however, there is no smoking gun. Those situations require a bit of investigative work. The first step in troubleshooting is developing a history:
- Has this failed before
- It there is history of failures
- Has it been worked on recently
- Is it new
- Has it been installed properly
- It it old
- Has it been effected by some outside force like lightning or a power surge
This is where good maintenance records or maintenance logs come in handy. Recently, I have found many places that lack any type of maintenance documents, which means the repair history is unknown. This makes it difficult to find a good starting point and can greatly increase the amount of time required to troubleshoot a problem.
Once the pertinent history is gathered, it can be organized and analyzed for clues. For example, if something has been worked on recently, that is a good place to start. If something has a past history of failures, that is a good place to start. Newly installed equipment is subject early failures under warranty due to component failures. Old equipment may just be plumb worn out. Improperly installed equipment can exhibit all kinds of bizarre failure modes. That information coupled with known symptoms would indicate a good starting point for troubleshooting the problem.
If no good starting point can be discerned, then the next step is to recreate the failure. This usually means turning the thing back on to see what it does. Chances are good that whatever the problem is, it will still be their. Once a good set of symptoms have been identified, then it is time to start working at one end of the problem unit the fail component is isolated.
Often times, equipment manuals will have troubleshooting guides. These can greatly speed up the process for large, complicated things like transmitters, generators, and so on. There is also the tried and true troubleshooting chart:
Generic transmitter power supply trouble shooting chart
This is an example of a troubleshooting chart for a transmitter power supply. Many equipment manuals will have this type of information in the maintenance sections.
It is also important to note that when working on high voltage systems, it is necessary to have two persons on site at all times.
Good troubleshooting skills have many applications.