T-1 outage

One of our stations relies on a T-1 (DS-1) to relay audio from the studio to the transmitter site (STL).  This station started as a piece of paper, with no format, no staff, no real estate, and no studio equipment.  There was a transmitter and an antenna installed on a leased tower site.

That being said, corners were forcibly cut.  Instead of installing a microwave STL system, a T-1 was ordered because we had a T-1 multiplexor.  Fast forward several years… The station is now successful, making a decent amount of money and having a popular format.

The station has two T-1 circuits on different cables with an automatic switcher.   Yesterday afternoon, the inevitable happened, both T-1s went out, along with most of the other TELCO circuits in the surrounding area.  A construction crew cut two 3600-pair cables a mile down the road.  The TELCO is racing to restore the service to all of the tenants on that tower by rigging a temporary aerial cable.

TELCO trucks, courtesy of <a href=Now the mad scramble ensues with conflicting requirements from the wacky program director.  Screw it, I grabbed one of the AudioVault express machines and took it to the transmitter site.  They are back on the air with a radio station in a box playing music until the T-1 gets fixed.

This site has had numerous problems since we have owned it.  In the 5 years since we launched the format, there have been six T-1 outages longer than 24 hours.  For backup, we have tried an ISDN line, a 3G wireless card in a computer, and a second T-1 circuit.  None of these have proved reliable as most circuit outages involved a cable cut, and multiple circuits were affected.

The real solution is a microwave STL, either a conventional 950 mHz system or a 2.4 or 5.8 gHz last mile system.  Either would work better than what we have now.  Station ownership, they don’t want to hear it.

Update: This took until Friday, September 4th to repair, for a total outage of 9 days, 2 hours, and 26 minutes.  During that time, the station remained on the air with the AudioVault server at the transmitter site and the program director updating it twice per day with voice tracks and commercials.

Lightning strikes

For about 4 years, I lived in a house next to a four hundred-foot radio tower.  Although I never actually saw lightning strike the tower, I heard it several times.  Like everything else, after a while, you get used to it.

This is a video of lightning hitting the WSIX STL tower in Nashville, TN. The camera work is a little unsteady, the strike occurs around the 1:36 mark:

Yep, that is what it is like.

American Airlines Flight 723

File under: Why we check the tower lights every day (or have an automated tower light reporting system):

56 years ago, on September 16, 1953 American Airlines flight 723 flew between the center and northeast towers of the WPTR antenna system while attempting to land at Albany County Airport.  The plane crashed about 3/10 mile away near NY route 5 (Central Avenue) killing all 28 persons on board.  To date, this is the worst aviation accident in the Albany, NY area.

Several years ago while cleaning out different AM transmitter site, I found a bunch of files about this accident in the trash bin.  It seems that some engineer had moved a file cabinet during the great consolidation of the 1990s to the wrong transmitter site.  In any case, I rescued the file and for your reading pleasure, have scanned the following documents:

Original telegram to FCC in Washington DC regarding tower/aircraft colision
Original telegram to FCC in Washington DC regarding tower/aircraft colision

Retel WPTR north and center towers struck three hundred foot level 0930 morning 16 September by American Airlines Convair.  Damage inspected afternoon 16 September by representative Zane Construction and afternoon 17 September by engineer Ideco tower used in array.  Both report slight damage to center tower requiring straightening above 300 foot level.  North tower has two legs bent result of wing passing through tower 18 inches above top guys at bolt intersection.  Tower above bent and twisted but all right unless subject to high east wind.  Impact sheered north beacon clevises and shattered glass.  Replacement ordered and beacon restoration expected early next week.  Measurements of directional pattern afternoon 16 September show pattern unchanged and nulls, base currents and loop currents within tolerances except center loop current which reads ten percent low.  Indications are center pickup loop has been jarred and pattern unaffected.  FCC representative Turnbull who arrived noon Wednesday concurred.  No time lost.   Operating at full power

George Wetmore, Assistant Genl Mgr, Radio Station WPTR

This is the statement of the transmitter engineer on duty at the time of the crash.

Statement from the Engineer on duty at the transmitter site
Statement from the Engineer on duty at the transmitter site

I, Robert S. Henry engineer for Patroon Broadcasting Co. would like to make this statement concerning the collision of an American Airlines Plane with our transmitting towers.

On Wednesday morning September 16, 1953 I was on duty alone at the WPTR transmitter when at approximately 9:30 A.M. a low flying plane was heard overhead. The carrier trip circuit which protects the transmitting apparatus from sudden overloads at the antenna almost instantly actuated and a low sound of explosion followed about 3 seconds later. I ran to the back door of the transmitter in time to see what proved later to be aluminum sheets fluttering down through the fog. I estimate the ceiling at the time to have been approximately 75 feet and not sharply defined. The centre (sic) tower of our three tower array swayed violently for approximately for approximately (sic) 1 minute. At the time I knew a plane had hit the tower but it was above the ceiling and invisable to me. I called W. R. David and George Wetmore and the incident was reported to the Albany airport.

Back in these days, the studios were located at the Hendrick Hudson Hotel in Troy, the transmitter site was manned by a licensed transmitter engineer whenever the station was on the air.

What is amazing is that the IDECO towers remained standing after being struck by Convair 240, a pretty good-sized aircraft.

Convair CV-240
Convair CV-240

An official report by the International Civil Aviation Organization:

American Airlines’ Flight 723 was a scheduled flight between Boston, and Chicago, with intermediate stops among which were Hartford (BDL), and Albany (ALB). The CV-240 arrived at Bradley Field at 06:57. Weather at the next stop, Albany, at this time was below the company’s landing minimums, but was forecast to improve to within limits by the time the flight arrived there. Departure from Bradley Field was made at 07:14. Because of poor visibility at Albany, several aircraft were in a holding pattern. The special Albany weather report issued at 07:50 indicated thin obscurement, ceiling estimated 4,000, overcast, fog, visibility 3/4 miles. Two aircraft left the holding pattern, attempted to land, but both executed a missed approach procedure. A third airplane landed at 08:16 following an instrument approach to runway 19. Immediately following this landing, Flight 723 was cleared to make an instrument approach to runway 19. Three minutes later the flight advised the tower that its approach was being abandoned because the aircraft’s flaps could not be lowered.
At 08:30 Albany Tower reported:”All aircraft holding Albany. It now appears to be pretty good for a contact approach from the west. It looks much better than to the north.” Flight 723 was then cleared for a contact approach to runway 10. On finals for runway 10, the Convair descended too low. The right wing of the aircraft struck the center tower of three radio towers at a point 308 feet above the ground. The left wing then struck the east tower. Seven feet of the outer panel of the right wing including the right aileron and control mechanism from the center hinge outboard together with 15 feet of the left outer wing panel and aileron separated from the aircraft at this time. Following the collision with the towers, ground impact occurred a distance of 1,590 feet beyond the tower last struck. First ground contact was made simultaneously by the nose and the left wing with the aircraft partially inverted.
The weather reported at the time of the accident was thin scattered clouds at, 500 feet, ceiling estimated 4500 feet, broken clouds, visibility 1-1/2 miles, fog.

PROBABLE CAUSE: “During the execution of a contact approach, and while manoeuvring for alignment with the runway to be used, descent was made to an altitude below obstructions partially obscured by fog in a local area of restricted visibility.”

The above reports notes that the aircraft traveled 1590 feet and struck the ground partially inverted.  I do not know what the flaps-up landing speed of a Convair CV-240 is but the cruising speed is 280 MPH.  It would be safe to say the aircraft was traveling in the 120 to 130 MPH range or about 220 feet per second.  At that speed, it was likely airborne for about 7 seconds after it hit the tower.  Enough time to look out the window, realize what was happening, and say “Oh, Shit!”

IDECO towers WDCD antenna system
IDECO towers WDCD antenna system, the northeast tower is farthest

IDECO stood for the Internation Derrick Company, they build cranes, derricks, and bridges as well as radio towers. Apparently, they made pretty good stuff because those same towers are still standing today.

Not that checking the tower lights would have averted disaster in this case, it appears to be pilot error compounded by bad weather that caused this incident.  But there have been more recent aircraft/radio tower accidents, some of which have involved possible faulty tower lights.  I wouldn’t want that on my conscience.

Module swap guys

Nautel V-10 Power Supply
Nautel V-10 Power Supply

Gone is the day when the radio station engineer had to troubleshoot down to the component level, often crawling in and out of transmitters to get at the suspected part.  I for one, spent many a long night at a transmitter site chasing some weird combination of symptoms down to the $0.34 1N914 diode in the directional coupler (see previous post about the MW-50).

It is a skill set now mostly confined to manufacturers’ repair departments, for which they charge a pretty penny.  Nowadays, the technician simply slides out one module or circuit card and slides in another.  If that doesn’t fix it, panic ensues.  I know of several class C FM radio stations that are now relying on the computer guy to fix transmitters, because, you know, it’s cheaper.

To be fair, most engineers are contractors and many of those simply do not have the time to troubleshoot to the component level.  So, they ship everything back to the factory and then pass the cost on to their client.

Then of course, most circuit boards these days are surface mount systems, which are hard to work on if you don’t have the right tools.  Normally an expensive temperature-controlled soldering station is required, as well as a magnifying glass.

All of these things combine to make circuit board work something to be outsourced.  Unfortunately, a night spent troubleshooting was often a great learning experience.  I have done some of my best work when my back was up against a wall and I was out of options.

I make the attempt to fix things locally unless the transmitter or other item is under warranty or not having a spare/attempting to troubleshoot will take the station off the air.  I think it is important to keep abreast of technology and keep my troubleshooting skills up to par.  Besides, I find it gratifying that at least I can still fix things.