disaster recovery – Engineering Radio

In this time of COVID-19

As most of you may know, I live in New York State (Hudson Valley region). This is very close to the COVID-19 outbreak in Westchester County. There has been one confirmed case in my town. As such, we are experiencing the outbreak ahead of the curve from the rest of the country. School has been canceled for at least two weeks and perhaps indefinitely. All public gathering places are closed; restaurants, bars, movie theaters, malls, churches, etc. As a radio engineer, the COVID-19 virus has several implications:

Things are still going to break and will need attention. The good news is that most transmitter sites are unmanned. The only social interaction may be during the travel phase (getting fuel, food, etc).
Many studios and offices are being abandoned as well. Over the last few days, we have set up DJs to operate from their houses. Most sales and office staff have been told to work from home.
Broadcasters have been designated critical communications infrastructure, and The Department of Homeland Security has issued letters that allow travel and procurement of fuel during the national emergency for critical personnel.
I managed, prior to the store shelves being emptied this weekend, to procure some PPE. I don’t know how effective it will be, but anything is better than nothing.
Since Hurricane Sandy, I have had in place many emergency supplies and equipment needed to restore service in the event of a long-term interruption of basic services.

There are many long-term economic implications. For commercial radio stations, the loss of income is going to be extreme. As the virus has spread, businesses have canceled pretty much all advertising. During past disasters, radio was often the only means of getting information to the general population. I am not sure if this is still the case. How relevant is radio these days?

The roof is on FIRE!

We don’t need no water, let the… oh, wait… The actual roof is actually on fire you say?

YES: Ahh! Time to run around like crazy people!

This happened over the weekend at one of our clients in NY. The back story is this; over the last two weeks, the area has received almost three feet of snow. This roof is pitched slightly toward the back of the building. The roofing material is some type of PVC, which is very slippery when wet. Thus, at some point the snow/ice pack shifted towards the back of the building, and it broke the natural gas pipe off where it entered the unit:

The next time the HVAC unit cycled on, there was a giant torch on the roof with flames reportedly eight feet high. A local firefighter just happened to be driving down the road and spotted the fire, thus likely saving the building from major damage. The fire department came and cut off the gas and electricity. The building was evacuated for about 20 minutes while they overhauled and checked for internal fires.

A second unit suffered the same fate, only with less damage:

The fire in this unit was contained in the controller area. Same situation with the gas pipe, only it looks like the pipe was not broken all the way off:

The other two units are shut off while the gas pipes are dug out of the snowpack and checked for damage. At some point, they will be turned back on so that the heat can be restored to the second-floor sales bullpen. Meanwhile, the salespeople; are complaining.

We threw a tarp over the unit with the cover ripped off because more snow is on the way:

Secure everything, assume nothing, take nothing for granted

The big lesson learned from Sandy is to take nothing for granted. For several days prior to the storm’s arrival, we checked everything; refueled and started every generator, checked the oil, water, and battery electrolyte, set up fuel deliveries ahead of time for the worst-case scenario, checked all the backup transmitters and STL’s, and so on. The one thing that I didn’t consider was a storm surge so high that the propane tank would float away. After all, those tanks are heavy.

However, a brief examination of elementary physics reveals that even when full, a propane tank will float:

One gallon of water weighs 8.3 pounds if it is freshwater or approximately 8.55 pounds for saltwater (depending on where it is from).

One gallon of liquid propane weighs approximately 4.1 pounds, thus it is about half as dense as water.

Most propane tanks are not full, being at most 80% liquid volume. It is always the thing that you didn’t think of.

We seem to be suffering a 500-year storm about once a year or so around these parts. I expect that things will only get worse. With that in mind, it is perhaps time to re-think our disaster preparedness and recovery plans to incorporate every worst-case scenario we can imagine. Everyone knows, but it bears repeating: Radio is the last link that people have when all other technology fails. Thus, when it comes to storm preparation, there is no such thing as too much. Thus:

Secure everything
Assume nothing
Take nothing for granted

Our assumptions about power utility and telephone network reliability and restoration may be wrong. Our assumptions about access to remote sites, our ability to use vehicles, availability of gasoline and other fuels may be over-optimistic. Our assumptions that basic foodstuffs, clean water, and secure resting areas may also be wrong. Get those items wrong and it does not matter how much equipment redundancy is built into facilities.

For remote transmitter sites, access can be a major problem after a storm. In low-lying coastal areas, flooding will be an issue. In those situations, having backup transmitter sites would be a key feature of any disaster plan.

All good disaster plans also have the human component; clean water, food, and safe, secure resting areas for the staff. As always, when the SHTF and there are no options and no ideas, there is the Bear Grylls survival method:

Which we really, really don’t want to do (from the TV show Man vs. Wild on Discovery).

A day in Pictures

The aftermath:

Long beach — Long Beach, used to be an isthmus, now it is a sand bar

A set of old stairs on the beach where the cottages used to be located.

100 lb propane tanks — Found the reason why the generator is not running

Propane tanks adrift from storm surge. There was a strong propane smell around these tanks, I secured all the valves.

WICC propane tank pad — Where the propane tanks should be

debris washed ashore during storm surge — Debris washed ashore during storm surge around the north tower, including a section of dock

Second high tide after Hurricane Sandy, noon on Tuesday, flooding the ground system

three phase power line down — Three phase power line down due to wind

Telco wires taken down by trees — Telco wires were taken down by trees

Generator room water level, as seen on the side of the battery

More work here tomorrow.

Update: Took longer than anticipated, but the station is back on the air with generator power as of 8:15 am, Thursday (11/1). Commercial power restoration is not expected until Monday or Tuesday at the earliest.

Update: Commercial power restored on Thursday, 11/8 for a total outage of 10 days. One good thing about incidents like this, I now have a fresh set of contacts for all the important people connected to servicing this site.