We think of cloud services as a creation of the modern digital world, but one of the first cloud services was installed almost 140 years ago, on Pearl Street in Manhattan, just south of Fulton Street.
On September 4, 1882, the Pearl Street Station began providing light for 400 lamps and 82 customers. Like many cloud services, Pearl Street Station grew. Within two years, it was providing power for 500 customers, powering more than 10,000 lamps.
Also: Top cloud providers in 2021: AWS, Microsoft Azure, and Google Cloud, hybrid, SaaS players
Rather than each individual customer building power-generating infrastructure, they all relied on this one centralized service. This service was the Edison Illuminating Company, which would eventually become Con Ed (Consolidated Edison), the $12B public utility that today provides power to most of New York City and Westchester County.
A quick update
This article was originally written in 2016 and was centered around discussion of Hurricane Matthew. In 2017, my wife and I left Florida permanently when our home was once again at ground zero, this time for Hurricane Irma. That storm left our home without power for more than a week.
We now live in Oregon, which is not without its own natural disasters. Last year, fellow Oregonians suffered terrible losses due to the 2020 wildfires. My family was spared, but many of our neighbors were not. And that’s not even considering the whole 2020 pandemic thing, which we’re all depressingly familiar with.
Just this weekend, we lost power for what seemed like a very long time. Oregon had a “once in a generation” ice storm, dumping 1.25 inches of ice on everything. The normal ice load most infrastructure can handle is about 0.25 inches, and the added weight downed trees, powerlines, and communications everywhere. Oregon public utilities worked miracles, but even with hundreds or thousands of technicians a state like Oregon is pretty large.
Texas is even larger and its big chill made things even worse for Texas residents, many of whom were completely unprepared for wintry weather. Texas, too, experienced a crippling power outage, and more than 4 million Texas residents lost power. By contrast, here in Oregon, barely an eighth of that amount were lighting candles.
Also: Extreme weather forecast? Essential gear for when the power goes out
The rest of this article will take you back to 2016 and is written from the perspective of someone who just lived through a hurricane. But the ideas presented about the power grid apply now, and apply to other natural disasters like wildfires and ice storms.
The bottom line, of course, is we wish you the best. Hang in there.
Shared characteristics
Where I live, in Central Florida, we rely on another regulated public utility, Florida Power and Light (FPL). Both of these utilities share a lot of the characteristics we’re used to in IT cloud services.
We’ve obviously devoted a lot of virtual ink to cloud computing, so I’m not going to rehash all the elements here. But it’s important to realize that both cloud computing providers and public utility providers are the keepers of the physical infrastructure. In the case of cloud computing, that’s servers, storage, and network. In the case of public utility providers, that’s power generation, power storage, and power distribution.
On one hand, that’s great for consumers of these centralized resources. If you want to start your own online application business, you no longer need to build out a physical infrastructure. Back in the late 1990s (before we had real cloud access), I did that. Each time I reached a server’s max capabilities, scaling involved a major investment to jump to the next level. But with cloud, you just scale smoothly, with only incremental expense.
Likewise, with power, I don’t have to build and maintain my own onsite generator, and figure out a way to manage and safeguard fuel deliveries. If I use a bit more, or a bit less, power each month, it’s simply reflected in my bill.
Also: How to protect your IT power from deep-freeze disasters
These services generally work reliably and consistently. I can check my Gmail, for example, without worrying about servers and infrastructure. I can brew a pot of coffee without worrying about whether or not the generator has been topped off with fuel.
We, as a society, have come to rely on cloud and public utility services to such an extent that they actually define our civilization. When these services fail, our lifestyle stutters. For example, when access to Facebook or Gmail goes down, we suddenly feel disconnected from our friends and colleagues. Sometimes, we’re unable to complete work on time, or stay in touch for critical communications.
When the power goes out, everything comes to a halt. There’s no air conditioning, no lights, no food preservation. Nothing.
Blasted back to the Stone Age
In most cases, failures are brief. They last a few hours, at most. But Hurricane Matthew blasted Central Florida back to the Stone Age. It wasn’t pretty.
Those of us who live on the southeast coast of the United States knew Matthew (which was a Category 4 storm, based on the Saffir-Simpson Hurricane Wind Scale) was coming for almost a week. It was due to hit the Space Coast (where I live) on Friday morning.
My wife and I spent Tuesday, Wednesday, and Thursday preparing the house. We disassembled the workshop, and turned it back into a reinforced garage to make room for one car. We moved our second car to the garage at my parents’ old house. We battened down all the hurricane shutters. We filled tubs and gallon jugs with water. We did our best to prepare.
By midnight on Thursday, we’d mostly finished our preparations. Our nerves were on edge. To distract ourselves as the storm approached, we decided to binge-watch the latest season of Game of Thrones. Power dropped out for a few minutes during episode one, and another few minutes during episode two. We kept glancing at an app on my phone to watch the track of the hurricane’s eye.
Read this
Video: IT heroes of Hurricane Sandy
ZDNet interviewed a panel of IT heroes who kept their organizations running During Hurricane Sandy with successful disaster recovery plans. Watch the full 40-minute panel discussion as these IT leaders share the lessons they learned.
Read More
At 4:43am, about 20 minutes into episode three of Game of Thrones, the power went off for the final time. The storm had arrived in full force.
We were terrified, because all the track maps essentially showed the worst of the storm, including 140 mph winds, making landfall pretty much on top of us. Fortunately, that didn’t happen. The storm mostly missed us, with the eye about 20 miles offshore. Even so, we were hit by winds in excess of 70-90 miles per hour. By about 7am, the worst of the storm had passed.
It was still unsafe to go outside, or even open up the internal windows behind the storm shutters. So we had no air flow in the house until about 3pm Friday. We tried to sleep. When the wind finally died down, and the limited LTE I still had on my iPhone showed that the storm had tracked northward, we removed the first of our fortifications (the shutters over our front door). We stepped outside into the fresh air.
We were very fortunate. Our house sustained no damage. A neighbor’s fence had blown down. A home down the street had some roof damage. No one was hurt.
But we had no power. On Friday, with the winds from the tail end of the storm still surging, at least we had a breeze. We thanked goodness that the supernatural heat of the summer was behind us. We opened our windows, and we lit some candles. The cross-breeze helped a little.
Verizon’s LTE was barely functional, so getting information was nearly impossible. We had no idea when power would be restored. My iPhone was down to 50 percent. Normally, I’m very happy with the battery on the iPhone 6s Plus. But with no idea when I’d be able to recharge, I started to get nervous. We had some D-cell batteries for the fans, but, again, we didn’t know how long we’d be without power.
All we could think about was how long we’d have to go without power, and how the hell we’d make it for however long it would be. When we finally regained some level of internet connectivity on the phone, the FPL status update site merely said that they were working hard to restore power. No time estimate was possible.
On Saturday afternoon, power did come back on… for 20 seconds. The lights came on, and I almost teared up with relief. It was short-lived relief. All of a sudden there was a boom. The lights went back off. A nearby transformer had exploded, probably from debris across the connectors. It would be another day before we got power back again.
All told, we were without power for three days. I’m ashamed to say that I didn’t take my forceable removal from civilization well. I was miserable, uncomfortable, desperate, and a little crazy. I couldn’t sleep. I didn’t eat much. There was nothing to do, nothing to work on, and — as a person who usually enjoys the illusion that I’m very much in control of my own destiny — nothing I could do to improve our situation.
We simply had to wait
This is the problem with centralized services like cloud services and public utilities. The convenience, scalability, cost-savings, reduced maintenance, and general reliability come at the cost of self-determination. If those services fail, they take you down with them.
This is why, with cloud computing services, we often talk about redundancy, and keeping local backups. We can also employ a similar strategy with public utilities, although the implementation is much more complex, much more costly, and much less reliable.
I do not own a generator. I regretted that a lot over this very long weekend. However, while there are relatively inexpensive generators available, one that can power A/C for a longer duration is very large and incredibly expensive. Worse, there’s the question of how to safely store the fuel during the storm, and whether the actual generator will survive the pounding of the storm.
A similar concern exists for solar power. It would be great to put up solar cells and not have to pay the monthly power bill at all. But in a hurricane-prone area, solar cells are likely to be torn off the roof before they can provide the emergency power they’re intended for. It’s kind of a Catch-22.
The real answer is that the public utilities, the power companies, need to implement more robust power distribution mechanisms.
Okay, let me stop here for a moment. Before I criticize FPL and its ilk, I want to give a huge shout-out to all the very hardworking repair teams who restored our power over the weekend. I spoke to some of the guys working the lines, and they told me they’d been flown in from out of state before the storm. They worked their way up the state, restoring power county-by-county, city-by-city. They had had almost no sleep for days, while having to work with live power lines in 90-degree heat. They’re champions and heroes.
Demand a better solution
That said, this is not how it should work. All our power lines (and broadband lines, for that matter) are exposed and hanging. This is unconscionable. The power services know that we’re prone to hurricanes, yet they allow these lines to remain open and exposed.
Image: David Gewirtz
Worse, they’re often poorly maintained during non-emergency times. The picture you see to the right is the transformer behind my house. Notice all the overgrowth? If a branch crosses over the connectors, that transformer will either spark or explode. It’s already exploded once. And yet, that’s how FPL distributes power in an area prone to wind storms.
Can you imagine such irresponsibility among cloud computing providers? It’s as if, knowing what they do about the prevalence of hackers and infiltrators, Google just didn’t bother using firewalls, intrusion prevention, or even password security. It’s as if Google’s entire cybersecurity strategy was “eh, call us when you’re hacked, and we’ll fix it when we get to it.”
No one would tolerate such a thing. But that’s because Google has competition, which keeps it agile and competitive. We’re stuck with our single power provider, FPL, who has no competition. As such, they can choose to prioritize repairs using a system that’s essentially waiting to see what breaks, rather than building in any preventive infrastructure.
There is no way, yet, to prevent these terrible storms. But the damage due to the storm is often the result of a failure in infrastructure planning, maintenance, or investment, not due to acts of Mother Nature. Katrina was a bad storm, to be sure. But it was the failure to maintain the levees protecting New Orleans that was the cause of most of the damage.
Here in Brevard County, we have a little over 300,000 power customers. Friday night, more than 200,000 of them were without power. By Saturday night, 100,000 were still without power. And even today, after Friday, Saturday, Sunday, and now Monday, some of our friends are still waiting to have their power restored.
It’s not that FPL didn’t have repair escalation plans in place, or dedicated workers. They did. Those folks are fantastic. They emergency response was well executed. But infrastructure as poorly maintained as the transformer in my back yard does not show an ongoing dedication to emergency prevention and disaster mitigation.
We allow our public utilities to be monopolies because of the enormous investments required to deliver service to all customers. We regulate them because they’re monopolies. But we don’t do enough to demand that they harden and protect their infrastructure — and that’s because they don’t have any competition.
If FPL had competition, the way Google has to compete against Microsoft and AWS, you can be sure we’d not only have spent this weekend in cool comfort, we’d probably spend a lot less each month on the services we do get.
Perhaps as companies like Tesla (and even Apple) develop more robust battery technology, we can replace generators and solar cells with highly efficient in-structure batteries. Then, maybe we’ll be able to withstand five days without power from the grid, simply by tapping into our own private pool of battery power.
Or, perhaps, similar to the way the internet has disrupted other forms of infrastructure, we’ll start to see new and innovative ways we can produce our own energy. Perhaps we will be able to replace the service we get from the grid or, at the very least, have an alternate source available for when storms like Matthew hit an entire region.
You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV. More
