You may or may not have heard, but there is a bash exploit that was recently uncovered called Shellshock, and it is being exploited right now, which is why we patched relatively quickly and without any customer communication.
- Shellshock Bash bug exploitation in full swing, warn researchers
- Cisco, Oracle find dozens of their products affected by Shellshock
We patched quickly, but we did test the patch in our dev environment. And actually, we patched several different servers yesterday afternoon, including the one that failed today. None exhibited any problems immediately after applying the patch.
So what happened?
It appears that there was a problem with a database driver in FreeBSD (which runs all of our DNS servers) related to the patch. Just after noon today we started to see “garbage” records in ns1, and when we looked further we saw that every record in ns1 was corrupt. The name server connects to our database to pull down the current records, and the bug in the database driver caused all of the zones to be corrupt.
We had a pretty good idea that’s what had happened, but since we weren’t completely sure at that point, the decision was made to rebuild ns1 from scratch. One of the benefits of doing that is we still have the old ns1 so we can do some forensic work on it to determine exactly what happened.
Why do these problems keep happening?
The timing of these things is never good, but it’s worse when they happen in succession, as this did right after the DDoS on September 22nd (and the email data corruption on a few servers a couple of weeks before that). The incidents are not related, but any one of them on their own would have been bad enough, we understand that.
We take a lot of preventative measures that you never see, because…well, they prevent problems from happening. But we cannot prevent every conceivable problem or dodge every bullet. Once in a while we’re going to get hit by something. You asked us to keep you informed via Twitter, Facebook and Google+, and I think we’re doing pretty well there. Keep in mind that the people posting there (including myself) are not system administrators, so we’re giving you as much information as we can get, but we may not have full details while a problem is happening.
Speaking of Twitter, Facebook and Google+…
The same few questions seemed to be asked by a lot of people, and I can’t answer them all individually, so let me address them here.
“Why are you doing this in the middle of the day?” A few reasons: first, the potential exploit is so great that we didn’t want to wait for a scheduled maintenance period. Secondly, it’s always the middle of someone’s day. Half of our users are outside of the U.S. So whenever we do something, it’s going to be bad timing for a good number of people. Finally, we did test the patch before deploying it, so we didn’t anticipate any issues.
“Why don’t you just roll back the patch?” The patch was applied almost 24 hours before we saw the corruption, so it wasn’t completely clear that the problem was caused by the patch. System administrators determined that they could rebuild ns1 in less time than they might spend troubleshooting, so that’s what they did. We can second guess that decision, but there’s no way to know how long it would have taken to “fix” the old ns1.
“Why would you install an untested patch on a production server?” As I mentioned previously, we tested the patch in our dev environment and saw no problems with it. And it’s worth remembering that the patch worked on every other server we installed it on without issue. We’ll be doing more tests on the old ns1 to see if we can find out why we had failure there but nowhere else.
– – –
When there’s an outage that affects a lot of you, we certainly understand that it’s bad news. We don’t take any kind of interruption for any number of users lightly, and there’s a lot of activity (and some shouting) in the halls here during those times. We never want to see anything fail, and when something does, everyone on our system administration team is lending their particular expertise and everyone is working together as quickly as they can on a fix.
I know I can speak for everyone here when I say we appreciate you hanging in there with us during times when you’d probably rather be throwing rotten tomatoes at us. Hey, I get it. I’ve wanted to throw my share at any number of companies. We really do appreciate your continued loyalty and understanding.
We’ve been talking about Everleap a lot lately, and understandably we’ve had a few of you ask questions about how it might affect DiscountASP.NET. The short answer is, it won’t.
Everleap is modern cloud hosting and DiscountASP.NET is traditional shared hosting. While the end result of both is your site on the web, the route to get there is quite different. There are advantages to both methods, of course, and if you prefer the traditional DiscountASP.NET service, it is always going to be here for you.
Technological advances happen so quickly these days that sometimes you can find yourself thinking, “Whoa, slow down, everything is working fine, let’s not touch it right now.” We get that. We know everyone isn’t going to flock to Everleap. If it ain’t broke…
But don’t worry, DiscountASP.NET will not be frozen in amber like an apartment building on Fringe. We’ll continue to keep everything up to date, provide the best support in the business and invest in infrastructure. That’s the way we’ve always approached the service and that isn’t going to change.
The landscape will definitely be changing more quickly though over at Everleap. Building the service on top of the Windows Azure Pack ensures that we’ll always have the latest modern cloud technology, and we’ve expanded the service significantly from the out-of-the-box WAP offering, so we’re always busy building something cool to enhance the fundamental cloud server hosting.
If that kind of thing gets you out of bed in the morning, by all means, check out Everleap! It really is the next generation of website hosting, and where all website hosting is likely headed.
But if you love DiscountASP.NET (like we do!) you can rest assured that isn’t going anywhere. It still gets the same attention we’ve always given it – and will keep giving it – for as long as you want to use it. Nothing’s changed there. We were one the very first specialized .NET hosts, and as you’ve made very clear over the past decade, the best .NET host!
And if I may be so bold, we always will be.
If you’re a DiscountASP.NET customer, you’ve probably already heard about this, but for the rest of you, we’re really excited to announce something new: Everleap.
It’s cloud website hosting! Okay, I know what you’re thinking: “Hey man, there’s nothing new about cloud hosting!” Well, that’s not exactly true. There is something new about true cloud hosting. Take a look at how Everleap works.
Think about it, most cloud hosting that you see is sort of a hybrid not-really-cloud-at-all kind of thing that isn’t very far removed from traditional shared hosting. They call it “cloud,” but by their definition we could probably call DiscountASP.NET “cloud,” and as you know, it really isn’t.
Then there are the real cloud services; Azure, Amazon, and the handful of others that aren’t Azure or Amazon. If you have used one of those big cloud services you know that virtually everything that you might consider necessary to run your web site is offered as a separate service, metered and billed separately. And forget about support. If personalized support is available, it likely comes at a hefty additional cost.
With Everleap we set out to provide all the technical benefits of the big cloud providers along with the all-inclusive bundled services of traditional web hosting. So every Everleap site includes things like SQL, MySQL, SQL Reporting Services, SSL support, email and DNS service, and our excellent Technical Support that you know and love from DiscountASP.NET is included. Things you will most definitely pay extra for at the big cloud providers.
When I say Everleap is something new, it really is new. It’s the first hosting service built on Microsoft’s Windows Azure Pack. In the coming months you’re going to see other hosts coming out with Azure Pack offerings, but those will be generic, out-of-the-box plans. It isn’t possible for them to do what we’re doing at Everleap because they don’t have the experience that we do.
We have been up to our elbows in the technology underlying Azure Pack (Antares) for almost two years. We have built a world-class infrastructure to support the load balancing and flexibility available with Azure Pack, but not only that, we’ve also built our own Control Panel that allows us to quickly make adjustments and add features, something those generic guys aren’t going to ever be able to do.
Everleap is a premium service, something many of you have been asking us us to build for a long time. Well, here it is. And if I don’t say so myself, it’s really cool.
We get a lot of treats here at the office. Whenever anyone goes on vacation outside the country (or even inside the country) they often bring back some sort of edible tidbit or another for everyone to devour.
Recently Dmitri brought in some Маcсандра Crimean dessert wine, which ranks pretty highly in terms of treats. For some of us, anyway.
In fact, he was kind enough to update the wine bottle with a new location…
You don’t want to miss the demo and all the other great info laid out for you by Mr. Papa and our own Michael Ossou in this jam-packed hour.
What are you waiting for?