You may have noticed that there were a couple of outages last week, related to moving all of the hardware in our network to a new, larger space in the Los Angeles data center (and if you didn’t notice, forget I said anything).
A large-scale move like that is a major, unusual occurrence, and probably (hopefully!) a one-time thing. For a major move like that, you may have expected some brief periods of downtime related to it.
But we’ve been successfully preventing and avoiding other types of global outages for some time now (not to jinx anything), though individual server outages are still something that happens from time to time. In fact they’re scheduled to happen every month when we do Windows updates.
I know that a lot of you notice when there is an outage, because I talk to you about it here, on Twitter, Google+, Facebook, in the helpdesk, standing in line at the movie theater, at stop lights on my way in to the office…
I hear what you’re saying, no one likes an outage, and we take them very seriously around here. I know that many of you rely on us to provide service to your clients, and when there’s a problem, they rattle your cage, not ours.
But our goal here is to be honest in all of our communication with you, and honestly, things are going to fail. We’re going to have occasional problems. Some small, some large. And as a result, your web site will not be up 100% of the time.
Yes, I said it.
And while it may be a little strange to see someone from an established website hosting company saying it, it really shouldn’t be too surprising. It’s the reality of the situation. Hardware fails, networks fail and humans fail.
For what it’s worth, I think we do provide nearly perfect service
Nearly. But it can’t work 100% of the time because there are a million variables at work here every minute of every day, and we can only control about 999,000 of them.
I’m speaking for us, here at DiscountASP.NET, but what I’m talking about applies to every service that you use. All of them. On the Internet (it’s difficult to think of a single online service or utility that hasn’t had an outage in the past six months), in your home, in your car, on the train – pretty much everything you rely on to always be there – it’s all going to fail at some point.
Considering the complexity of the network that is the Internet, and the interconnectedness of thousands of different kinds of hardware and software, it’s really kind of a minor miracle that it works as well as it does. But no one – ourselves included – likes an interruption, even in their miracles.
Whenever there is an incident that affects a lot of you (or all of you – like a major DDoS), we spend a lot of time after the fact analyzing what went wrong and determining how we can prevent or better react to something similar in the future. That’s time well spent, because every improvement we make, large or small, has a positive impact on the quality of the nearly perfect service we provide to you.
We also spend a considerable amount of time and money preventing and mitigating problems before you even know they’re happening. Network monitoring, intrusion and exploit detection, hardware and software retirement and migration – it’s an ongoing process, and we constantly tweak and improve all of our processes. And by “constantly,” I mean every single day.
But the major improvement we’ve made isn’t actually on the DiscountASP.NET platform
It’s the introduction of a completely new platform at Everleap.
That’s our cloud hosting system built on Windows Azure Pack. It’s all of the good parts of the big cloud combined with the good parts of traditional hosting. Meaning the resilience, redundancy and flexibility of the big cloud, but with the inclusion of a lot of traditional hosting services that you’ve come to expect, but cost extra at the large cloud hosts: email, databases, SQL Reporting Service, usage stats, SSL, DNS, expert, in-house tech support, etc.
What makes Everleap different is that it is much more fault-tolerant than a traditional server set up like we have here at DiscountASP.NET. If an Everleap server goes down, all of the traffic for those sites is routed to a healthy server within seconds. That technology also allows us to do Windows server updates with no web server downtime. Something that’s impossible on a traditional Windows server.
That alone is pretty cool, but you can also run your site simultaneously on multiple servers that are automatically load balanced, increase resources like memory and CPU much more easily, and even get Reserved Cloud Servers and Managed SQL servers – all the resources of a web or SQL server dedicated to a single user. Reserved Cloud and Managed SQL are like having your own server – no unruly neighbors to drag you down – but without any of the maintenance headaches that come with a VPS or dedicated server.
If all of that sounds like a sales pitch, it is! A little bit. We really believe that Everleap is the future of web hosting, and we want everyone to benefit from the advances. Now I know that some of you are probably wondering, “If you believe in it so much, why didn’t you just replace the DiscountASP.NET platform with the Everleap technology?”
And the answer is, we seriously considered it. But it would have been unnecessarily disruptive for a lot of folks, and we feel that there’s still a place for a traditional hosting platform. So ultimately we decided to offer Everleap on its own so that everyone who enjoys DiscountASP.NET just the way it is, thank you, can remain right where they are. Choice is always good.
But if you’re outgrowing the traditional hosting platform, or you just want the speed, flexibility and greatly improved uptime of the new platform, you may want to give Everleap a try. It’s free for 30 days, so you’ve got nothing to lose. If you like it, we’ll even help you move and apply any credit remaining on your DiscountASP.NET account to your new Everleap account.
The future is now! Come on over and see it for yourself.
Note: beginning with Chrome version 46 the yellow caution triangle has been removed from the https URL when Chrome encounters minor errors such as those described in this article.
If you use an SSL certificate (https) on your site, you may have seen a couple of new things happening in Google Chrome version 41 or later. Various warning messages such as, “The identity of this website has not been verified,” “Your connection to <domain> is not encrypted,” or other visual indications that the https connection is not secure have started to be displayed.
Those appear when your SSL certificate uses a SHA-1 signature (most SSL certificates issued before 2015 use SHA-1).
To fix the problem of browser security warnings you must re-key your SSL certificate for SHA-2. If you don’t see those warnings in Chrome and you purchased your certificate recently, it may already be SHA-2. You can verify using this test site.
If you purchased your SSL certificate from us, here’s how to re-key:
1) Contact us and we will re-generate and re-submit the CSR.
2) You’ll then get an email from GeoTrust with a link to complete the process. When completing the re-key on the GeoTrust site, be sure that SHA-2 is selected as the “Hashtag Algorithm.” You can find step-by-step instructions (and a video) here.
3) After you’ve completed the reissuing process, you’ll receive an email with the new certificate. Go to Control Panel and paste the new certificate into the SSL Manager.
If you purchased your SSL certificate elsewhere:
1) Contact us and we will re-generate the CSR and email it to you. Then you’ll have to contact the issuer of your certificate to get your certificate re-keyed for SHA-2.
2) When you receive the re-keyed certificate, go to Control Panel and paste the new certificate into the SSL Manager.
“Obsolete cryptography” message after re-keying with SHA-2
There is another potential problem after you’ve re-keyed your SSL certificate. While the address bar will show the green lock icon, if visitors look at the certificate details in Chrome, they may see an “Obsolete Cryptography” message.
What’s happening is the Chrome Browser is ignoring the cipher preference we use on the server (which includes their preferred ciphers) and pointing out any “weak ciphers” they find. You might notice that many large corporate sites are also insecure according to Chrome, for similar reasons:
That “obsolete cryptography” message may persist for a while because Google is not providing any information on exactly what they want from the server to stop calling it insecure. It would appear that Google would like to see every server everywhere remove support for all older cryptographic methods.
We understand the reasoning behind that, but the problem with removing some of those methods is doing so will shut out visitors using some older browsers and operating systems that don’t support newer methods (such as Windows XP). Since our servers are shared by many customers, it isn’t really an option for us to make global changes that prevent some visitors – even a small number – from accessing our customer’s sites.
We do maintain special servers that do not support any of the older cryptography methods, and they are available if you’d like to move your site. The servers are primarily used by customers who need a “hardened” server to pass a PCI compliance scan. But the added security does introduce some issues, such as older browsers being unable to connect to sites on those servers via https. There are also a few other caveats that may require adjustment or a work-around on your part. But if you’d like to move your site to such a server, or need more information, let us know.
We continue to monitor information from Google on recommended server configuration, as well as testing various configurations ourselves to prevent the “obsolete cryptography” message.
If you have any trouble re-keying a certificate, or if you have any questions about these ongoing changes, let us know and we’ll do our best to help.
DDoS stands for Distributed Denial of Service. When someone launches a DDoS attack, hundreds (or thousands) of computers and servers around the world simultaneously send traffic to a web server – or most often, a specific site on a server – in an attempt to take the site down by overwhelming the server.
When a site on our network is the target of a DDoS the effect on your site can range from none, to slowing it down, to making it completely unavailable. The reason for that is DDoS attacks vary in method and severity, and many of them are counteracted before anyone even notices a problem. Others are more intense or sustained or difficult to counteract, and everyone notices those because they can potentially cripple the network.
Why does an attack on a single site affect the entire network?
A sufficiently large attack on a single site can send enough traffic to the network to overwhelm the routers that live at the entrance to our network. The largest measured DDoS at the time I’m writing this was over 400 gigabits per second – that’s 400 billion bits of data. Per second.
To put that in perspective, some of the most massive and expensive network switches available can handle 100 Gbps, and most common switches are built to handle only 1 or 2 Gbps of traffic. That may sound small compared to a 100 Gbps switch, but it’s more than sufficient for most networks. We host tens of thousands of sites, and our average network traffic is around half a gigabit.
So you can see why an attack large enough to overwhelm the switches can affect every site on the network, including the main DiscountASP.NET site, email, Control Panel, helpdesk, etc.
The method for dealing with large attacks is essentially the same as dealing with smaller ones, but the overall impact is naturally worse, since everyone is affected. Attacks on a scale large enough to effect the entire network are still uncommon, but becoming more of a threat every day, for reasons I’ll spell out in a minute.
What does DiscountASP.NET do to counteract a DDoS?
The methods we use to counteract DDoS attacks are varied and have included just about every method available: DDoS mitigation services, intrusion detection devices, null routing, etc. There are a lot of methods out there, but often the most effective thing we can do is be reactive and responsive. Our network is continuously monitored for malicious traffic, and we have direct control over null routing on all of our backbone connections.
When a DDoS targets a specific site, they are relatively easy to counteract. Though more often than not these days, DDoS do not directly target a domain or an IP, so it takes a bit of time to determine the target (and determining the target is necessary to counteract the attack).
In the past we could just throw massive amounts of bandwidth at an attack to absorb the traffic and mitigate the attack’s effect. But that approach has become much less effective as of late. The botnets have become too large, and a rapidly increasing number of the compromised computers are on broadband connections in homes or corporate servers in large data centers.
While there still isn’t any way to prevent a DDoS before it happens, be assured that we react to every incident of possible malicious traffic immediately and respond with whatever methods are likely to be most effective as quickly as possible.
Why do DDoS attacks happen?
There are a lot of reasons, ranging from political protest to personal grudge and a million other reasons in between. Humans launch these attacks and or course humans can be unpredictable and irrational. When we determine the target of DDoS attacks there is often no outward reason why the site would be attacked. So the reason isn’t always obvious.
The problem – and the reality – is that no matter what we do, inevitably some DDoS attacks are going to have an effect on the network, and possibly your site. It isn’t just us, it’s every site and host everywhere, including the biggest sites on the Internet. Unfortunately, if they can take down Microsoft or Amazon, they can take down DiscountASP.NET. It’s something we are all coming to grips with and trying to learn to prevent.
Where to go for information in the event of a large DDoS attack
If you suspect that a large scale attack is happening, you can check our Twitter, Google+ and Facebook pages for updates and information. We will also be moving our community forum to a server outside of our network sometime soon in order to keep that communication channel open in the event of a large attack.
If a DDoS affects your site you can be sure that we are doing all we can to stop it and return the network to its maximum capacity.
You may or may not have heard, but there is a bash exploit that was recently uncovered called Shellshock, and it is being exploited right now, which is why we patched relatively quickly and without any customer communication.
- Shellshock Bash bug exploitation in full swing, warn researchers
- Cisco, Oracle find dozens of their products affected by Shellshock
We patched quickly, but we did test the patch in our dev environment. And actually, we patched several different servers yesterday afternoon, including the one that failed today. None exhibited any problems immediately after applying the patch.
So what happened?
It appears that there was a problem with a database driver in FreeBSD (which runs all of our DNS servers) related to the patch. Just after noon today we started to see “garbage” records in ns1, and when we looked further we saw that every record in ns1 was corrupt. The name server connects to our database to pull down the current records, and the bug in the database driver caused all of the zones to be corrupt.
We had a pretty good idea that’s what had happened, but since we weren’t completely sure at that point, the decision was made to rebuild ns1 from scratch. One of the benefits of doing that is we still have the old ns1 so we can do some forensic work on it to determine exactly what happened.
Why do these problems keep happening?
The timing of these things is never good, but it’s worse when they happen in succession, as this did right after the DDoS on September 22nd (and the email data corruption on a few servers a couple of weeks before that). The incidents are not related, but any one of them on their own would have been bad enough, we understand that.
We take a lot of preventative measures that you never see, because…well, they prevent problems from happening. But we cannot prevent every conceivable problem or dodge every bullet. Once in a while we’re going to get hit by something. You asked us to keep you informed via Twitter, Facebook and Google+, and I think we’re doing pretty well there. Keep in mind that the people posting there (including myself) are not system administrators, so we’re giving you as much information as we can get, but we may not have full details while a problem is happening.
Speaking of Twitter, Facebook and Google+…
The same few questions seemed to be asked by a lot of people, and I can’t answer them all individually, so let me address them here.
“Why are you doing this in the middle of the day?” A few reasons: first, the potential exploit is so great that we didn’t want to wait for a scheduled maintenance period. Secondly, it’s always the middle of someone’s day. Half of our users are outside of the U.S. So whenever we do something, it’s going to be bad timing for a good number of people. Finally, we did test the patch before deploying it, so we didn’t anticipate any issues.
“Why don’t you just roll back the patch?” The patch was applied almost 24 hours before we saw the corruption, so it wasn’t completely clear that the problem was caused by the patch. System administrators determined that they could rebuild ns1 in less time than they might spend troubleshooting, so that’s what they did. We can second guess that decision, but there’s no way to know how long it would have taken to “fix” the old ns1.
“Why would you install an untested patch on a production server?” As I mentioned previously, we tested the patch in our dev environment and saw no problems with it. And it’s worth remembering that the patch worked on every other server we installed it on without issue. We’ll be doing more tests on the old ns1 to see if we can find out why we had failure there but nowhere else.
– – –
When there’s an outage that affects a lot of you, we certainly understand that it’s bad news. We don’t take any kind of interruption for any number of users lightly, and there’s a lot of activity (and some shouting) in the halls here during those times. We never want to see anything fail, and when something does, everyone on our system administration team is lending their particular expertise and everyone is working together as quickly as they can on a fix.
I know I can speak for everyone here when I say we appreciate you hanging in there with us during times when you’d probably rather be throwing rotten tomatoes at us. Hey, I get it. I’ve wanted to throw my share at any number of companies. We really do appreciate your continued loyalty and understanding.
We’ve been talking about Everleap a lot lately, and understandably we’ve had a few of you ask questions about how it might affect DiscountASP.NET. The short answer is, it won’t.
Everleap is modern cloud hosting and DiscountASP.NET is traditional shared hosting. While the end result of both is your site on the web, the route to get there is quite different. There are advantages to both methods, of course, and if you prefer the traditional DiscountASP.NET service, it is always going to be here for you.
Technological advances happen so quickly these days that sometimes you can find yourself thinking, “Whoa, slow down, everything is working fine, let’s not touch it right now.” We get that. We know everyone isn’t going to flock to Everleap. If it ain’t broke…
But don’t worry, DiscountASP.NET will not be frozen in amber like an apartment building on Fringe. We’ll continue to keep everything up to date, provide the best support in the business and invest in infrastructure. That’s the way we’ve always approached the service and that isn’t going to change.
The landscape will definitely be changing more quickly though over at Everleap. Building the service on top of the Windows Azure Pack ensures that we’ll always have the latest modern cloud technology, and we’ve expanded the service significantly from the out-of-the-box WAP offering, so we’re always busy building something cool to enhance the fundamental cloud server hosting.
If that kind of thing gets you out of bed in the morning, by all means, check out Everleap! It really is the next generation of website hosting, and where all website hosting is likely headed.
But if you love DiscountASP.NET (like we do!) you can rest assured that isn’t going anywhere. It still gets the same attention we’ve always given it – and will keep giving it – for as long as you want to use it. Nothing’s changed there. We were one the very first specialized .NET hosts, and as you’ve made very clear over the past decade, the best .NET host!
And if I may be so bold, we always will be.
If you’re a DiscountASP.NET customer, you’ve probably already heard about this, but for the rest of you, we’re really excited to announce something new: Everleap.
It’s cloud website hosting! Okay, I know what you’re thinking: “Hey man, there’s nothing new about cloud hosting!” Well, that’s not exactly true. There is something new about true cloud hosting. Take a look at how Everleap works.
Think about it, most cloud hosting that you see is sort of a hybrid not-really-cloud-at-all kind of thing that isn’t very far removed from traditional shared hosting. They call it “cloud,” but by their definition we could probably call DiscountASP.NET “cloud,” and as you know, it really isn’t.
Then there are the real cloud services; Azure, Amazon, and the handful of others that aren’t Azure or Amazon. If you have used one of those big cloud services you know that virtually everything that you might consider necessary to run your web site is offered as a separate service, metered and billed separately. And forget about support. If personalized support is available, it likely comes at a hefty additional cost.
With Everleap we set out to provide all the technical benefits of the big cloud providers along with the all-inclusive bundled services of traditional web hosting. So every Everleap site includes things like SQL, MySQL, SQL Reporting Services, SSL support, email and DNS service, and our excellent Technical Support that you know and love from DiscountASP.NET is included. Things you will most definitely pay extra for at the big cloud providers.
When I say Everleap is something new, it really is new. It’s the first hosting service built on Microsoft’s Windows Azure Pack. In the coming months you’re going to see other hosts coming out with Azure Pack offerings, but those will be generic, out-of-the-box plans. It isn’t possible for them to do what we’re doing at Everleap because they don’t have the experience that we do.
We have been up to our elbows in the technology underlying Azure Pack (Antares) for almost two years. We have built a world-class infrastructure to support the load balancing and flexibility available with Azure Pack, but not only that, we’ve also built our own Control Panel that allows us to quickly make adjustments and add features, something those generic guys aren’t going to ever be able to do.
Everleap is a premium service, something many of you have been asking us us to build for a long time. Well, here it is. And if I don’t say so myself, it’s really cool.
We get a lot of treats here at the office. Whenever anyone goes on vacation outside the country (or even inside the country) they often bring back some sort of edible tidbit or another for everyone to devour.
Recently Dmitri brought in some Маcсандра Crimean dessert wine, which ranks pretty highly in terms of treats. For some of us, anyway.
In fact, he was kind enough to update the wine bottle with a new location…
You don’t want to miss the demo and all the other great info laid out for you by Mr. Papa and our own Michael Ossou in this jam-packed hour.
What are you waiting for?