You may or may not have heard, but there is a bash exploit that was recently uncovered called Shellshock, and it is being exploited right now, which is why we patched relatively quickly and without any customer communication.
- Shellshock Bash bug exploitation in full swing, warn researchers
- Cisco, Oracle find dozens of their products affected by Shellshock
We patched quickly, but we did test the patch in our dev environment. And actually, we patched several different servers yesterday afternoon, including the one that failed today. None exhibited any problems immediately after applying the patch.
So what happened?
It appears that there was a problem with a database driver in FreeBSD (which runs all of our DNS servers) related to the patch. Just after noon today we started to see “garbage” records in ns1, and when we looked further we saw that every record in ns1 was corrupt. The name server connects to our database to pull down the current records, and the bug in the database driver caused all of the zones to be corrupt.
We had a pretty good idea that’s what had happened, but since we weren’t completely sure at that point, the decision was made to rebuild ns1 from scratch. One of the benefits of doing that is we still have the old ns1 so we can do some forensic work on it to determine exactly what happened.
Why do these problems keep happening?
The timing of these things is never good, but it’s worse when they happen in succession, as this did right after the DDoS on September 22nd (and the email data corruption on a few servers a couple of weeks before that). The incidents are not related, but any one of them on their own would have been bad enough, we understand that.
We take a lot of preventative measures that you never see, because…well, they prevent problems from happening. But we cannot prevent every conceivable problem or dodge every bullet. Once in a while we’re going to get hit by something. You asked us to keep you informed via Twitter, Facebook and Google+, and I think we’re doing pretty well there. Keep in mind that the people posting there (including myself) are not system administrators, so we’re giving you as much information as we can get, but we may not have full details while a problem is happening.
Speaking of Twitter, Facebook and Google+…
The same few questions seemed to be asked by a lot of people, and I can’t answer them all individually, so let me address them here.
“Why are you doing this in the middle of the day?” A few reasons: first, the potential exploit is so great that we didn’t want to wait for a scheduled maintenance period. Secondly, it’s always the middle of someone’s day. Half of our users are outside of the U.S. So whenever we do something, it’s going to be bad timing for a good number of people. Finally, we did test the patch before deploying it, so we didn’t anticipate any issues.
“Why don’t you just roll back the patch?” The patch was applied almost 24 hours before we saw the corruption, so it wasn’t completely clear that the problem was caused by the patch. System administrators determined that they could rebuild ns1 in less time than they might spend troubleshooting, so that’s what they did. We can second guess that decision, but there’s no way to know how long it would have taken to “fix” the old ns1.
“Why would you install an untested patch on a production server?” As I mentioned previously, we tested the patch in our dev environment and saw no problems with it. And it’s worth remembering that the patch worked on every other server we installed it on without issue. We’ll be doing more tests on the old ns1 to see if we can find out why we had failure there but nowhere else.
– – –
When there’s an outage that affects a lot of you, we certainly understand that it’s bad news. We don’t take any kind of interruption for any number of users lightly, and there’s a lot of activity (and some shouting) in the halls here during those times. We never want to see anything fail, and when something does, everyone on our system administration team is lending their particular expertise and everyone is working together as quickly as they can on a fix.
I know I can speak for everyone here when I say we appreciate you hanging in there with us during times when you’d probably rather be throwing rotten tomatoes at us. Hey, I get it. I’ve wanted to throw my share at any number of companies. We really do appreciate your continued loyalty and understanding.
As a major hosting provider, we deal with compromised sites on a daily basis, so we’ve seen just about every site compromise scenario. If your site is compromised you may wonder, “Why me? What is the benefit to the hacker?”
Chances are it’s not your site specifically that’s being targeted, but rather any site that can be compromised. You just happen to fall into that category. In general, hackers compromise websites for one of the following reasons:
- To get access to a well-connected web server to launch an attack on another network.
- To steal sensitive files or data, e.g. a database containing personal information and credit card numbers.
- To use your site to host spyware, malware or phishing pages.
- To use your site to send out spam.
How do they get through?
Based on our experience, hackers typically compromise sites in the following ways.
Through known security holes in your application
For example, if you are using a wordpress plugin that has security issue and you’ve neglected to update it, hackers can seek out your site using search engines like Google and perform an automated bot attack that will compromise your site. Last month over 50,000 WordPress site were hacked through plugin vulnerability. It can happen to anyone.
Weak Password on your third-party application
Every day we see bots coming into our network scanning for well known applications. Once one of those applications is identified, the bot attempts a brute force dictionary attack to crack the administrator password.
Insecure upload form
This is a very common problem we see virtually every day. Many websites have a photo/document upload mechanism for their users. If the upload application is not secure, hackers can easily upload a webshell. Once the webshell is uploaded, the hacker can upload more files to further compromise your site.
Compromised FTP account
If your local PC is compromised, a hacker can easily install a key logger to capture all your traffic, including email and FTP usernames and passwords. Once they have your account credentials, they can upload anything to your site. If you delete the malicious files but aren’t aware that your credentials have been compromised, they will likely upload the files again every time you delete them.
What we are doing to help
We started noticing a rapid increase in the number of compromised sites about a year ago. We also found that most of our customers needed help fixing and securing their sites. That’s not surprising, considering the lengths many hackers will go to in order to cover their tracks. So we have taken a number of steps in order to help alleviate the problem.
Regular scans for known compromises
We scan every web server looking for known exploits, and we will notify you if we find anything.
SiteLock is a third-party company that provides a daily scanning service that can automatically remove malware and alert you to weaknesses.
Site Cleaning Service
As I mentioned, a lot of people receive a notice from us that their site has been compromised and aren’t really sure what their next step should be. We recently began offering a site cleaning service that will remove malware and compromises, try to identify how they happened, and provide a 30 day follow up to make sure you aren’t compromised again. If we identify a compromise on your site we will provide details about the service.
What you can do to avoid being hacked
There are a number of things you can do to secure your web applications.
Keep your applications up to date
We have seen some customers running third-party applications that are several years old and several major versions behind. If your application doesn’t notify you of updates, make it a point to check for updates yourself every few months. This is the easiest, most effective way to keep your site secure. If you use an application that is no longer being developed or updated, find a replacement that is actively developed! It may be a pain to make that change, but it is worth the effort.
Change the default password
There are bots on the Internet that scan for software that is still using the default password, or administrative user name. WordPress, for example, creates the user “Admin” when it is installed. You should change that username, or create a different admin user and delete the default.
Install Anti-virus software on your computer, and keep it up to date
A free antivirus is better than no antivirus. There are a number of decent programs out there that you can use at no cost. Though a paid version of one of the big antivirus programs is usually going to afford more up to date and comprehensive protection.
Configure FTP to allow only your IP address to connect
You can do this in Control Panel with the ISS Tools FTP Manager. Look for the FTP IP RESTRICTION section.
Use complex password for your web applications, FTP and email (actually for everything!)
We recommend at least 8 characters with at least one upper case letter, one digit and one symbol. The longer it takes to crack your password, the more likely it is that a bot will give up and leave for greener pastures.
If you site has any upload functionality, do the following:
1) Your code should block users from uploading executable file extensions like .asp, .aspx, .php, .exe, etc.
2) Execute permissions should be disabled on the folder where you allow users to upload files. To disable execute permissions, create a web.config file in the folder and include the following:
<configuration> <system.webServer> <handlers accessPolicy="Read" /> </system.webServer> </configuration>
Protecting your site from malicious bots and hackers is more important than ever. Times have changed and a “small” site is no longer safe. They are looking for any site, anywhere, and if you don’t make it difficult for the bad guys to get in, they are going to hit you. It’s not a question of if, but when.
Acquia Drupal 7.30.35
DotNetNuke 7.3.1 Community Edition
Gallery Server Pro 3.2.1
SilverStripe CMS 3.1.5
Umbraco CMS 7.1.4
We’ve been talking about Everleap a lot lately, and understandably we’ve had a few of you ask questions about how it might affect DiscountASP.NET. The short answer is, it won’t.
Everleap is modern cloud hosting and DiscountASP.NET is traditional shared hosting. While the end result of both is your site on the web, the route to get there is quite different. There are advantages to both methods, of course, and if you prefer the traditional DiscountASP.NET service, it is always going to be here for you.
Technological advances happen so quickly these days that sometimes you can find yourself thinking, “Whoa, slow down, everything is working fine, let’s not touch it right now.” We get that. We know everyone isn’t going to flock to Everleap. If it ain’t broke…
But don’t worry, DiscountASP.NET will not be frozen in amber like an apartment building on Fringe. We’ll continue to keep everything up to date, provide the best support in the business and invest in infrastructure. That’s the way we’ve always approached the service and that isn’t going to change.
The landscape will definitely be changing more quickly though over at Everleap. Building the service on top of the Windows Azure Pack ensures that we’ll always have the latest modern cloud technology, and we’ve expanded the service significantly from the out-of-the-box WAP offering, so we’re always busy building something cool to enhance the fundamental cloud server hosting.
If that kind of thing gets you out of bed in the morning, by all means, check out Everleap! It really is the next generation of website hosting, and where all website hosting is likely headed.
But if you love DiscountASP.NET (like we do!) you can rest assured that isn’t going anywhere. It still gets the same attention we’ve always given it – and will keep giving it – for as long as you want to use it. Nothing’s changed there. We were one the very first specialized .NET hosts, and as you’ve made very clear over the past decade, the best .NET host!
And if I may be so bold, we always will be.
With all of the excitement over the launch of our new cloud hosting solution over at Everleap, I overlooked announcing that we have successfully renewed our Microsoft Partner Gold Hosting Competency for 2014! We are going into our 9th year of maintaining our Microsoft Gold Partner status.
Even though we’ve been announcing our Microsoft Gold Partner renewal annually, it is no small feat to continue keeping this status. Microsoft has raised the bar significantly to attain the Gold level status over the last few years.
To earn a Microsoft Gold Competency, organizations must employ more Microsoft Certified Professionals who must pass various up-to-date certification exams related to the competency, submit multiple customer references and demonstrate their commitment to customer satisfaction by participating in an annual survey and score well. There are Cloud marketing assessment tests to pass as well.
Achieving the Gold level status is an investment that we make every year because it truly shows our commitment to stay on top of Microsoft-related technologies, helps us maintain a strong relationship with Microsoft, and serves as a big differentiator among all the other hosts.
As an example of our commitment, it is through this strong relationship with Microsoft that we were able to be the first host to launch a production scalable cloud hosting solution for websites and web applications using Windows Azure Pack.