PlanetLotus and the Unexpected Server Rebuild
It all started Wednesday when the dedicated server I have hosting all my websites went down and became totally unreachable. As a rule, if I can't ping it I automatically log into my account with the hosting provider and request a power cycle (server restart).
And then I got this email, moments later from the Network Violations Team...
Regarding your server:
After a review of your hosting account it appears your Virtual Dedicated Server has been compromised. It appears there was a vulnerable setup.php script in PhpMyAdmin. The server was compromised on at an administrative level on or before 10/06/2010, allowing attackers to upload multiple attack tools. These tools resulted in complaints by our abuse department, once notified our security team immediately disabled the process and removed the attack tools. This server must be re provisioned to avoid further issues.
I actually only read or saw one word... "re provisioned" and didn't want to believe what it actually meant. So I dropped a dime and called, fuming.
I was that guy, saying everything I could to get them to reconsider, even escalating it, to no avail, all i wanted was a little more time to get some important files before reformatting. They said no... and then an amazing thing happened.
It turns out, **lack of internal communication saved the day. The server reboot team didn't' get the word from the network violations team, so my server was back on and I was pulling down files faster then you can say efilnikufesin.
I waited it out, the second shut down, and took my time moving all my non-vital domains to a hosted service. My main server goes down a lot. It's really just me supporting it and is a single source of failure so it goes down or, gets 'infected' occasionally. Hardware failures, hacks, stupid mistakes, it happens, a server room I have not.
So the rest of the story, if you're still reading, (I know Andrew is), the hacker exploited the phpMyAdmin setup.php file. I guess it's pretty common, and used it to upload a bunch of hacking tools. The network violations team removed all the tools but shut it down and forced a reprovision because they couldn't be sure they removed them all.
I get it, however the problem I have is, yes, you could say it's my problem, <rant from a Linux novice> a "zero day" issue but my server has Fedora 8 and every time I update the packages they're up to date. Even yesterday I tried, everything's current... the reprovision offering, the "Team" wants me to move to, Fedora 8. So what is stopping the same person from exploiting the same issue? Only time will tell. </rant from a Linux novice>
Email #2 from the Network Violations Team:
We have included the log files as well as a list of the files that appear to be causing the attack. The vulnerability in the phpMyAdmin script allowed the upload of these files. These vulnerabilities are common in open-source PHP applications and it is strongly recommended that you keep all applications updated and patched with the latest security and application upgrades to prevent this from happening again. <plug>If you need assistance upgrading or patching an application we have a pay-per-use administrative service where we will do this for you.</plug>
In order to resolve the current issue the server must be re-provisioned. I reviewed the issue with several peers and admins and there is no other solution that would work.
FOUND ATTACK TOOLS
<removed just in case it helps some one do evil>PHPMYADMIN EXPLOITATION
<removed just in case it helps some one do evil>[etc...]
I've changed around the IP numbers but you get the drift, update your packages, brush you teeth, eat an apple, blah, blah, blah. Until next hack, thanks for reading.
** There could be a small chance, OK, very small chance that the tech I spoke with let the server reboot slide. If so, the guy made my week, and Thank you, cool tech guy, if you ever come across this! (you never know).
November 7th, 2010 - 05:17
Fedora 8 (dating 2007–11–08) is out of support for some time. You normally won’t get any updates because there aren’t any. If your Linux knowledge is improvable then think about switching to Debian Linux. You will get security updates for several years (depends on the release cycle but always longer than Fedora), have in-place major release upgrades and the default setup is quite good.
Security updates basically is aptitude update (to get latest packages) followed by aptitude dist-upgrade (to upgrade all including security) and you are done. You can subscribe to the Debian security mailing list so your mailbox will bing when you need an upgrade. Of course this might all work with Fedora too.
For all admin tools you should change the default pathname to something only you are familiar with.
Crawlers will check for those Urls like http://yourdomain/phpMyAdmin. You can change it quite easily. If you know the Url you can do a search on the /etc directory to get all files that deal with it.
Something like
grep -il phpMyAdmin /etc -R
will list all files that contain the string phpMyAdmin (the i is for ignoring cases)
It seems that you run your server without making backups. I consider this a bad idea but no risk no fun.
Install a network traffic tool like vnstat and monitor network traffic. If your server gets compromised traffic normally will grow exceptionally. Maybe your hosting provider also has some traffic alarm tools. If you know of a problem before your provider you might get some time to save files.
If you run everything on one server consider setting up virtual machines (you will need several ip adresses for this). This will cause some overhead but if one vm partition get compromised some others might still be clean.
If your servers get compromised several times your provider will end your contract. In theory you are often even responsible for any traffic or damage that has been done by your compromised server. This can become quite expensive.
In Germany you could get managed servers for very little money but I am aware that this is a local phenomena.
Of course I wish you all the best that your server now stays clean. Your blog has a nice Cannot load “mcrypt extension. Please check your PHP configuration” message at the bottom. In Debian this would be “aptitude install php5-mcrypt”
November 7th, 2010 - 09:46
if you have access to the OS why not make a local backup every hour so if its needed to reprovision you dont risk to much data
November 7th, 2010 - 10:48
Yancy,
Sorry to hear about this. I truly admire the work you do for the Lotus Community. You Rock!
November 7th, 2010 - 11:10
Yes – thank you very much for the service you do for all the “yellowblooded” people. PlanetLotus is the most important portal to get the newest Infos about Lotus. I allways tell the peope visit this site every day and you are “up to date”.
November 7th, 2010 - 11:49
Thanks for working so hard to restore an important community service. I think we all tend to take it for granted some time how much effort is required to keep all thing planet lotus running smoothly. @belgort +1 You Rock.
November 8th, 2010 - 11:17
I agree with everyone else, great job Yancy! Sorry you had to spend time on this. I believe in painful and slow death for all hackers and spammers.
November 8th, 2010 - 14:02
First of all Yancy, thank you. Secondly, thank you again.
Thirdly, this strikes me as yet another example of the dangers of relying on the “cloud”. Specifically, privately run clouds. Private entities have various motives for maintaining some service, be it profit or altruism or simple enjoyment. The dangerous part comes when a specific service reaches a point where a significant community of people come to depend heavily on the service, to the extent the very health of that community will be impacted by it’s demise. When something that important is held together by a single individual, we have a problem.
We’ve seen this before, on a much grander scale but the idea is the same. A century ago many American cities were blessed with pretty good public transportation networks in the form of streetcars. These services were integral to the health and vibrancy of the urban communities they served. They were largely if not completely privately run. After WWII, as cars and suburbs diminished the ability of these companies to run their services profitably, they pretty much all disappeared before the communities that depended on them figured out what they had lost. The basic mistake these communities made was not to realize that *public* services cannot be trusted to private companies, at least not over the long term. Some manner of public subsidy is sometimes a required, and sensible, measure to ensure those essential services are maintained. This particular failure helped ensure decades of urban blight across many American cities that fractured those communities in too many ways to cover here.
How this lesson applies here or more generally to other, much bigger, private clouds (Gmail, Facebook, Yahoo Groups, Flickr, et al.) is a question that obviously concerns me. It should probably concern everyone. Unless that is you don’t care what happens to your stuff in 5, 10, or 50 years.
November 9th, 2010 - 15:23
maybe time to switch to lotus notes?
#justsayin
November 9th, 2010 - 17:29
John, I think you may be on to something! Short of printing everything on your Facebook onto acid-free paper and storing it in airtight clay jars in some desert cave, I can think of no better long term archival tool than Notes for saving *content* in a searchable form. Won’t help keep the Facebook community going in the long term, but in the long term we’re all dead and the historians will have all they need anyway.