PlanetLotus and the Unexpected Server Rebuild

It all started Wednesday when the dedicated server I have hosting all my websites went down and became totally unreachable. As a rule, if I can't ping it I automatically log into my account with the hosting provider and request a power cycle (server restart).

And then I got this email, moments later from the Network Violations Team...

Regarding your server:

After a review of your hosting account it appears your Virtual Dedicated Server has been compromised. It appears there was a vulnerable setup.php script in PhpMyAdmin. The server was compromised on at an administrative level on or before 10/06/2010, allowing attackers to upload multiple attack tools. These tools resulted in complaints by our abuse department, once notified our security team immediately disabled the process and removed the attack tools. This server must be re provisioned to avoid further issues.

I actually only read or saw one word... "re provisioned" and didn't want to believe what it actually meant. So I dropped a dime and called, fuming.

I was that guy, saying everything I could to get them to reconsider, even escalating it, to no avail, all i wanted was a little more time to get some important files before reformatting. They said no... and then an amazing thing happened.

It turns out,  **lack of internal communication saved the day. The server reboot team didn't' get the word from the network violations team, so my server was back on and I was pulling down files faster then you can say efilnikufesin.

I waited it out, the second shut down, and took my time  moving all my non-vital domains to a hosted service. My main server goes down a lot. It's really just me supporting it and is a single source of failure so it goes down or, gets 'infected' occasionally. Hardware failures, hacks, stupid mistakes, it happens, a server room I have not.

So the rest of the story, if you're still reading, (I know Andrew is), the hacker exploited the phpMyAdmin setup.php file. I guess it's pretty common, and used it to upload a bunch of hacking tools. The network violations team removed all the tools but shut it down and forced a reprovision because they couldn't be sure they removed them all.

I get it, however the problem I have is, yes, you could say it's my problem, <rant from a Linux novice> a "zero day" issue but my server has Fedora 8 and every time I update the packages they're up to date. Even yesterday I tried, everything's current... the reprovision offering, the "Team" wants me to move to, Fedora 8.  So what is stopping the same person from exploiting the same issue? Only time will tell. </rant from a Linux novice>

Email #2 from the Network Violations Team:

We have included the log files as well as a list of the files that appear to be causing the attack. The vulnerability in the phpMyAdmin script allowed the upload of these files. These vulnerabilities are common in open-source PHP applications and it is strongly recommended that you keep all applications updated and patched with the latest security and application upgrades to prevent this from happening again. <plug>If you need assistance upgrading or patching an application we have a pay-per-use administrative service where we will do this for you.</plug>

In order to resolve the current issue the server must be re-provisioned. I reviewed the issue with several peers and admins and there is no other solution that would work.

FOUND ATTACK TOOLS
<removed just in case it helps some one do evil>

PHPMYADMIN EXPLOITATION

<removed just in case it helps some one do evil>[etc...]

I've changed around the IP numbers but you get the drift, update your packages, brush you teeth, eat an apple, blah, blah, blah. Until next hack, thanks for reading.

** There could be a small chance, OK, very small chance that the tech I spoke with let the server reboot slide. If so, the guy made my week, and Thank you, cool tech guy, if you ever come across this! (you never know).