Post hit counts have been off lately. A bot had found a hidden script that moved older hit records to an archive table. I had created this script to reduce the number of records in the hit table, the 5 million+ hit records over the past couple of years was causing occasionally performance issues. I am very sure the bot problem has gone away.
Going forward a rolling 180 days of hits will be kept on file for real-time stats and the rest will come from a static number updated on schedule.
It all started Wednesday when the dedicated server I have hosting all my websites went down and became totally unreachable. As a rule, if I can't ping it I automatically log into my account with the hosting provider and request a power cycle (server restart).
And then I got this email, moments later from the Network Violations Team...
Regarding your server:
After a review of your hosting account it appears your Virtual Dedicated Server has been compromised. It appears there was a vulnerable setup.php script in PhpMyAdmin. The server was compromised on at an administrative level on or before 10/06/2010, allowing attackers to upload multiple attack tools. These tools resulted in complaints by our abuse department, once notified our security team immediately disabled the process and removed the attack tools. This server must be re provisioned to avoid further issues.
I actually only read or saw one word... "re provisioned" and didn't want to believe what it actually meant. So I dropped a dime and called, fuming.
I was that guy, saying everything I could to get them to reconsider, even escalating it, to no avail, all i wanted was a little more time to get some important files before reformatting. They said no... and then an amazing thing happened.
It turns out, **lack of internal communication saved the day. The server reboot team didn't' get the word from the network violations team, so my server was back on and I was pulling down files faster then you can say efilnikufesin.
I waited it out, the second shut down, and took my time moving all my non-vital domains to a hosted service. My main server goes down a lot. It's really just me supporting it and is a single source of failure so it goes down or, gets 'infected' occasionally. Hardware failures, hacks, stupid mistakes, it happens, a server room I have not.
So the rest of the story, if you're still reading, (I know Andrew is), the hacker exploited the phpMyAdmin setup.php file. I guess it's pretty common, and used it to upload a bunch of hacking tools. The network violations team removed all the tools but shut it down and forced a reprovision because they couldn't be sure they removed them all.
I get it, however the problem I have is, yes, you could say it's my problem, <rant from a Linux novice> a "zero day" issue but my server has Fedora 8 and every time I update the packages they're up to date. Even yesterday I tried, everything's current... the reprovision offering, the "Team" wants me to move to, Fedora 8. So what is stopping the same person from exploiting the same issue? Only time will tell. </rant from a Linux novice>
Email #2 from the Network Violations Team:
We have included the log files as well as a list of the files that appear to be causing the attack. The vulnerability in the phpMyAdmin script allowed the upload of these files. These vulnerabilities are common in open-source PHP applications and it is strongly recommended that you keep all applications updated and patched with the latest security and application upgrades to prevent this from happening again. <plug>If you need assistance upgrading or patching an application we have a pay-per-use administrative service where we will do this for you.</plug>
In order to resolve the current issue the server must be re-provisioned. I reviewed the issue with several peers and admins and there is no other solution that would work.
FOUND ATTACK TOOLS
<removed just in case it helps some one do evil>
<removed just in case it helps some one do evil>[etc...]
I've changed around the IP numbers but you get the drift, update your packages, brush you teeth, eat an apple, blah, blah, blah. Until next hack, thanks for reading.
** There could be a small chance, OK, very small chance that the tech I spoke with let the server reboot slide. If so, the guy made my week, and Thank you, cool tech guy, if you ever come across this! (you never know).
Little does he know, he changed my mind. I was going to email him today to let him know. I'll just let the site decide, it has an amazing way of doing that, not only from hits but from blog posts that people write.
Read from the bottom up. I'm posting the raw email so you can see how I tried to work with him. I give him options, I ask him if things sound fair, I state my concern (the readers), I even asked "What would you do if you were me?"
Bottom line, this was an unfinished conversations between two professional trying to work something out. I can't figure out why he brought it public when no action was taken or no decisive decision was made.
And yes, my analogy sucked, and my spelling as well.
UPDATE: Forgot this second to newest email: "BTW: The only reason it concerns me is that some of my readers might now rely on your site to find out I've posted. if you remove my site then I lose readers. It's not "fair" that you abuse that power."
Subject: Re: SharePoint
From: Rockall Design <email@example.com>
Date: Mon, 12 Jul 2010 21:47:19 +0100
To: Yancy Lent <firstname.lastname@example.org>
My last seven post have had between 100 and 300 hits on your site. I'd say that meant your visitors were interested in them (and two were about lotusscript anyway). People like the posts as they know they can read about Sharepoint from the POV of a Domino developer.
Do what you like though, I'm not really that bothered. I just think you're doing a dis-service to your visitors.
I talk about what I'm doing at the time. Right now it's Sharepoint. Next month it could be something amazing with Domino. Who knows...
If you do delist me I think you owe to the people who rely on your service to alert them to my new content to tell them that it's not me who's given up.
On 12/07/2010 20:03, Yancy Lent wrote:
Right, but they're not going to Planet Lotus to read your posts, they're
going there to read all posts about Lotus. It's not a site that
aggregates "people who dig Lotus" it aggregates people that post about
Lotus, something that you've, from the looks of it, have moved on from.
Plus, it's not like your posting about Java, you're posting about
SharePoint, it's like posting about Manchester United on a Liverpool or
So, given we both developers, can't you just create a private Lotus
categorized feed, so PL only aggregates your Lotus posts? Many have done
this, i do this. No one that reads PL cares about the cycling posts I've
put up lately, and if they wanted to learn more about SP, or cycling
they could just go other places to get it.
I don't see how the filtering of content to stay on topic is unfair.
On 7/12/2010 2:14 PM, Rockall Design wrote:
BTW: The only reason it concerns me is that some of my readers might
now rely on your site to find out I've posted. if you remove my site
then I lose readers. It's not "fair" that you abuse that power.
On 12/07/2010 17:42, Yancy Lent wrote:
The problem is many don't know that and think they'll see a steady
stream of SP posts from here on. What would you do if you were me? I'm
thinking of delisting in the interm, sound fair? It is a Lotus blog
after all .
On 7/12/2010 11:16 AM, Rockall Design wrote:
It might well be a short-lived trend. I wouldn't move it just yet ;o)
On 12/07/2010 15:47, Yancy Lent wrote:
I'm sentencing a trend .
Would you like me to move you over to Planet SharePoint or could you
give me two feeds? One Lotus, one SharePoint?
Here is his blog post....
The owner of Planet Lotus, Yancy Lent, asked me yesterday if he should move this site to "Planet Sharepoint" instead. At first I thought it was a joke, but it turns out he has a sister site just like the Lotus one that's only for SharePoint feeds.
Anyway, I said no, don't move it, as the SharePoint posts on here are likely to be short-lived, sporadic and intermingled with Lotus postings. And that, either way, they're of interest to the readers of the Lotus stuff as they're written by a Domino developer.
However, he seems quite insistent and hinted that he'd have to "de-list" me in the interim and asked if I thought was "fair". To which I said, no, I didn't think it was fair. Not unfair on me, but on the users who've grown to rely on his site as their one-stop source of Lotus postings.
If he de-lists me what happens then? Do the visitors I'd lose (7% of my hits) assume I've fallen off the face of the earth? Or do they then have to remember to check here directly or via RSS instead.
It all stinks a bit of control-freakery to me. Expecting to be able compartmentalise a list of blogs in to one bag is bound to be an impossible task. If he didn't have a Planet SharePoint site would his email ever have arrived I wonder?
Yancy said that was I was doing was like:
Coming to a Manchester United forum to talk about Arsenal.
But it's not though is it. Nothing is ever that black and white and his analogy just doesn't work.
What I said to Yancy is:
Do what you like though, I'm not really that bothered. I just think you're doing a dis-service to your visitors.
I also said:
I talk about what I'm doing at the time. Right now it's Sharepoint. Next month it could be something amazing with Domino. Who knows.
What I'm getting round to saying is that if you trust Planet Lotus as the definitive source of Lotus blog posts then maybe you need to be aware that what you're reading is being dictated to you.
I don't know what he's planning on doing, but wanted to make you aware of what he might do. If you're a Planet Lotus visitor and see no posts from me in the coming days or weeks then you know why. Might I suggest you use RSS or Twitter instead.
I put a quick survey here to see if you think what he's suggesting is a good idea. Let the people decide I say.
I read through some of the comments.
- IBM has nothing to do with this site.
- Planet Lotus is censored in the same way Sports Illustrated censors stories on the migratory patterns of the Green Turtle.
- There is a policy for what it takes to get a blog listed, this logically applies to what keeps a blog on the site. If you go off topic why wouldn't I address that?
Thank you for reading.
Voting has started for the 2nd annual Planet Lotus Blogger of the Year award. The voting is the same as last year, simply log into PL and vote. You can change your votes as often as you wish. Voting will end at 5pm EST on the Friday before Lotusphere.
UPDATE: Dropping the Project of the Year all together....
As many of you have seen, a couple days ago I implemented a way to post inline replies or lets call them what they are, comments; on the front page of Planet Lotus. The comments appear right under the blog post itself and any registered member of the forum can make them. This is a half ass effort of mimicking the functionality of Facebook and not an attempt to steal the conversation way from blog posts. Half assed in that a pure ripoff would be ideal; it's the single best thing, in my opinion, about Facebook, and why I loath twitter.
So, is it stealing the conversation away from the blog post? It certainly could be seen as that, but couldn't you also group the posts that get sent to http://twitter.com/planetlotus or the Planet Lotus Facebook group in the same way? Go take a look, people have made comments to those posts, but i don't think those comments are the same as the comments made on the blog post itself. I think the commenter fully understands that it's not a message to the poster but rather to those deciding weather to read the post. The same could be said for Twitter. The comments or replies made to a post are about the topic in general, the poster, etc. This is the same way I hope the replies on PL are used. A way to comment on the post as it relates to being listed on PL.
If PL has jumped the shark, maybe this is a way to go back in time, to that dreaded day; and tell Fonzie to stay home. You could use this as a way to make a comment without it showing up on the actual persons' blog post, just a message to other potential readers pondering the click. Think about it, take a hyperbolic title, you could call someone out on it. How about the 18th post saying 8.5.1 is available for download?
In the end, the user will decide. Just like in many other areas of the site, it could die a slow death. It looks like this is the case since I'm the only one using it .
Some good news, however I'll wait a good month to rule out any DoS suspicions; hopefully it's just the paranoia.
Thank you for contacting Server Support. I understand your server is down after doing a reboot. After reviewing your server it appears that the servers fan went bad. We have replaced the fan and the server is running as normal.
As a result of this, the process of making things more efficient and up-time will continue to be front and center. Much of this is in the way of growing pains; for example the table that tracks hits was 3.3 million rows; this needs to be addressed over time; I have ideas; all of them will take time to implement. In the short term the de-emphasis of certain hit counters will appear, like job postings, how many hits a job posting receives just isn't that important. This theme will continue; but never leave the front page posts
Google Sidewiki seems like a nice way to post graffiti like comments on Planet Lotus throughout the day. I've gone ahead and posted a note to get things started. If you're new to sidewiki here is the first article I found on it while looking up the date it was launched; Sept 23rd.
Not sure how I stumbled across this tonight but its odd. All week I've been wanting to do something similar to PL after figuring out how to use Twitter OAuth for TwunLog.com, it would be nice to have an ajax like way to post small comments about posts on the PL main page.
So I'm curious, is anyone else using sidewiki?
I try to get to all Planet Lotus requests as soon as possible however a request like this moves right up the queue... in this case, processed right away.
Location, Location, Location.
Blogs have arrived on Planet Lotus. The blogs are based on WordPress MU (multi user) and to start things off I've added 100+ blog themes to choose from.
Start yours here: planetlotus.org/blogs/
The FAQ and rules:
- Be Good.
- New blogs will not show on the front of Planet Lotus. You will need to submit your blog in order to have it appear.
- Beta? - Yes, until the wrinkles are wordpress'd out.
- Widget support - Yes
- Themes to choose from - 100+
- Example blog url... planetlotus.org/blogs/yourblog/
- Size limit - I'm not sure what it should be so the default of 10mb will stay and we'll see how that rides out. If you need more ask.
This sounds simple enough.