« Modeling Only the Finest in Modern Sheep Wear | Main | 2006 Author Interview Archive »
December 26, 2006
Comment Moderation, Etc
I've gotten a couple e-mails on this recently from folks concerned when a comment of theirs drops into the moderation queue and then doesn't eventually show up. So I'm going to post this and let it stand as a resource going forward.
First, understand that this site gets a huge number of spam comments a day -- something along the line of 2,000 of them in a day, every day, all year long. The reason you don't see most of them is because I actively and aggressively monitor and filter the threads. My own special cocktail of IP blacklists, keyword moderation, link policing and other secret ingredients punts the majority of these spam comments either into the spam queue or the moderation queue. The spam queue gets about 90% of these; I never look at the spam queue because it's really almost impossible to land there unless you're spamming. 10% of these drop into the moderation queue, which I look at least a couple of times a day. 10% of all spam messages is still a couple hundred messages a day.
When you try to post a message and it doesn't appear immediately, it usually means that it's gone into the moderation queue. You may or may not get a message from my site telling you this. If you've landed in the moderation queue, it's usually for one of the following reasons:
1. You put in more than one URL link in your message (I've specified 3 links as the point at which moderation kicks in, however, other factors may cause you to get punted into moderation for as little as two links)
2. You've got a word in your comment that fires up one of my keyword comment moderation filters. These are largely pharmaceutical-based, although there are certain specific phrases that are commonly used in spam that are also filtered. If you try to tell me I have a "nice site," for example, you'll likely get the boot into moderation.
3. You're writing from an IP address from which spam emanates. Use your anti-virus programs, people.
When you post a message, and any of these three factors are in play, they are weighed against other factors, including whether or not you've posted before (either by IP detection or by name detection). The MovableType software then assigns your comment a rating, and if the rating is less than my threshold rating for posting, off your comment goes, into the moderation queue.
Here are things you need to know about the moderation queue:
1. As noted, normally I check it at least a couple of times a day; usually during an average work day I'll check it once an hour or so. So generally speaking, your comment should be released no more than 12 hours after posting, and usually (during workplace hours) within an hour or two.
2. I don't vaporize comments made from real humans or otherwise leave them in the comment queue. If I find something of yours sufficiently jackassed enough to delete, I'll let it post and then go into the comment and delete it, and leave behind my reasons for deletion. In other words, if you've offended me, what's left of your comment will tell you why.
3. Inevitably, given the amount of spam that lands in the moderation queue, I will from time to time accidentally delete a comment post from a real live person. This person might be you. Please understand that it doesn't mean I hate you, or that you've offended me in some unknowable way -- as noted above, if you've offended me, I'll let you know -- or that you've lost your posting privileges. It just means I've accidentally deleted your post. It happens.
If you've posted a comment and you do not see it on the site, here's what you do:
1. Don't send me an e-mail about it. As noted above, comment moderation is not my passive-aggressive way of controlling people, it's my way of dealing with spam. I don't moderate people for their comments this way.* So you don't need to worry about that. Now, if you've sent me an e-mail about this subject prior to me posting this message, please don't send another e-mail apologizing. I'm not upset. I just don't want you to think there's a problem.
2. Do wait a few hours to see if your comment eventually shows up. I do occasionally have a life outside this Web site, so sometimes it'll take a while for the comment to get published.
3. If after about 12 hours you do not see the comment -- or alternately, do not see what used to be a comment from you with my note about why I deleted its contents -- it's likely I've accidentally deleted it. After you calm your righteous rage at my sloppy comment moderation skills, the solution is simple: post the comment again. The chances of my accidentally deleting the comment a second time are (hopefully) fairly small.
So there you have it: The ins and outs of comment moderation on the Whatever. If you have any questions, please feel free to leave them in the comment thread.
(* I say that I don't use the moderation queue to deal with real live human commenters, but I will note that there are at least a couple of people who, after being warned several times in the comment threads to behave themselves, have lost their posting privileges entirely, and at least part of that enforcement includes moderation. Should you worry that you'll be placed on this list? No. You really have to annoy me over an extended period of time, and if you do, I will let you know that you're headed toward Blacklist City long before you get there. Most of you are not even close to achieving this sort of dubious distinction.)
Posted by john at December 26, 2006 10:52 AM
Trackback Pings
TrackBack URL for this entry:
http://www.scalzi.com/mt2/mt-tb.cgi/4297
Comments
Christian | December 26, 2006 02:08 PM
> I do occasionally have a life outside this Web site!
What! [mock outrage]. The entire universe revolves around this website!
In all honestly, I hope you go a registration system some day. Yeah, I know - it will reduce the fast and flexible feeling for first time visitors, who simply want to say howdy, and then get out of Dodge. On the other hand, I'd be glad to see you get an extra hour free every day (instead of spam swatting), which in turn could be used to amuse Krissy, Athena, - or us poor blokes here in cyberland.
Another thought might be to install a "scramble box" (Turing test) with mixed up numbers/letters next to the "Post Your Comment" button. Surprisingly enough, it only takes about 3 or 4 hours of programming.
Perhaps you can find a whiz-kid right here on the site to brew this up for you.
http://www.php-mysql-tutorial.com/user-authentication/image-verification.php
MWT | December 26, 2006 03:08 PM
Out of the 10% that lands in the moderation queue, how many of those turn out to be legit comments?
Do posts released from the mod queue show up at the end of the comments, or wherever they would've been at the time of posting?
Jon | December 26, 2006 05:40 PM
You should definately consider using a Captcha -- as long as they aren't TOO hard to read, I don't mind them.
John Scalzi | December 26, 2006 07:04 PM
MWT:
About 1% of those comments that end up in the moderation queue end up being comment from real people. The comments, when released, show up as being posted at the time they were initially submitted.
Jon: I don't like captchas. I'll avoid using one for as long as I can. Comment moderation doesn't actually take up a huge amount of time overall. At least not so far.
Nathan | December 27, 2006 01:11 AM
Scalzi said,
"You really have to annoy me over an extended period of time, and if you do, I will let you know that you're headed toward Blacklist City long before you get there. Most of you are not even close to achieving this sort of dubious distinction.)"
You do know there are some of us who might take this as a challenge, don't you?
John Scalzi | December 27, 2006 04:57 AM
Those who would, ultimately wouldn't be missed, I think.
Yaron | December 27, 2006 10:14 AM
You're writing from an IP address from which spam emanates. Use your anti-virus programs, people.
That one is, ahem, a bad one. Well, either a bad one, or a terrible one, depending on what list you check against.
There are common plug-ins that check against lists like the dsbl.org one. These are lists of IP addresses that generated, sometime in the past, email spam. Which is not related to comment spam. Different programs for either kind, different mechanisms, and doing one does not reflect on doing the other. Even if someone actually does use a computer that generates email spam, that does not really increase the likelihood of it generating comment spam (beyond the basis of already knowing this is possibly a compromised computer and may be less secure).
Second, these lists also keep dynamic IP addresses. Which renders the whole point moot. Someone uses his home computer to send spam, the address gets into a blacklist, and a few hours later the IP address is assigned to an entirely different person. A different person who might even not be a spammer.
I had several cases of comments blocked in 2006 because my dynamic IP address was used once in 2003, and another one once in 2005 (or was it 2004?) to send email spam.
Not cool.
CAPTCHAs work, but they're annoying, problematical for many people, and (depending on the method used) problematical for some computers and browsers.
What tend to be less problematical are the purely textual ones, like "how much is 2+2?", "What is John Scalzi's last name?", or "How is the closet dwarf plant to Earth called?"
Zonker | December 29, 2006 09:57 AM
I don't know if your blog system supports it, but I use WordPress with Akismet and the Bad Behavior plugins. They zap about 99.9% of the spam that hits my blog. I don't get 2K a day, but I get a fair amount -- and they have professional keys that are supposed to be able to handle that sort of traffic. You might look and see if Akismet is available for your system or consider moving to WordPress.
John Scalzi | December 29, 2006 10:10 AM
I'm not going to move to WordPress because there's no point in changing software (and entry URLs, etc) four years in. I like MT just fine. Also, I already catch about 99% of my spam before it hits. I just have more of it than most people, so what I do miss is more noticeable.
Yaron | January 1, 2007 02:05 PM
Not that it's relevant right now, but just for the record Akismet does work with MT just fine (or, at least, just like it works with WP. Whether it's fine or not is a matter of some debate, but it usually gets very good reviews).
And I agree, switching a perfectly working software isn't a good idea. As long as you're happy with MT, you should stick with it. Though you could keep the old URIs working if you switch, if that's a concern. It's not entirely elegant, but it works.
MatGB | January 1, 2007 08:34 PM
Hej, as I just got here from the 'you have been modded' page, after I deliberately kept it to two links when a third was good. Second Zonker's recommendation of Aksimet, Movable Type plugin, and really think Bad Behaviour should get ported by someone, it's available for a number of BBs and MediaWiki as well now, so MT should get it at some point. It's soooo good.
Admittedly, I was only getting 2000 spams per week not day, but still. Drawback is I have a dynamic IP. So I sometimes get blocked from my own blog. Nice.
Robert | January 8, 2007 10:23 PM
Jeez ... 2k a day?
I thought I had it rough with 300 - 400 daily garbage.
I used to use MT about one and a half years ago but couldn't handle the amount of comments I had to moderate (I obviously didn't understand how to use the MTBlacklist effectively) I've since moved to WordPress and run SpamKarma2 and rarely have to even touch the moderation queue.
Nick Stump | January 11, 2007 02:00 PM
must have been the mention of my wife's voice being like a certain well-know tranquilizer. Oh well, it's not like the world is gonna miss much not reading one of my comments. I had no idea this spam deal was such a problem. How you manage to keep Whatever running and produce all the great work you do is pretty amazing.
Alan | January 11, 2007 10:54 PM
I have been reading SF since 1958, well over 4,000 titles. I am a normally a patient reader and wait for paperback editions to release.
In your case, I can't wait and have to buy the hardbacks right away! I haven't enjoyed story lines, fresh concepts and writing like yours for decades. Please keep it up! Squeeeeee!
Buy Cheap Paxil Online | February 16, 2007 09:03 AM
text.txt
dedulya | March 15, 2007 11:37 AM
Not that it's relevant right now, but just for the record Akismet does work with MT just fine (or, at least, just like it works with WP.
Mary | December 26, 2006 11:27 AM
Damn. I knew spamming had gotten ridiculous, but it still shocks and saddens me that maintaining a commented blog like this one requires so much of the owner's time. Thanks for the continued effort, John. I really can't remember the last time I saw spam here, and I generally check in several times a day.
I'm also reassured by your clear explanation of your moderation procedures. A few weeks ago, I submitted a smartass comment that I regretted the moment I clicked "Post". It was meant to be funny, but it was pretty heavy-handed. When it never showed up on the page, I was both relieved and embarrassed that you apparently saw fit to consign my comment to the memory hole. But if I'm reading you right, if you had been offended by my comment, I would have been spanked in public for it. So, thankfully, it just got lost in the process. Whew!