nofollow revisited

I finally broke down and wrote a script to fix all of the comments, user profiles and signatures so that links will include the 'rel="nofollow"' attribute. I've created a baseline of all user submitted data so that all new content or changed content should be discovered by diff'ing. This should let me catch spam quicker and more accurately.

javascript list

evolt.org officially took over the Javascript list on Aug 15. It looks like the list has finally taken hold as we are starting to get the usual flood of spam to that address. :) The transition went surprisingly well. There was no traffic during the time of the move which made things much easier.

David Merchant's crew at LA Tech, made the mbox and various exports of the members list available (nomail, digest-plain, digest-mime, regular) to me. One gotcha is that the mbox file contained messages that used 'From' as the first word following a blank line. mbox format uses that pattern to delimit mail messages. This threw off mailman's archiver making it look like some messages were dated Jan 1970. A couple ways to handle that: either put a space in front of the 'From' or "escape" them using '>From'. After doing the former, running the archiver over the mbox worked fine.

The other pain is that mailman doesn't have a way of importing members with list options set. So there's no way of importing the list of people that had nomail set or a way to distinguish between digest-plain vs. digest-mime. So I had to write a withlist Python script. I remember doing something like this for the initial move from Ron Dorman's setup to ServerMatrix but couldn't find the scripts anywhere. So in order to save myself the hassle of re-writing it again later, here is the script (with filenames hardcoded):

#! /usr/bin/python
#
# bin/withlist -l -r set_opts.set_digest_plain javascript
#
# bin/withlist -l -r set_opts.set_nomail javascript

import sys;
import os;

import paths
from Mailman import Utils
from Mailman.i18n import _
from Mailman import Defaults

def set_digest_plain(mlist):
    addrs = {}

    fp = open("/home/mailman/javascript/digest-plain")
    lines = filter(None, [line.strip() for line in fp.readlines()])
    fp.close();

    for line in lines:
        addrs[line] = 1

    for member in mlist.getMembers():
        email = mlist.getMemberCPAddress(member)

        if (addrs.has_key(email)):
            plain = mlist.getMemberOption(member, Defaults.DisableMime);
            if (not plain):
               print "Set plain digest for", email
               mlist.setMemberOption(member, Defaults.DisableMime, 1);

    mlist.Save()

    return 1

def set_nomail(mlist):
    addrs = {}

    fp = open("/home/mailman/javascript/nomail")
    lines = filter(None, [line.strip() for line in fp.readlines()])
    fp.close();

    for line in lines:
        addrs[line] = 1

    for member in mlist.getMembers():
        email = mlist.getMemberCPAddress(member)

        if (addrs.has_key(email)):
            nomail = mlist.getMemberOption(member, Defaults.DisableDelivery);
            if (not nomail):
               print "Disable delivery for", email
               mlist.setMemberOption(member, Defaults.DisableDelivery, 1);

    mlist.Save()

    return 1