I'm sold on Trac. As I said in my post about ruby, I was a little nervous about the ability of spammers to clog up a trac install via the wiki and trouble tickets. However - by using the command line tool trac-admin, you can really control who has access to do what.
The one show stopper is that Trac has no authentication mechanism other than that of the webserver, which is either clear text or digest based, digest requiring an isolated user database. The clear text password support allows a hook into pam or similar authentication mechanisms, but would require SSL for preventing sniffers. Most of the trac traffic would not require sniffers though.
I think I will setup Trac using SSL and provide the certificate signing authority to specific users. I'll probably be able to redirect non-SSL connections to trac.cgi/login to SSL, and any other connections to non-SSL. That would be the easiest in the long run.