I tried changing from aggressive to main line mode, but now I get:
racoon: ERROR: not acceptable Identity Protection mode
Right, you must use aggressive, according to the m0n0wall docs by Chris:
You must use aggressive mode, as only IP addresses can be used as identifiers in main mode.[*]
I never was able to get the VPN to work from the library, probably a firewall issue.