Quite a few people go to Le Mans via Plymouth, usually on the Tursday before the race.
The atmosphere is pretty laid back on the ferry, with a mixed crowd, young and old.
One big advantage of going via Plymouth is that Brittany Ferries don't seem to jack the price up like they do on the Portsmouth route. I've been as a foot passenger for about £50 return, and a mate took us in his car for about £160 return.
The only downside is the relatively long drive to and from the circuit, but you can't have everything I suppose.
The fishing is said to be pretty good by the way.