i've always found that you can get back anytime but going you are usually restricted to the train your booked on
we were booked on the 4.30pm on the monday got to the terminal at 1.30 pm and got on the first train available 2.00 pm back to england
have just done been on the fast cat poole/c'bourg and may do this route next year
it appears you either spend a few more hours driving or a similar time sailing - both terminals are equi-distant from home
week days are likely to have less pukeing kids stinking the boat out