Ah...at last. There IS another CA'er living in God's county (I am ignoring those people with funny accents living north of the Wash who think God's county is somewhere up there!!).. I live in the Polden Hills a few miles east of Bridgwater, so not to far from you.
Regarding your question, theoretically, you should be able to pitch on any campsite, as you only need a pass for a vehicle. However, arriving on the saturday, you'll have to hunt for space, but it sounds like you won't need much of that. Find a space, get to know your neighbours, and chain your bike to someones car. I wouldn't leave them just lying around - the camp sites and car parks are pretty deserted for a few hours around the start of the race - except of course for the ever-present Pikey's.
Best of luck with your extended bike ride.
Del