There are only 33 users (including guests) on the site at the moment, and it has been faster than this with larger numbers, so I am not convinced that that is the cause.
A ping response is not always a true guide to a web conection speed. Big difference between a single (or 4 successive) packet and what is generated by a browser request.