Well - according to The Le Mans 24-Hour Race 1949-1973, publ. 1974, in October 1922, Charles Faroux and Emile Coquille met with Georges Durand "to discuss a new form of motoring event based on endurance rather than speed, which was to demonstrate the reliability of the cars, and bring about essential improvements. Electrical systems then left much to be desired and tyres were still far from robust. Faroux proposed a race for touring cars - as catalogued - lasting 8 hours, 4 in daylight, 4 in darkness, whereupon Durand cried "Why not 24 Hours?" ".
Although that doesn't really answer your question, I suspect a 4 pm start allows for marshalls etc to meet and get into position in good daylight, and allows time for people to get home afterwards. It gives a fair time in daylight and night. The race is not always held close to the longest day - so this would give even time to day and night driving regardless of when held.
