Hermanus, as you possibly know, is very well known for whale-watching in South Africa. Locals, and tourists, gather in large numbers, during whale season - turning the otherwise comfortable and easy-paced seaside town into a bustling and somewhat congested mangling of cars and people.
Sunny weekends between September and the end of October is certainly not the time to get stuck trying to get into or out of this little town; except if you're either prepared to endure the hours of stop-start traffic, or if you're happy to take the drive to and from Cape Town while everyone else is sleeping. :)
That said, if you're looking for a vibey seaside town, book some weekend time at a hotel or guest house in late September or early October.