I remember years ago rummaging though records at street vendors outside the Cape Town station - but that was back in the day when cassettes were the only other option. It's more tricky to find LPs these days - specifically new releases.
If you're into vinyl, and living in Cape Town, then I suppose you'd already know about Revolution Records and Maby Vinyl. Besides for flea markets, do you know of any other shops that sell vinyl?