Kep, Cambodia has a dozen small islands just off the coast. The main island, Koh Tonsay (or Rabbit Island) has a few small restaurants and a half dozen places to stay for $5 to $15 a night, and is currently undergoing development.
You can swim, fish, snorkel, scuba dive or just take a boat ride around the islands.
One island just off Kep's coast is not in Cambodia, and you can not go there without a passport and visa. Phu Quoc Island is in Vietnam, and is several times larger (and more developed) than all of Cambodia's islands combined.