Records are very rare, and not much is known about them yet. Comments on Notch's blog post seem to indicate that they can be found as loot from killing creepers.
A theory would be that these records would play music made by C418 as he is the official Minecraft music composer.