Were they on the farm, or did you place them off the farm, possibly in a location where they were in the path of a passing NPC who obliterated them? If the latter, perhaps it was just coincidence that it was the same day you got the skull key.
Also the Crystalarium is a craftable item, so it too is ultimately replaceable.