There are actually 3 villagers with purple hair - the Wizard, Abigail, and Jas. Both Abigail and Jas spend time in the woods near the Wizard's Tower, and for Jas, there are virtually no facts known about her parents. The claim is that they died, but do we have more proof of that claim than we have proof that Pierre is Abigail's father? The only children which are clearly the offspring of their parents are Sam, Vincent, and Maru. I'd buy Abigail, Jas, or Emily as the daughter of the Wizard. As far as Caroline, I hadn't previously considered it, interesting theory! Her green hair certainly is an outlier which isn't often discussed.
I do want to throw this out there as well - I personally think it's implied that if Abigail is the daughter of the Wizard, it's not as a result of an affair, but rather a side effect of some occult ritual. After all, he's engaged in occult research, he indicates "the affairs of mundane folk matter little to me", and the specific line he says regarding his daughter is "You musn't tell anyone... but I have reason to believe that one of the locals is actually my daughter... I could be mistaken... It's rare, but it does happen." The idea of him being disinterested in mundane folk and them being scared of him plus this doesn't to me lead to having an affair with Caroline, since that would make the "It's rare, but it does happen" line pretty odd.
Plus, Caroline says she "doesn't know where she gets it from" in relation to Abigail's interest in the occult, she believes Abigail's natural hair color is chestnut brown, and she says she was a free spirit before she met Pierre (implying she was not afterwards). If she had been engaging in an affair with the Wizard, she would know where Abigail's interest in the occult comes from and would not think she was dying her hair. So I think Caroline was just standing too close to the Wizard's tower when something was going on, and one thing led to another. That's my theory at least.