I could write two backstories that fits what we know from dialogue.
In one, the wizard sleeps once with his wife (the "it's unusual but it's been known to happen" comment). She divorces him for unknown reasons, give the baby up for adoption, turns green and starts cursing the countryside for equally unknown reasons. The adoptive parents named the baby Caroline. Caroline marries Pierre, they move to the valley, Caroline has a romantic relationship with the wizard and when he realizes they may be related he breaks things off without explaining why (which is why Caroline still is worried that Pierre might get jealous if he learns that she visited the wizard in secret). And for some reason it takes the wizard 36-40+ years to realize his magic ink is missing.
In the other, Caroline and Pierre moves to Pelican Town shortly after marriage. While exploring the valley to get to know the neighbors, Caroline chances upon the wizard's tower, meets him and they start a relationship, culminating in them sleeping together once (the "it's unusual but it's been known to happen"), then breaks up with her hoping to avoid his wife finding out. The witch finds out anyway, divorces him, turns green with envy and rage and flies around trying to exact revenge. (the shrine in the shop may have protected Caroline and kept her from getting identified). Abigail gets born, and Caroline wants to keep Pierre from suspecting that he might not be the biological father. It takes 18-20+ years before the wizard realizes his magic ink is missing.
I can see why a lot of you want to think of Caroline having tried to seduce her biological father, but to me that's just ick.