This is what works for me (I play with the sound off and I'm partially colorblind so this ignores both sound and colors):
I label the five crystals in my head (top right; low right; top left; low left; middle is what I do but you can use whatever works for you). Them I name them (again, in my head) once as they flash, a second time to recall the pttern, and again as the farmer hits them.
I suggested this to a friend who was struggling with the puzzle and it helped (she had previously been trying to use sound like she did with the old Simon game). She used the colors as labels and found it worked better for her if she said the patterns aloud.