From what I recall using a controller on PC, you don't see a cursor that changes when you can interact with stuff (unless you engage the right joystick, I think, and I'm not sure the cursor will change to the plus in controller mode - I personally have never seen it). Have you tried equipping an empty inventory slot and trying to harvest anyway? You should still be able to pick up crops and forage. You just walk up to it and hit the interact button.