I got used to fishing on the Switch which also uses a controller, and the best way to Fish is rapid (or not so rapid) fast taps on the X which is seems you're doing. Try to keep those taps as short as possible. Same thing on PC with the mouse click but the idea is the same, little clicks and vary the speed and amount of them to control the green Fishing bar.
One key thing I did notice is that when the controllers are directly connected to the Switch, there was an imperceptible delay between me tapping the button and the bar going up, whereas if I used the controller wirelessly, there was a more noticeable delay, and that made catching Fish harder. Maybe the XBox doesn't have this little delay when used wirelessly but I don't know that one way or another. Something to keep in mind.
Different Fish run away from the bar at different speeds and with different behaviors so the strategy is a little different for each one. I think at some point someone mentioned the best location for catching overall easier Fish, which may have been from the Rivers in town. I'll see if I can find that.
Edit: I just read your other post, definitely try to Fish with your most dextrous hand. Fishing with any additional challenges is a really tall order.