Switch Crab Pot problems

[farflightless]

So I'm trying to bait my crab pots, they're in the water but when I try putting bait in it with Y or A it just picks the crab pot back up.

What am I doing wrong?