This does not happen on PC or PS4. You should go to the Help & Bug Reports section and report it. Include which system(s) and version(s) you've experienced this issue on.
If you don't have challenge bait on your rod (double check if you accidentally put it on), then this is a bug, probably to do with the game mixing up some sort of internal ID distinguishing each bait or something
Definitely do as REtroGeekery said and report it, I haven't seen this issue reported or even mentioned before so your contribution would be helpful