For multiplayer issues, I would start here: https://www.stardewvalley.net/multiplayer-troubleshooting-guide/
As you are playing with mods, it's also important that everyone is launching the game through SMAPI. Some mods also require that all players on a server have that mod installed, while others only require the host to install it.