Series S vs XBox One shouldn't matter, they're both the same "platform" as long as it's the same game. The only games that would matter for that are Series X|S exclusives, but for those you can't play them on the XBox One at all. (They're both basically PCs actually, the Series X|S is just a more powerful PC so it plays the same games better, and the X|S exclusive games are exclusive because they require a beefier system than the XBox One / OneX|S.... Thanks for the super intuitive product naming, Microsoft! /s)
If you're on different networks it's probably a network performance issue, specifically your NAT type. If you have access to your wireless router's management page (the admin login) you may be able to improve performance by enabling UPnP or NAT-PMP on both systems. UPnP (Universal Plug-n-Play) allows devices (the XBoxes) to create dynamic NAT rules, which should improve your NAT type to type 2 at least. I personally created NAT-PMP rules for my Nintendo Switches [the only systems I ever really played multiplayer on, I don't really have friends :( ], restricting the rules to those individual Switches (which had static IPs assigned) but that's a more advanced setting that not all routers give you access to.
Do this with caution, UPnP and/or NAT-PMP can be less secure! They aren't inherently insecure, but these settings do allow devices to create dynamic NAT rules so they theoretically could allow stuff through your firewall without you realizing/wanting. Realistically there isn't a huge risk since the device making the rules has to already be on your network, and your other devices (game consoles, PCs, etc) should have basic security of their own, but I personally don't allow UPnP or NAT-PMP for devices like my teenagers' PCs, knowwhatImean?