I'm also having a similar issue, though I'm using Nintendo switch and split screen, the host is getting prismatic shards randomly while I'm mining in the skull caverns and they are in their house or doing something else (I'm using the latest version 1.5.4.2 switch). One thing to note is that when they go mining with me the prismatic shards spawn normally so I suspect the shards are spawning in the same coordinates as the place they were supposed to but are spawning in the map the host is in (lets say I broke an iridium node at coordinates 100x 100y 100z in the skull cavern and got a prismatic shard, if the host is in pelican town, I think the shard would spawn at 100x 100y 100z but in pelican town instead).