Return to Crossways Takes Me to New Map

I am currently in a party with another player and we have been playing together from the beginning. We have our respites on the same map. When I click "Return to Crossways" it takes me to a new realm that I think was the realm I initially spawned in when I am trying to go to the crossways in the same map.

It's not returning me to the crossways in the abeyance where my respite is, it's returning me to the crossways in the abeyance I initially spawned in after the tutorial.