Realm Address Card Pointless

Prior to recent update when holding a Realm Address Card I could Travel to Respite of the visiting realm. Now if I Travel to Respite (or Crossways) I return to my game Respite or Crossways. Today I joined a friend in his game he gave me a Realm Address Card and when I loaded down with loot and wanted to return to his estate I returned to my estate instead when I traveled to Respite. This is going to ruin the idea of a group being able to share the workload on one estate if they have to be re-invited to the realm each time they Travel to Respite (or Crossways). Additionally, as a guest I was not able to craft and have resources be used from storage containers.