When I tried Gloom Site of Power the follower doesn't come at all to downstairs. I had to completely fight all enemies alone. I tried it several times, every time the same. It has been worse since the update 0.3. Companion gets stuck somewhere and doesn't follow at all. They do teleport to home realm though luckily with you. And little less annoying bug: Duplicate companions (because of previous glitches) do not disappear from the home realm and they still chop trees. I had to rehire them and forbid every action.
On my end, after some point in time, it started to stick and not move at all during the battle. It adds fuel even when fuel addition is disabled and there seems to be a lot of bugs in the behavior.