When all villagers selected, not using closest villager for some actions

As in title, for example, if i want to add my villager to farm (or build something) - so VERY often actions that you are using during game so bug is very important - not closest villager is coming to do that actions. With my experience pretty often the most distant villager is automatically choosed to do that action...