City buyers ditch plans to stay close to the office