There's a number of things this could be such as misaligned suspension, tire pull etc.
For tire pull, you can switch wheels around and see if persists. If you have unidirectional tires, then swap the rears to the fronts. If you have non-directional tires, swap the fronts over. Then test...