In reality, time doesn't pass; we pass. Time itself is invariant. It just is. Therefore, past and future aren't separate locations, the way New York and Paris are separate locations. And since the past isn't a location, you can't travel to it.