It is also interesting to note how they always:
- Go down to the land/city of Zarahemla
- Go up to the land of Nephi
- Go down to the land of Desolation
These lands do not have the spiritual significance of Jerusalem so it is reasonable to assume they refer to geographical altitude. The references are numerous, so there is no way this is a coincidence.
But there is one exception: Ammon and his small group are going "down into the land of Nephi" in Mosiah 7:6. Which again turns out to be consistent because at this point, they find themselves on a hill right by the land of Nephi. (See Mosiah 7:5).
The Book of Mormon is remarkably consistent.