4 And it came to pass that he departed into the wilderness. And he left his house, and the land of his inheritance, and his gold, and his silver, and his precious things, and took nothing with him, save it were his family, and provisions, and tents, and departed into the wilderness.
Lehi's house and the land of his inheritance were not the same thing.
Alma 10:3 states Lehi was from Manassah, which was one of the northern tribes who lost their land in 720 BC and later regained access to it on heritage grounds. His house was in Jerusalem, where some displaced Israelites had taken up residence after the loss of the northern kingdom. This becomes more clear in 1 Nephi 3:16, when Nephi and his brothers "go down" from Jerusalem (elevation 2474') to the land of their inheritance to collect riches in an effort to purchase the plates from Laban.