In Jacob 7:4, we learn that Sherem (who contends with Jacob in this chapter) has "a perfect knowledge of the language of the people. This makes no sense in the context of Nephites living in isolation from all other people, but makes a lot of sense if they live among native people who came along with them. Jacob's first spoken language would certainly have been Hebrew. If Sherem has a language advantage over Jacob as it relates to the people, it makes sense his first language is a New World tongue.
For other examples of "Outsider evidence", see this (growing) list.