In Alma 8:20, when Alma first meets Amulek in Ammonihah, he asks him for bread. Amulek responds with this words: "I am a Nephite." This is a very bizarre response if Nephites lived in isolation, but makes a lot more sense if they lived among other peoples with other religions/cultures.
In other words, this supports the idea that the Nephites were a small group of people living in a bigger culture, not a monothilic culture dominating the area at the time.
For other examples of "Outsider evidence", see this (growing) list.