In Wilderness Adventures, it says:
[. . . ] most campaigns benefit from confinement, if only for the GM's sanity. Confining adventurers is tricky. The old standbys of writing "Here Be Monsters" on the map and telling horror stories about what waits beyond the Mountains of Insanity won't work on fearless monster killers."
He's right. "Here be monsters" is a signpost adventurers go toward, not away from. The book has a lot of suggestions to keep adventurers in the sandbox area. But I have a simple one I usually rely on:
Stamp the sandbox map with "Here Be Adventure."
In other words, there is a whole world out there - but the exciting stuff is happening right here. The sandbox area has the greatest concentration of adventure - the artifacts of great power, the people of consequence (including the PCs), the great treasures, and the events that will affect the world. It's the hotspot of the world. Within its bounds are the main exciting things in the game.
Yes, you could leave the sandbox area and see what's beyond the mountains or past the northern-, southern-, western-, or easternmost edges of the sandbox. But it's not going to be quite as exciting. What's more, it will be hooked back into the sandbox. The magic items you find will need identifying by someone in the sandbox. The map you find leads back to a hidden spot in the sandbox. The bad guy that gets away? She flees to her allies - yes, back in the sandbox.
I've found this method works pretty well. I just tell the group this before play begins, and generally they stick around in the area I mapped out. People want adventure, and they're generally happy to have you say, "this is where it is."
It's better to give a pointer to the fun than try to hang up warning signs about where the fun isn't. It's easily coupled with any in-game explanation (the Mountains of Insanity drive you insane, the Infinite Ocean requires a special ship to cross, it's illegal to kill and murder for loot outside of Krail's Folley, etc. But on a meta-game level, "go here for fun" works.
What about exploration?
This can put a damper on a game of exploration, unless the sandbox either a) is a place to explore or b) has places that need to be explored. If the game is about exploration (instead of conquest, trade, dungeon-delving, or whatever), just make the sandbox the area you explore. Maybe town is off the map, and you just go there and come back. The Isle of Dread works on this principle. There is a map of the "Known World" but the adventure itself isn't there, it's on the island. The Known Worlds are a place to sell your stuff and recruit new adventurers in an X1-centered game.
Some players just won't be satisfied with a bounded area. In that case, you can either satisfy them with occasional forays into the lands beyond the map edge, or just ditch the whole idea of a limited sandbox. Usually the occasional forays will be enough - that big trip to the Imperial Capital, the 5 or 6 session trip to the Island of the Evil Stone Head, a brief trip to another world. If it all hooks back, it both helps the GM by limiting the game prep to the main sandbox area plus a side trip area, and rewards the players for both going on the excursion and for coming back to the sandbox to exploit their rewards for the trip.