I could see it being a map reference originally but the first useful-looking web search gives me its first two instances from Shakespeare - who is inconsistent on the matter. In The Merchant of Venice, it's the "four corners of the earth", while in King John, it's the "three corners of the world". I suppose maps can be triangular as easily as square; but they aren't commonly so as far as my knowledge of that period goes.
Equally, I understand how the world contains corners; it's a common feature in buildings. But does that make them the corners of the world, or just corners in the world?