Actually I don't think any person can state with full certainty that if there was a brown dwarf we would all see it by now. Especially if coming up from the south.

After all, they did put up that costly south pole telescope for a reason...

I think there is no one on earth who can say they know for sure. No one living today has been alive during the passage of a brown dwarf so it is all based on theory as to when we would see it or how things would unfold. And all we can do is watch the world around us and look at the writings left from the people of long ago...