It took us until the 1930's to find Pluto. And that was by a guy who knew it was there, roughly where it was and it took him a couple of years of looking to finally find it.

The only reason we see comets at all is because they develop a very bright and visible tail. If X is on a highly elliptical orbit like a comet and never develops a tail, we won't see it until it is right on top of us, likely inside the orbit of Pluto or closer.

If the orbit is highly elliptical it will hardly move against the background stars making it hard to spot. That's even if it is illuminated enough by the sun to actually see it. We may have found X 20 years ago, but since it doesn't appear to move against the stars we just assume it is a very small and distant star.