Someone would immediately go back in time to kill the person who invented time travel, before it was invented. So that this small group of people could have time travel all to themselves, without interference or meddling from others.

For example, you could safely go back and buy $10k of Dell stock, knowing you can sell it for $1 million in a few years. But if everyone did that, it might screw things up somehow, for everyone who'd be in on it.

Furthermore, they'd kill anyone else who'd be on the verge of discovering time travel. For the same reason, to prevent interference and meddling.