I ended up buying an 8" 14mm wrench and a 3' iron pipe. You are only able to get a wrench on it with just your finger tips. The pipe helped with leverage. I had to twist the axle shaft in order to (very, very) slowly twist the bolt out with my finger tips.
Once that last bolt was out the crazy began. The bracket itself needed to (somehow) be twisted around in order to knock the axle out. I couldn't just knock the axle out once the bracket was unbolted because there is a slight angle on the engine block right next to the bracket. So I had to pull slightly on the axle shaft with one hand while trying to twist the bracket with the other.
This is easily the worst design I have ever seen. I can't imagine how a real mechanic would do it differently. Toyota clearly has a design flaw in this model.

I'd love to save the $100 for the new bracket (which I had to buy from the dealer) but it is straight up fused. I placed a bolt in the "snug hole" (I guess it's called that) and tightened it until the bearing broke. Upon looking inside more closely at it, it is just fused solid. Even if I managed to get the bearing out (which I don't think is possible) I would have to spend a few hours trying to grind the rest of it out. I had no idea that metal could fuse like that. It's actually really impressive albeit expensive.