The crux of the matter is the loneliness. Buddha said, “If you find no better or equal on life’s path, walk alone; loneliness is better than the companionship of a fool.”
The hard part is finding that peace within yourself, day after day, alone. It’s why solitary confinement is so effectively maddening. It’s also why suicide rate increases with age.