Problem with irrational neighbors is you may have to live with them for the next 20-30 years, and it's not worth the grief to risk escalation given the potential next steps they might take.

I'd still fence off your property as the old saying goes 'good fences make good neighbors.