Why do you have to stop gardening to talk? Can't you multi-task?
Buy him his own little kit of gardening tools. Give it to him as a present. Keep it with your stuff and when he comes by, break it out and hand it to him. He offered to help, right? Give him his own garden gloves, rake & trowel and take him up on it. It'll probably make him feel useful and talk less. Make sure you put his stuff away though when you're done though. Don't want him wandering over with them to work when you're not out there.