You need to install a vent direct to the outside that feeds the fire burn chamber with air.
That way it doesn't pull cold air through your air leaks at windows and doors in your house.
It will draw air directly from outside to fuel the burning process, keeping the warm air in your house and not causing a negative pressure that pulls cold air in.