It's a challenging to do however it is possible. Go past it and look for a cleaning to clear trees if you can't afford to lose the camp. Now build your wall there and it will instantly extend there! And you get to keep the vagrant camp!