That fence may have been there since before World War II. All owners -- on both sides -- may helped build the fence and thought that it accurately marked the boundary between the properties.
But if someone conducts a survey and it turns out that the boundary actually runs elsewhere, you've got to move the fence.