If your neighbor agreed you'd build a fence together-pay for it together-your husband would build it-and it veered to his property a few inches on 3 sections is there legal action he can pursue?

A fence that isn't straight is considered to be poor workmanship. Many people get upset about poor workmanship and don't want to pay for it. You'll need to work out the details with your neighbor but don't be too surprised if he/she isn't happy about it. Is the agreement in writing? If not you are on very weak standing. Anything built on your neighbors property legal belongs to them, try to negotiate. I would not get confrontational with the neighbor, keep in mind that they could charge you for taking down the fence and keep the material! The rights of neighbors in this situation is discussed in a treatise called the Restatement (Second) of Torts Section 164 (1977).