Can a relief pitcher hit instead of the DH?

Yes. There is no obligation to use the DH in baseball.

A somewhat confusing question.

A relief pitcher can be used AS the DH. However normal DH rules apply and that player could not pitch in the game.

A relief pitcher can pinch hit FOR the DH as well. That pitcher then becomes the DH and normal rules apply. I am pretty sure that a relief pitch that enters the game as a pitcher can not be inserted into the lineup for the DH but I could be wrong.

Of course you don't have to use the DH but you can't use a relief pitcher in your lineup unless he's playing a position in the field.

Yes. When you bring in the relief pitcher, you can at that time have him hit in place of the DH. You can also have him pinch-hit for the DH at a later point in the game. Note that in both cases, there is no longer a DH and all pitchers must hit. Baseball rule 6.10.