If a borrower dies with a promissory note are heirs responsible for payment?

They are not personally liable, except in the sense that the assets in the estate must be used to pay the promissory note. Thus there will that much less in the estate for them to inherit. If there are insufficient asset to pay the debt, then the holder of the note loses out on the amount that cannot be paid out of the estate.