Both, but sometimes not. Because sometimes factly is considered a word sometimes not. So it depends how you would like to say it.
-- No wonder real war becomes so matter-of-fact to our young people.
-- Matter-of-fact manner in which he had earlier given me some details of his farm work.
-- This may be because it was too matter-of-fact to record.
-- "Well," he said matter of factly,"technically, turtles are reptiles.
-- "yeah" she said matter-of-factly.