I would say it depends on the context.
Using "as a matter of fact" would be useful when making an addition to your previous statement that may even include extra information.
Using "fact of the matter" would be useful in a rebuttal during a debate or discussion which may also provide extra information.
The Fact of the Matter was created in 1892.
In a matter of fact, yes.
yes . is a matter of fact ice cream is matter . matter is anything that takes up space !
Yes! As A matter of fact Im going out with him right now
yes, a matter of fact it is happening right now
Earth is a planet. As a matter of fact you live on Earth and are standing on it right now!
yes. as a matter of fact, I'm on my iPod right now :3
While taking an oral survey John replied " I am a democrat" as a matter of fact.
"I love cats, and as-a-matter-of-fact they love me too!" is a sentence with those words.
Matter of taste. Everyone is beautiful, just in their own special way(:
It must be "as a matter of fact" and not "factly" to be correct.
Bill Kaulitz is right handed. As a matter of fact, they are all right handed.