Which type of cable will have a better quality signal coaxial or composite RCA?

Composite RCA will be better than a coaxial cable (RG6).

The order of quality and maximum resolution from best to worst is:

HDMI (including HDMI 1.4 3D)

DVI

Component Video

S-Video

Composite

Coaxial cable