It means to get down to business, to get to the matter at hand, to get to work.
Brass tacks are commonly used to tack upholstery down to the frame. If you "get down" to the tacks, you have stripped away the upholstery and stuffing, and are down to the bare frame again.