Why is the governor general not considered to be the most powerful position in Australia and New Zealand?

The questions proceeds from a misconception. The Governor General's role, while mostly ceremonial, actually yields more power than people realise. The Governor General has the power to remove the Prime Minister, but this power has only ever been utilised once in Australia's history.