Why did the USSR finally decide to invade Japan?

Because the United States was about to defeat Japan in a matter of days, and Russia wanted to be in on the post-war decision-making processes. By declaring war once Japan was on the ropes, they didn't really risk much militarily.