When a star enters the red giant stage what causes its outer layers to expand?

When a star does this it moves off the "main sequence" which mean that its core no longer contains enough hydrogen to fuse into helium, (the core is now all helium).

When this happens the core stops producing energy and gravity shrinks it making it denser (and heats it up in the process).

This collapse initiates the process of fusing the helium in the core into heavier elements and the core starts producing energy again.

Also, the region round the core (which is still mainly hydrogen) begins to fuse hydrogen into helium.

These two events cause the central regions of the star to produce more heat that it was doing before and the outer envelope of the star is forced away from the core by the pressure of the energy released. The star therefore expands to become a Red Giant.