If the edges of a glass block were rounded instead of square what would happen to the density?

Nothing. Density of a particular substance remains the same no matter how much of the substance is present or what shape it is in. Since density is measured in mass / volume, adding or taking away material adds both mass and volume or subtracts both mass and volume, keeping a constant proportion.