If you have no infamy then you don't have to lower your infamy
because you can't. However, you still have to visit each wayshrine
so that you get a vision from Pelinal Whitestrake. While visiting
the wayshrines would usually lower your infamy to zero, if you have
no infamy then it doesn't affect your infamy in any way.