The only reason you'd worry about infamy is if you were to do the DLC "Knights of the Nine", which requires you to cleanse yourself of infamy by taking a pilgrimage to a shrine each Nine Divine deity. Also, said DLC prevents you from getting their shrine blessings if you have any infamy at all. Personally, I just said "fuck that shit" and uninstalled said DLC (I bought the add-on pack that included every DLC besides the later released castle house).