A few points on comments so far:
1. Yes, the freezer nullifies auras, both good and bad.
2. If you can not see a particular Totem Piece, and you have enough Notoriety, you may need to send over some Minions to do more Plotting in that region.
3. Don't keep the pieces in a non-freezer room, even if locked. Surprisingly, sometimes the negative aura can be felt on the other side of doors and walls. I once watched Mariana Mamba use her "Savage Allure" ability on the OUT side of my base, causing one of my minions IN side the base to defect!

The hint I had for this long ago was to (also like body bags) store them in a freezer room. Not sure if this helps the negative attributes - other than no minions go into the freezer for any other reason except putting bodybags there.

Also on this playthrough this time, it auto-assembled them for me (they were all in the same freezer, fairly close together (but not next to each other). I'm sure in the past you had to lay them in a line in the correct order, but I guess not anymore.

good to know

I got the 2nd, 3rd and 4th piece, but can't get the 1st piece which sucks.

Nice! helped me get a better looking inner sanctum. thanks!

Wish I saw this before I finished the game.
I kept the cursed pieces in the private room and let nobody in, simple.
Towards the end of the game when you have that one annoying agent that never leaves, you start to stop caring about everything and just want to finish the game!