ive heard that by just adding a muffler you lose power? (torque or hp not sure which one)

i have a 95 3.4 5 spd with some generic muffler and magnaflow cat. (the muffler sounds really good)

i had to replace my cat and now i feel like my car has less get up and go. Could it be that im crazy or is this common with just upgrading parts of an exhaust and not the whole thing?

if i decide to get headers would i get a bigger gain than if i had all stock parts?