Usually, a resonator works in conjunction with the muffler to further quiet down the exhaust. They are common with Lincolns where the exhaust needs to be super quiet, for the old people, (just kidding, LOL) with out hurting the performance much.
A resonator (helmholtz resonator) uses a tuned cavity to actually actively cancel out sound. It creates a band-pass filter that eliminates problem frequencies - so they tune it for a specific loud spot in engine operation. It targets only a small range of frequencies.
A muffler is usually a series of expansions that take energy from the gas flow to decrease sound. Sometimes they have resonating chambers built in, but typically they operate on a purely expansive vane - expanding a gas so many times into such a volume muffles certain frequencies.
Basically, in the real world, with a resonator you'll be able to quiet specific exhaust notes - usually targeted to a cruise RPM to cancel that frequency, but you'll still have a lot of harshness from the high frequencies the resonator passes, which a muffler will cut out.
The good thing with resonators is that they usually have zero flow restriction, so adding them to quiet an existing setup will cost no power.
So to answer your question - they work best together, but if you were to have only one, I would go with a muffler - it will kill the harsh notes, even if its not specifically targeted for the engine note.
Yep! some states require a muffler and they are not fooled by a resonator because resonator work best at a small range of frequencies at specific rpm ranges better than others. - Resonators have a "rattle" sound (when used alone) when you are taking off and pulses are high pressure and slow. Some resonators produce more back pressure than a good muffler that is quieter.