Why would I std::move an std::shared_ptr?

Answers to this question point out that moving a std::shared_ptr is all about speed, but nobody explains why it is faster in detail. How expensive is it really in comparison? Is it worth optimzing when one uses it a lot?