I always, always take it with me. But I don't have a helmet lock so that might be part of it . And even if I did, I would take it with me. Too many bad thoughts of someone walking along and pouring coke into your helmet.
I always carry mine aroun too. If I go in a store, I usually grab a cart and put my helmet and jacket in the main basket and use the little fold-out-thingie to hold whatever I am buying. Get some pretty funny looks this way.