I don't wear them as often as I should. Like most of the others, I always try to remember them when taking a long ride, and always wear them on the track.
The following is a guidline recommended by medical professionals:
100 decibels- limit unprotected exposure to 2 hours a day.
115 decibels- limit unprotected exposure to 15 minutes a day.
140 decibels (gun muzzle blast, jet engine

Noise causes pain and even brief exposure injures unprotected ears.
Exposure to noise levels above 85 decibels for extended periods will cause hearing loss. With an aftermarket exhaust and combined windnoise, most riders are probably subjecting themselves to around 130 decibels.
Some people react to loud noise with anxiety and irritability, an increase in pulse rate and blood pressure, or an increase in stomach acid. If you do wear your earplugs for a long ride, I can guarantee you'll feel a lot less fatigued at the end of the day.
It's your hearing, so do what you want.
I SAID, IT'S YOUR HEARING, SO DO WHAT...