No...I would highly doubt any thing on your bike would break b/c of riding on a dirt rode.
Yes...I would think that riding on a dirt rode would require you to really stay on top of maintence items like your air filter and chain. I could also see you having more paint chips on your bike(talking years here)then one that only saw paved roads.
Other then that the only thing that will hurt your bike on a dirt/gravel road is your riding skills. Just use your head and you will do alright.
there's hope in the words and emotion in the eyes
it's so easy to be misled by the savvy gentle guise
and like fools we trust the delivery
but it's all just drunk sincerity