I made my first trip to a track last weekend (Brands Hacth in Kent). I was only there to watch the British Superbikes, not to take part. I was amazed just how fast those riders were going It doesn't look that fast on TV.
As for track days, the thing that puts me off is the cost and the fear of damaging my bike. I'd love to have a go, but on a hired bike, not mine.