Grand Canyon North Rim and Zion National Park are well worth the stops! I highly recommend them.
Also, it's a shame that you'll miss CA 120. I rode that in 2005 on my way to Laguna Seca, it's a hell of a rollercoaster ride!
I'm sure you'll still have a blast! You may still hit some gravel. It's hard to tell from the map you posted on FB, but if you're taking Hwy 41 into Yosemite, well, last July when I was there, they were in the process of repaving about 15 miles of 41 going into the park.