I seem to recall that from Monterey to San Luis Obispo, that on a straight drive through, 1 was about an hour or so longer than going via 101. Some of the difference is made up in getting back to the 101 from Monterey. I find I stop more often on the 1 for viewing, leg stretches, photography, etc., and don't do straight through drives that way. But since at times I've stayed in Morro Bay, Cambria and Paso Robles, I'm kind of guessing because I haven't always looped the same ways or from the same places. Highway 1 Big Sur drives, especially in summer can be slower due to the presence of more vehicles, slower drivers and RVs. It's not an easy drive and can be very stressful for drivers not used to windy roads and cliffs.
Most of the route using 101 is freeway or 4 lane divided highway. Most of the 1 is not and in some stretches is two lane rural roads and in some places doesn't lend itself to fast driving at all. Besides the Monterey to San Luis Obispo stretch, there are segments of 1 south of San Luis Obispo and Pismo Beach to near Gaviota and that would definitely be slower as well. 1 south of Ventura through Malibu is also usually slower than the 101/Ventura Freeway.
If planning a trip, I used to figure if I was sort of hustling, to get to Santa Barbara in something over 2 hours from Pasadena, two more hours to San Luis Obispo. Then 4 hours to San Francisco. Although I'm usually not doing a one day drive all the way anyways and usually take more time than that for stops, meal breaks, etc.