Very interesting hypotheticals. Paul Ryan WAS the face of the Freedom Caucus/Tea Party/whatever you want to call the insane, uncompromising wing of the GOP. But lately, he has lost control of the asylum. They don't really listen to him a whole lot anymore. And the funny thing is that a lot of that section of the party has consolidated into Trump's camp after the nomination. They pretty readily abandoned Ted Cruz.
The moderates/center-right types are far more inclined to crossover and opt to vote for Clinton this election
My personal ideal scenario is that they do split from the GOP. Take Trump's cult, whichever of the Tea Partiers that choose to go with, and shift to the right. Hopefully, that accounts for less than half of the party.
I think for a healthy, functioning government in this country, we need two sane, responsible parties that don't necessarily agree, but can talk to each other, compromise, and work together.
In this scenario, that would be the left and the center-right/old GOP. I assume Ryan would stay in this camp because he's fairly establishment at this point, Trump (and his cult) seem to hate him now, and the original Tea Partiers are already unhappy with him.
If this happens, I'd think the Trump cult would be the most uncompromising and least pragmatic of the three parties. Hopefully, people would realize they're just being angry and complaining all the time instead of actually accomplishing anything, and that party slowly dies out.