I've just realised I've been making this too hard. I'd been playing assuming that hammer-ons and pull-offs had to be performed as real hammer-ons and pull-offs, if you didn't want to strum them.
But they're not really implemented like that, are they? You can just treat them as "fret-tappable" notes.
I was assuming that a red->yellow hammer-on had to be performed by pressing yellow while red was still down.
And that a yellow->red pull-off had to be performed as a pull-off - you had to be releasing yellow with red down as the red gem hit. That was the thing I was particularly stumbling over.
But no, you can just tap the frets for the gems, exactly the same as using the lower frets in a solo.
Well, that makes it a bit easier. (Although real HOPOs will be the easy option in some cases).