Musical fighter: not going to work out well. It would sound like a less structured beatmania that makes sounds when you smack an opponent or maybe just a PvP face off by hitting buttons on a flowing charts... like a weird Frequency clone.
Musical Combat: How does the music factor in without being too hard, unfun, or gimmicky? As a weapon? "Gitaroo Man" did this and was really really hard to play as it came from the era before plastic guitar. As a summoning device? "Occarina of Time" did this and it felt gimmicky to me. And then there's musical RPGs.... mix all of the above with level grinding, possibly linear gameplay and character classes. There's more than one reason many people don't like "Power Gig" and "Warriors of Rock"!
Musical Puzzle: It can work, really, but you get pretty niche. The same people who would buy RB4 are usually the ones who also own things like "Chime" or "Lumines" ("Qubed"). Honestly, I have seen more players interested in playing Rock Band with me versus the other games when I have left them the option. People like melding with the music more than fighting against it.
And then there's "Rez" and it's sequel, "Child of Eden". Both are Musical rail shooters that use massive doses of Synesthesia to suck you into the game. Hard as nails games, but surprisingly effective use of music. I think this avenue could use more exploring and could hold a lot of promise if approached correctly. "Rock of the Dead" is an example of this failing horribly.