Yeah, it seems the "magic number" with that game is four investagators. I also had to create a flow chart of sorts to help me keep up with all the rules.
I can just about gaurantee you are playing it wrong in some way. PM me your e-mail and I will send you my rules flow document. It includes rules for the first two expansions, but it shouldn't mess you up using it for only the base game... believe me, it really does help.