Definitely don't be afraid to check out a small, "do it yourself" type home studio. Today you can make awesome recordings pretty cheap. I own a small studio like this.
If you want to go the "do it yourself" route, that isn't a bad thing either. I would suggest going a DAW route. Use the best computer you can (which ever band member this consists of) and a really decent computer soundcard for cheap. Those are so easy to come by today. Pick whatever software you'd like to use. Free is always nice... but there are Pintos and Ferraris in softwares. (meaning some that are decent, slow and will get you there and some that are pretty streamlined and speed up the process) Software suggestions will vary for everyone. But since you're doing this for RBN... perhaps Reaper would be the way to go. Kill two bird with one stone... so to speak. Don't forget to try and find some helpfull plug-ins. These aren't the cure all or anything but they will help you. I know some of the plug-ins can be MAJOR $$$, but there are a lot of decent free ones around too.
As for recording tips... there was some very good suggestions from everyone else. These are the ones I would give you from my experience.
RECORDING
1. Practice the song a LOT before recording. Practice with a click track or metronome as much as possible if you can. (You may already do this.)
2. Find the tempo or BPM for the song in the software.
3. Record a rough version IN TIME (all instruments together if you need to) with this and outline your arrangements in the software. Most have flags, markers and such.
4. Then go back one instrument at a time and record with this "rough" version. I find this helps most bands who aren't used to recording multitrack or for a band member who doesn't remember the arrangements or play solitary well.
5. I find this order goes the best with most bands. Drums > Rhythm Guitar or "Main" Guitar > Vocals > (at this point you can usually STOP using the rough version you recorded all together and go by the new tracks) Piano/Synths/Other > Lead Guitar or Solo Guitars> Bass Guitar> Any Others.
POST PRODUCTION
1. Be sure to clean up the tracks. Cut out dead air or hiss when the instrument isn't playing.
2. Add panning for separation. This can be done a LOT of different ways. There's tons of articles and how to(s) online. Experiment and depend on your ear and what sounds the best for/to you. (If you keep everything center you're most likely gonna get a muddy mess unless you have a very sparse instrumented song.)
3. If you can use most of your FX here (Reverb, Delay, Chorus) the better you will be. You'll have more control and it will be cleaner since they won't be processing all the NOISE you cut out earlier. (If you HAVE to use FX in the production stage "aka: Guitar player uses a reverb, modulations 'chorus, phaser, flange' or delay on his guitar' make sure it's IN TIME" also make sure you don't OVER saturate it. You can usually boost it afterwards as where before... the egg is already in the cake and can't be removed so to speak.)
4. EQ sparingly. Always try to take surrounding frequencies away instead of harshly adding them. Find an EQ frequency chart and what they do and how they effect most instruments. These are pretty abundant online. Use it as a "guide." Again go with your aural interpt here.
5. When you mix it. Try this... play the song and get everything in a relative slider position that you want it to sound. (It's most likely not going to stay the same from part to part do to natural dynamics in playing, but get it close to the point where you can play the whole song through and it be decent WITHOUT tweaking any of the instruments volumes through the slider.) After this try using the Volume, Pan envelope nodes most software have today. Try to give things balance by taking volumes DOWN as well as UP on individual instruments. (Example: During a 'Guitar Solo' take the rhythm guitar volume down a bit. This should help the solo stand out more without a lot of volume increase for it.)
There aren't any set rules naturally because things can be achieved differently in many different ways. There is TONS of info online so don't be afraid to get your "eyes" dirty and go read.
I hope this helps you somewhat on doing it on your own. Good luck with the recording and getting it in RBN!
