I've seen a couple of threads with people having trouble getting good calibration settings, either with auto-calibration or manually, and I think I've managed to figure out how to get good results.
Step 1: Video calibration
This is the easy part. Use the guitar to auto calibrate for video. You should do it four or five times and get an average of the results -- I've seen it vary by as much as 15 MS. It's best to keep the room as dark as possible when doing the video calibration.
You can test the calibration by muting the sound and trying to play something just based on the video. Usually I use something easy like Psycho Killer or New Kid in School on drums. Things like All The Small Things are good on guitar. Don't worry about it being super-accurate at this point; it should just be "close". If you don't have the RB2 guitar, just start with the recommended setting in the instruction manual. For most TVs the number will be somewhere between 0 msec and 40 msec.
Step 2: Audio calibration
Take the drum kit and calibrate based on the sound, where you have to hit the green pad along with the audio. I've found I'm much more consistent with the drums than guitar, and it's easy to tell when you rush/slow down and should not count the results. Again, you'll want to do it four or five times and take the average.
The reason you're better off doing this manually than using the autocalibration is that the speed of sound is around 1 foot per millisecond, so that the sound will hit guitar's audio calibration well before it gets to your ear. In addition, especially if you turn on Dolby Digital, your stereo may have some sort of mixing that introduces different delay into different channels (for instance I like to play while using a "Music Video" mix, which I know introduces a lot of echo into left and right, but not as much into center). This is really important for people who are doing bar/concert setups.
Step 3: Tweaking the settings.
Now go play something you find somewhat tricky but that you can do consistently. I like to use verse 1B + Pre-Chorus on Afterlife or the Chorus on You Outta Know. See how it feels. The other thing you can do is take something easy (New Kid In School or Wanted Dead Or Alive are good for this) and try to see how much stretch there is in the timing window. There should be roughly even amount in each direction.
If you feel like you have to play late, that is, you think you hear the note before you hit the pad, or there's more give in the timing window after the note is played, move the video lag number down; if it was at 40 msec, try 35, etc. On the flip side. If you feel like you have to play early, that is, you hit the pad and then hear the note, or there's more give in the timing window before the note is played, move the video lag number up. It's doubtful you will need to move the video number more than 10 msec in either direction.
In the end I found that even two or three milliseconds can make a difference, and had to use the guitar to set the number to 37 msec; 40 and 35 felt good but just a hair off. Once I got it the drums felt totally solid, and the guitar felt much better than any previous setting I had used. Remember that you will need to re-calibrate if you change anything about your A/V setup.
At this point, you're probably done. But, if you end up with a negative video lag setting after doing all of this, or have an unrealistically low video lag (5 msec on a projector, etc), it probably means you rushed during the audio calibration. Try increasing both the audio and video lag figures by the same amount, and move back to something close to the video lag setting. This will only affect the timing window as it relates to video, so you'll have to put the sound on mute if you want to test out the settings.
If you try this out let me know if you have questions or problems, or if you think it works.