Noppita
My PID setting could not be easier, defaults. If your seeing oscillations then you likely have too many vibration getting to your AQ board. The long arms might be amplifying prop and motor vibrations. You could try my trick of damping the vibrations with a string or any way to connect the motors together. You have a motor mix that should work and your able to get in the air with some control so my guess is that vibrations are your problem.
You can see my frame has shorter arms than yours but I still have too much vibrations. Larger diameter carbon arms would likely solve the vibration for my octo but the string of Vectran is much cheaper and I'm able to fly even with a number of chipped props. Any type of string will work but Vectran would likely be the best.
http://www.youtube.com/watch?v=_7im1ExXEugAfter you get your octo flying in manual you may have to move your ESC out on the arms near the motors. Your compass might have problems with them mounted near the AQ board. Be sure to have all your power wires twisted so they do not generate a magnetic field.
I always lower Tilt Rate D to 5000 before I try to use RTH and then raise D till I see a little overshoot home when using RTH. Then I lower D a little till RTH has the AQ copter stop right on top of home.
It took me a long time to understand that getting just a few things right while building an AQ copter, motor mix, vibrations, wiring, parameter file and using default PID is all that is needed to have a nice flying copter.
Larry