Hi Stefan,
EDIT: glad you solved it... I was cross posting. Leaving this here for posterity
A couple "obvious" things to make sure of... 1) that you don't already have the COM port open (either via QGC or another program). 2) Check the serial port pins solder connections on AQ -- it's fairly easy to get a cold solder, especially on the ground pin.
When AQ is in bootloader mode, only the red LED should be on, steadily.
Basically the error you're seeing means that the stm32flash utility can't open the COM port or properly communicate with AQ board (either AQ isn't reachable at all or isn't in bootloader mode). All 3 available baud rates should work, and actual firmware file shouldn't matter. You could eliminate QGC from the process entirely if you want to by calling stm32flash directly from a command line (the full command is shown in the QGC flashing status window).
HTH,
-Max