Building with GCC - success!

News, Setup, Compiling, Flashing, Development

Re: Building with GCC - success!

Postby aBUGSworstnightmare » Fri Apr 08, 2016 6:35 am

I would check the .map files to find out about memory usage/free memory...
How long time will it take you to figure out how many WPs you can/want to use? You can download a 30days full featured eval version of Crossworks so no need to spend 150bucks for figuring this out.

By the way: do you live in your business rooms? License CW for your private use (home address). As long as you don't start earning money from your Personal Non-Commercial License nobody/will care how and where you use it (ever heard of 'bring your own device' --> no problem to take your PC with you wherever you want to ...)
aBUGSworstnightmare
 
Posts: 1460
Joined: Fri Jun 22, 2012 5:24 pm

Re: Building with GCC - success!

Postby Max » Fri Apr 08, 2016 11:10 am

Ye, this bring up a point I've been meaning to mention. The ARM architecture, as you may know, requires proper memory alignment of data structures (in 32-bit registers in the case of STM32). For AQ specifically, this means all stack sizes have to be multiples of 8, and in many cases how the variables are laid out in the data structs (eg. navData, controlData, etc) can make a difference as well.

When alignment isn't correct, the problem may (or may not) manifest itself in different ways. The first "victim" seems to always be printf/scanf, which start corrupting the output/input when dealing with floats/doubles. If the misalignment is "bad enough" it will lead to random crashes or lockups.

printf/scanf are mostly used in AQ for writing/reading params from DIMU EEPROM and SD card file. So, a good test of alignment (after making changes or whatever) is to do a test write/read of params to/from SD. It is immediately obvious when something is wrong because all the numeric values (the floats) have unreasonable values (most are FLOAT_MIN it seems like, but it varies).

Saving calibration params to EEPROM can also reveal the issue, but having corrupt data in IMU settings can cause other problems, so that test is best saved for last after writing to SD is sure to work.

So, in your testing with GCC, could you (if you haven't yet) please verify that writing/reading params is OK?

(And yes, something as seemingly simple as changing # of waypoints CAN affect the structure alignment and potentially lead to other problems. I've spent hours and days chasing down alignment issues -- the last time because I changed a struct member from 8b int to 16b w/out thinking much of it, then several days later discovered the issue and had to backtrack until I figured out what caused it... PITA.)

Cheers,
-Max
Max
 
Posts: 2814
Joined: Mon Aug 13, 2012 9:45 pm
Location: Near Ithaca, NY, USA

Re: Building with GCC - success!

Postby Max » Fri Apr 08, 2016 11:15 am

The best check of RAM capacity is the ELF size report generated after a build -- either by CW (with pretty graphics) or by the script I wrote in the current Makefile (no graphics but more data).

This is just an example from some random build I was working on last... not meant as a baseline, just an example of the output.

Code: Select all
---- Size report ----

Section Details:
section      size(B)            addr   loc
.ccm           57304      0x10000000   C
.stack          1024      0x1000dfd8   C
.vectors         392      0x08000000   F
.init            560      0x08000188   F
.text         224876      0x080003b8   F
.rodata        24164      0x08037224   F
.data            680      0x20000000   F
.data_run        680      0x20000000   S1
.bss           92084      0x200002a8   S1
.heap          18432      0x20016a5c   S1

Totals:         KB     usage   of ttl   KB free
 Flash:     244.80    23.91%     1024    779.20
   CCM:      56.96    89.00%       64      7.04
 SRAM1:     108.59    96.96%      112      3.41
 SRAM2:       0.00     0.00%       16     16.00


(My report separates SRAM1 and 2, although they're a contiguous block of memory and CW mixes them together in its report... for our purposes right now they're just one memory block.)

-Max
Max
 
Posts: 2814
Joined: Mon Aug 13, 2012 9:45 pm
Location: Near Ithaca, NY, USA

Re: Building with GCC - success!

Postby wangyeee » Sat Apr 09, 2016 4:54 am

Max,

Thanks for pointing out the alignment issue. I did tests on save/load params this morning and here's the result:
  • Internal Flash: save/load success
  • EEPROM: failed to load calib data. However, DIMU tare and calib save is working. calib data are also successfully loaded on startup.
  • SD: save/load not working. AQ seems to load params.txt on startup, but the value of parameter doesn't change when I view it in qgc.

I'm scared upon the results as I was flying the same build few days ago. I'm looking into the alignment issue now...

Regards,
Ye
wangyeee
 
Posts: 37
Joined: Sun Feb 22, 2015 5:20 am

Re: Building with GCC - success!

Postby wangyeee » Sat Apr 09, 2016 1:18 pm

Max,

Some updates upon load/save eeprom.
By using instructions here https://blog.feabhas.com/2013/02/develo ... cortex-m4/ I managed to find some details of the problem.
The sscanf function actually could return normally, but after that, the first time OS did a context switch(yield functions call in this case), it ended up in hard fault handler. The value of CFSR was 0b00000000000000100000000000000000, which based on AN209 means "Invalid state Usage Fault"
the processor has attempted to execute an instruction that makes illegal use of the Execution Program Status Register (EPSR). When this bit is set, the PC value stacked for the exception return points to the instruction that attempted the illegal use of the EPSR. Potential reasons:
a) Loading a branch target address to PC with LSB=0.
b) Stacked PSR corrupted during exception or interrupt handling.
c) Vector table contains a vector address with LSB=0.

I check the stack as well:
R0 = 0x0
R1 = 0x10004b3d
R2 = 0x7fffffff
R3 = 0x0
R12 = 0x8
LR = 0x800968b (https://github.com/wangyeee/aq_flight_c ... ask.c#L776)
PC = 0x0
PSR = 0x40000200
The PC is zero, which I could only think of one explanation.
1. sscanf is called to parse data from eeprom, and it returned nornally.
2. Another SPI txn is kicked off to read more data from eeprom. yield is called to wait for DMA to complete.
3. OS saved the current contexts in (comm) task stack before switching to another task.
4. SPI DMA completed and comm task became running again. But when it comes back, for some reason the OS failed to restore all registers correctly which turned into hard fault.

To further narrow down the cause, I did two experiments:
First I removed all yield calls in eeprom.c and tried a read operation. It did't error out immediately but as you can't stop OS from switching contexts. It ended up in hard fault as soon as OS switched context.
I also removed sscanf calls in configParseParams(). This time everything seems to work without any error. But as you have mentioned, sscanf is the first casualty of memory mis-alignment, replacing it blindly can't eliminate the cause of problem.

Regards,
Ye
wangyeee
 
Posts: 37
Joined: Sun Feb 22, 2015 5:20 am

Re: Building with GCC - success!

Postby Max » Sat Apr 09, 2016 2:00 pm

Hi Ye,

You do know how to dig into a problem, bravo!

Yes in fact at one point printf/scanf was replaced by a custom formatter function which didn't cause the data corruption... but that only hid the underlying problem and eventually it caught up. Granted that printf/scanf with floats/doubles is a known trouble-maker... many would tell you never to use it in embedded systems like this. But when it works, it works fine, "just" eats a bit of heap space while it's working.

Anyway, I recently discovered that the RUN_TASK stack is not aligned properly in the current code. Somehow it escaped the "big re-alignment" :) of 2015. Right now it doesn't seem to be affecting anything with the CW builds, but with the different toolchain, who knows.

So I would start there and bump it up to 256. Ideally this will cure it. There's also a chance it will cause some other structure which is used within runTask to become misaligned... and there are quite a few. :evil:

My current working version has all this fixed (I hope)... and lots of the major structures re-aligned properly. (Speaking of RAM, this actually resulted in a nice net gain of a few % in available CCM & SRAM.) But this is all part of a whole slew of changes I've been working on. I'm just now starting to break it up into sensible commits to new feature branches (which is tricky when you re-arrange a quarter of the header files :roll: ).

So, if fixing the RUN stack doesn't fix it quickly, maybe I should send you what I'm working with now. It's all slated to be pushed to GH anyway at some point soon, but you know how that goes...

Best,
-Max
Max
 
Posts: 2814
Joined: Mon Aug 13, 2012 9:45 pm
Location: Near Ithaca, NY, USA

Re: Building with GCC - success!

Postby wangyeee » Sun Apr 10, 2016 5:28 am

Hi Max,

I think I've got this issue resolved by the following changes:
1. Correct run task stack size as you mentioned.
2. Increase stack size of dimu and filer task. The gcc version of printf/scanf seem to consume more resources than their CW counterpart.
3. Force 8 byte alignment for all struct placed in CCM.
Code: Select all
typedef struct {
    OS_TID task;
    ...
    uint8_t calibReadWriteFlag;  // 0=no request, 1=read request, 2=write request
} __attribute__ ((aligned (8))) dImuStruct_t;


For eeprom, I could save/load params w/o any problems. The values restored are verified on QGC as well.
For SD card, it's also working with one minor issue as the following screen-shot shows:
aq_params.PNG
aq_params.PNG (41.03 KiB) Viewed 2263 times

On the left is params stored before I force aligned all structs, as you can see, the contents are nonsense. On the right is params stored in SD after the force alignment. The values make sense now. But these values are all in scientific notation. They look quite ugly comparing to those generated by CW compiled firmware.

To summarize, there are not only differences between how gcc and cw links objs, the libc with those tool chains are different too. The cw printf/scanf seem to be smarter that gcc ones.

Regards,
Ye
wangyeee
 
Posts: 37
Joined: Sun Feb 22, 2015 5:20 am

Re: Building with GCC - success!

Postby Max » Sun Apr 10, 2016 7:28 pm

Hi Ye,

Nice work fixing it.

wangyeee wrote:Increase stack size of dimu and filer task. The gcc version of printf/scanf seem to consume more resources than their CW counterpart.


How much bigger? I'm guessing supervisor and/or comm stacks take a hit also? Eg. when requesting params read/write via mavlink or stick commands.

I know we moved the IMU calibration read/write at some point so it always runs in DIMU task/stack. Maybe we could/should do similar with all the other params... eg. only read/write them from supervisor task (which already has more stack space allocated than it needs).

wangyeee wrote:Force 8 byte alignment for all struct placed in CCM.


I guess I'm not entirely sure of all the implications of doing this... though it does seem like an easy fix. What, if anything, does this do to the size of these structures in RAM? If the widest member of a struct is 4 bytes (pointers/floats/32b ints only, no doubles), can we use 4-byte alignment or does that not make any difference?

wangyeee wrote:On the left is params stored before I force aligned all structs, as you can see, the contents are nonsense. On the right is params stored in SD after the force alignment. The values make sense now. But these values are all in scientific notation. They look quite ugly comparing to those generated by CW compiled firmware.


Yep, on the left is exactly what it looks like when broken. The right side looks correct, and the same as CW version makes for me. What other output are you looking at for comparison? Some older firmwares (with the printf workaround) had "prettier" output but also chopped the precision down dramatically. It's ugly for humans, but precise... we're using some of the floats as bitfields (24b max.) so we really need that precision.

Code: Select all
sprintf(buf, "%-17s\t\t%.10e\n", configParamMeta[n].name, (double)configGetParamValueForSave(n));


So the output you're getting looks correct to me.

wangyeee wrote:To summarize, there are not only differences between how gcc and cw links objs, the libc with those tool chains are different too. The cw printf/scanf seem to be smarter that gcc ones.


I would be mildly surprised if this is the only big difference. But yea, clearly they do something custom for printf/scanf, as evidenced by these funky symbol defs in their loader flags... w/out those, formatting floats, doubles, and 64b ints doesn't work at all.

Code: Select all
-defsym=__vfprintf=__vfprintf_double_long_long -u__vfprintf_double_long_long -defsym=__vfscanf=__vfscanf_double_long_long -u__vfscanf_double_long_long


I seem to recall that GCC-ARM also had some way to deal with this (exclude the "long" versions if one doesn't need them, since they take up flash space), but I don't remember the details now.

BTW, here's a good article on struct packing in general, in case you haven't seen it (or for anyone else curious): http://www.catb.org/esr/structure-packing/

Cheers,
-Max
Max
 
Posts: 2814
Joined: Mon Aug 13, 2012 9:45 pm
Location: Near Ithaca, NY, USA

Re: Building with GCC - success!

Postby Max » Sun Apr 10, 2016 8:33 pm

PS. Regarding formatting of params for storage, now that we can define the actual value types we want, they could be formatted "pretty" based on the type, instead of assuming all float values and worrying about preserving bitfields properly. The longer-term goal is to store the param values in a union (I think) and be able to retrieve the exact data type w/out casting (implicit or otherwise).

-Max
Max
 
Posts: 2814
Joined: Mon Aug 13, 2012 9:45 pm
Location: Near Ithaca, NY, USA

Re: Building with GCC - success!

Postby rput » Sun Apr 10, 2016 9:32 pm

No expert on this, but can you do something with pragma?

http://infocenter.arm.com/help/index.js ... FEEDG.html
rput
 
Posts: 83
Joined: Sun Feb 10, 2013 5:45 pm

PreviousNext

Return to AQ Firmware

Who is online

Users browsing this forum: No registered users and 4 guests

cron