Booting sama5d2 with lpddr2

I'm having a SAMA5D26C with a LPDDR2 EDB1332BDBH-1. The CPU boots successfully ROMBOOT and is able to load at91bootstrap from QSPI memory. I get debug output from at91bootstrap, but I fail to load linux.

Investigating this further show me that the RAM initalization works fine but the CPU hangs when trying to load the device tree from flash into RAM.

So I add this loop:

volatile char * ptr = (char *)0x20000000;
while(1) {
    dbg_info("ptr: %x value: %c\n", ptr, *ptr);
    ptr += 1;
}

That gives me output as expected:

ptr: 0x20000000 value: U
ptr: 0x20000001 value: U
...

Up until:

...
ptr: 0x203ffffe value: U
ptr: 0x203fffff value:
ptr: 0x203fffff value:
ptr: 0x203fffff value:
ptr: 0x203fffff value:
ptr: 0x203fffff value:
ptr: 0x203fffff value:

It seems as I can't access memory above 4MB even though my RAM size is 128 MB. I've verified that I can write and read the written value from RAM for a value below 4MB.

Why isn't all physical memory accessible and and how come ptr isn't increased after 0x203ffffff?