Problem switching from EL3 to non-secure EL1

Cortex-A35 processor, AArch64 mode. Before setting up MMU and GIC, I'm trying to go from EL3 to non-secure EL1:

        msr VTTBR_EL2, xzr
        mov x0, SCR_EL3.RES1 or SCR_EL3.NS or SCR_EL3.RW or SCR_EL3.ST
        msr SCR_EL3, x0
        mov x1, SPSR.M.AArch64_EL1h or SPSR_EL3.A or SPSR_EL3.I or SPSR_EL3.F
        msr SPSR_EL3, x1
        adr x2, __el1
        msr ELR_EL3, x2
        ; all other system registers are set to their reset values. 
        ; SCTLR_EL1 = 0x00C50838
        ; HCR_EL2   = 0x0000000000000002

        eret

__el1:
        mov x10, 0xff220000  ; this simply turns on the LED on the board, 
        mov w11, 0x0020      ; for testing only
        str w11, [x10, 4]    ;
        str w11, [x10, 0]    ;
        b.al $ 

Switching to the secure EL1 (SCR_EL3.NS not set) works fine and the LED turns on.
However, it doesn't work when I try to go to non-secure EL1.

I also tried setting HCR_EL2.RW, but it didn't work:

        mov x0, HCR_EL2.RW
        msr HCR_EL2, x0

What am I missing?