TTBR0_EL1 translation fault level 3 on 4KiB blocks where 2MiB blocks work.

I'm building kernel that is running ARMv8. I'm currently facing an issue with mapping of user processes.

TTBR1_EL1 flat-maps (identity mapping) 8GiB of memory via "PGD → PUD → 8x PMD → 512x 2MiB each" tables.
TTBR0_EL1 identity mapping works, but switching it to user process mapping fails with "translation fault, level 3". I managed to narrow it down to the size of the blocks I'm using: the user mapping works, if I use block descriptors in PMD, i.e. do a 2MiB mapping. Unfortunately, using it this way would require turning the whole allocator upside down, so I would really like to be able to use 4KiB.

The descriptors in question have the following form (modulo the PA which changes):

0x000000010000e701

My current SPSR, TCR, and MAIR setup looks as follows:

/// MAIR:
pub static MAIR_EL1: i64 = 0x0000_0000_0000_44_00ff

/// - [34:32] Set physical address range to 44 bits - this is the same as the
///           value reported by the ID_AA64MMFR0_EL1 register.
/// - [29:28] Set memory shareablitity to inner-sharable for TTBR1_EL1
/// - [31:30] Use 4KB granule size for high memory.
/// - [27:26] Outer cacheability set to W-B, R-A, W-A for TTBR1_EL1.
/// - [25:24] Inner cacheability set to W-B, R-A, W-A for TTBR1_EL1.
/// - [21:16] Size offset for high memory in bits - use 48 bits.
/// - [13:12] Set memory shareablitity to inner-sharable for TTBR0_EL1
/// - [15:14] Use 4KB granule size for low memory.
/// - [11:10] Outer cacheability set to W-B, R-A, W-A for TTBR0_EL1.
/// - [9:8]   Inner cacheability set to W-B, R-A, W-A for TTBR0_EL1.
/// - [5:0]   Size offset for low memory in bits - use 48 bits.
///
/// Used by boot.S
#[no_mangle]
pub static TCR_EL1_CONF: u64 = (0b100 << 32)
    | (0b10 << 30)
    | (0b11 << 28)
    | (0b01 << 26)
    | (0b01 << 24)
    | ((64 - 48) << 16)
    | (0b00 << 14)  // XXX: TG0 uses different values than TG1.
    | (0b11 << 12)
    | (0b01 << 10)
    | (0b01 << 8)
    | (64 - 48);

/// - [25]  Set exception endiannes at EL1 to little-endian (0)
/// - [24]  Set explicit data access at EL0 to little-endian (0)
/// - [19]  Do not mark writable regions as eXecute Never (0)
/// - [12]  Enable Instruction cache (0)
/// - [2]   Enable data cache (0)
/// - [0]   Enable MMU
/// - [...] RES1 - reserved, set to 1
#[no_mangle]
pub static SCTLR_EL1_CONF: i64 = (0 << 25)
    | (0 << 24)
    | (0 << 19)
    | (1 << 12)
    | (1 << 2)
    | (1 << 0)
    // Reserved-1 values
    | (1 << 23)
    | (1 << 22)
    | (1 << 20)
    | (1 << 11)
    | (1 << 8)
    | (1 << 7);