Cache memory needs to be invalidated before it is being used or initialized.
Kindly go through the following steps :
1. Clean and invalidate cache memory
2. NOP for certain clock cycles as specified by TRM
3. Initialize the I cache and D cache
4. ISB / DSB should be executed.
The above steps should be carried out for both L1 and L2 cache memory.