[GIC500v3][ARMv8][A72] Interrupt is not handled correctly after A72 change from EL3 to EL1

Dealing with the SGI question first.

When we're talking about SGIs, you need to consider two things:

• How has the target CPU has configured the INTID (GICR_IGROUR0 and GICR_IGRPMODR0)
• How is the sending CPU generating the interrupt (ICC_SGIxR_EL1/ICC_ASGI1_EL1)

There's a table (plus explanation) on this page: https://developer.arm.com/documentation/198123/0302/Sending-and-receiving-Software-Generated-Interrupts

But in summary - the Sender and Receiver need to agree, otherwise the interrupt is dropped.

For example:

If SW at EL3 send INTID 0 using ICC_SGI0R_EL1, it's sending the SGI as G0.  If the receiver has INTID 0 configured as NSG1 or SG0, the interrupt will be dropped because of the mismatch between the sender and receiver.

The most likely explanation for what you're seeing is that you have different values programmed for GICR_IGRPUPR0 and GICR_IGRPMODR0 for INTIDs 0:14 compared to INTID 15.

There's unfortunately not enough information in code snippet you shared to answer.  Here are some things that I would check:

• Have you set ARE_NS and ARE_S in GICD_CTLR?
• Have you set ICC_SRE_EL2.SRE? (requires that SCR_EL3.NS is set to 1)
• Have you set ICC_SRE_EL1.SRE in _both_ banked copies of ICC_SRE_EL1?  (easy point to forget)
• While in EL3, did you remember to use an ISB after changing SCR_EL3.NS? (if not, you might not be accessing the banked copy of a register you think you are)
• Did you set the NS.G1 enable in the CPU? (remember ICC_IGRPEN1_EL1 is banked)
• Did you set the NS.G1 enable in the Distributor?
• Before leaving EL3, did you program ICC_PMR_EL1 to a value in the NS range?
• Before leaving EL3, did you use GICx_IGROUPRn/GICx_IGRPMODn to make the appropriate interrupts NS.G1?
• Before leaving EL3, did you do the GICR_WAKER sequence?  (in your case "yes", as you've been getting interrupts in EL3)
• Did you remember to clear PSTATE.I/F for the EL/Security state being entered?  (I always kick myself if I've forgotten)

Assuming all of that checks out, when you think you should be getting an interrupt:

• Do you see the interrupt as pending in GICx_ISPENDRn? (if not, there's no interrupt to be sent)
• Do you see the interrupt as NOT active in GICx_ISACTIVERn? (if the interrupt is Active, or Active&Pending, it won't be signalled)
• Have you checked the individual enable, group, priority and target configuration?
• What do ICC_HPPIR0_EL1 and ICC_HPPIR1_EL1 report? (these will tell you if the interrupt was forwarded to the CPU and aren't effected the PMR or RPR)
• What does ICC_RPR_EL1 report?  (if not "idle", then a new interrupt will only be signalled if it meets the pre-emption rules)
• If the interrupt is reported in one of the HPPIs, will its priority beat the PMR? (remember, must be numerically lower than PMR)

Also... you probably don't want SCR_EL3.FIQ cleared while in Non-secure state.  In GICv3, FIQ is used for G0 interrupts and interrupts for the "other" Security state.  Meaning in NS state, FIQs will either be G0 interrupts or SG1 interrupts - neither of which you can really manage in Non-secure state.

Kael Hong said:

• I see SGI can be generated by either ICC_SGIxR_EL1/ICC_ASGI1_EL1 or GICD_SGIR. 
• What's difference between the SGI that generated by different register? 

GICD_SGIR is used in legacy operation only (when GICD_CTLR.ARE=0 and ICC_SRE_ELx.SRE=0).

ICC_SGI0/ICC_SGR1/ICC_SSG1 let you control what you send:

• ICC_SGIR0 - Send a G0
  • Typically used either for IPIs within the EL3 firmware, or to request EL3 firmware does something.
• ICC_SGIR1 - Send G1 for current Security state
  • Typically used as IPIs within the host OS/hypervisor.
• ICC_ASGI1 - Send G1 for the other/non-current Security state
  • Typically used as part of a calling convention between Secure and Non-secure worlds.

I say "typically" in the examples because they are just examples.  You can use them for whatever you want within the architecture rules.

Kael Hong said:

• Are they distributed by different SS? distributor and redistributor? 
• Are they generated and handled at different EL? 

The register used matters to decide if the interrupt will be set pending or not.  That's what I wrote before about comparing what the Sender sent to how the target CPU has the interrupt configured.  Beyond deciding whether interrupt is set pending or not, the register used for the SGI does not matter.

Kael Hong said:

• I thought that before, NS/S Group1 SGI is not allowed to be handled at EL3, and is expected to be handled at other EL.
• But seems this not the truth. right? It's better to not have them being handled at EL3, but actually they can be. right?

That's right.

What the manual is describing is the common/typical usage.  If you want to handle everything at EL3 you can - and in test or bring up software that might be useful. 

Kael Hong said:

SGI generated by CPU interface is distributed by redistributor, and SGI generated by distributor is distributed by distributor? 

The architecture doesn't specify how a GIC is built, only how it behaves/functions.  And I don't know the GIC-500 well enough to comment on how precisely it works.

But from a pure architecture perspective: A Redistributor only talks to a single CPU interface.  It's the Distributor which links all the Redistributors together.  Therefore, the notional path for an SGI in GICv3 is:

Source CPUIF --> Source Redistributor --> Distributor --> Target Redistributor(s) --> Target CPU interface(s)

In GICv1/2 it was:

Distributor --> [GICv3 legacy only -Target Redistributor(s)] --> Target CPU interface(s)

But, that's a little misleading.  In GICv3, SW writes to a local system register.  In GICv1/2 (or GICv3 legacy), GICD_SGIR is an MMIO register.  Therefore, although the distribution path in the GIC looks smaller in GICv1/2, i'd expect the GICv3 approach to better in most cases.  As MMIO is typically slow.