This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

GICv2 and GICv3 -- priority drop racing with deactivation through an LR

Consider a hypervisor using the GIC's help to route physical interrupts to virtual machines as described in section 6.3.1 of the GICv3's architecture specification, under "Usage model for associating a virtual interrupt with a physical interrupt": the hypervisor sets ICC_CTLR_EL1.EOImode to 1 and associates physical and virtual interrupts by setting ICH_LR*_EL2.HW to 1.

If the hypervisor runs on multiple PEs, it strives to have each physical interrupt target the PE running the virtual machine to which the interrupt is routed. This is not a guarantee, in particular because a multicore VM may dynamically change the virtual PEs targeted by its virtual interrupts, and despite the hypervisor's best efforts to retarget the physical interrupts accordingly, the hardware may react with a little delay. In short, the hypervisor may occasionally ack a physical interrupt P on a PE A, and decide to route it to a virtual interrupt V in some VM running on another PE B.

In this situation, the priority drop and deactivation of P race with each other. The following scenario is possible:

  1. On A, the hypervisor acks P.
  2. On A, the hypervisor decides to inject V associated to P in the VM running on B. By some inter-PE communication mechanism, it communicates its decision to the hypervisor thread running on B.
  3. On B, the hypervisor picks up the message and programs an ICH_LR*_EL2 accordingly: HW == 1, pINTID == P, vINTID == V.
  4. On B, the hypervisor switches to EL1 to run the VM in which it has just injected V.
  5. On B, the VM handles V, thus prompting the deactivation of V and P.
  6. On A, the hypervisor finishes its interrupt handler for P by writing to ICC_EOIR*_EL1.

As a result, P is deactivated before the priority drop at the end of interrupt.

Although neither of the GICv2 and GICv3 specifications explicitly mention this scenario, they discourage deactivation by a write to (G)ICC_DIR(_EL1) before priority drop by a write to (G)ICC_EOIR*(_EL1). GICv2 states in section 3.2.1 (emphasis mine):

[I]nterrupt handling software must:

  1. Perform a valid EOIR write, to cause priority drop on the GIC CPU interface.
  2. Subsequently, write to the GICC_DIR, to deactivate the interrupt.

GICv3 appears conflicted on this topic. On the one hand, it states in section 4.1.1, under "Deactivation" (emphasis mine):

The priority drop and interrupt deactivation are separated when ICC_CTLR_EL1.EOImode or ICC_CTLR_EL3.EOImode_EL3 in the CPU interface is 1, and the PE writes to ICC_EOIR0_EL1 or ICC_EOIR1_EL1. In this case:

  • The priority drop happens when the PE writes to ICC_EOIR0_EL1 or ICC_EOIR1_EL1.
  • Interrupt deactivation happens later, when the PE writes to ICC_DIR_EL1. A valid write to ICC_DIR_EL1 results in interrupt deactivation for a Group 0 or a Group 1 interrupt.

[...]

If software writes to ICC_DIR_EL1 when the following conditions are true, the results are UNPREDICTABLE:

[...]

  • The ICC_CTLR_EL1.EOImode or ICC_CTLR_EL3.EOIMode_EL3 is set to 1 and there has not been a corresponding write to ICC_EOIR0_EL1 or ICC_EOIR1_EL1.

which agrees with GICv2.

On the other hand, it states in section 12.2.8, under "Accessing ICC_DIR_EL1" (emphasis mine):

There are two cases when writing to ICC_DIR_EL1 that were UNPREDICTABLE for a corresponding GICv2 write to GICC_DIR:

[...]

  • When EOImode == 1 but no EOI has been issued. The interrupt will be de-activated by the Distributor, however the active priority in the CPU interface for the interrupt will remain set (because no EOI was issued).

which vaguely suggests that, in contrast to GICv2, software may write to ICC_DIR_EL1 and ICC_EOIR*_EL1 in any order.

This gives rise to the following questions:

  • Does GICv3 follow GICv2 or does it allow the write to ICC_DIR_EL1 to occur before the write to ICC_EOIR*_EL1?
  • Do GICv2 and GICv3 support the automatic deactivation of a physical interrupt through a (G)ICH_LR(_EL2) with HW == 1 before the priority drop?