I am now writing a hyper-visor based on arm-v8 aarch64. and now Linux kernel can boot successfully on this hyper-visor. and I test the code on DS5. But there is a cache issue.
The issue is:
after Linux kernel modify the content of translation table (such as called io_remap to map some device memory), it need to flush the related cache, otherwise when accessing the memory space which mapped by io_remap will cause an EL1 data abort.
I found in the DS5 cache view, when executed in EL1， only the content of translation table are cached, but other EL1 memory are not. so question is:
1: what the EL1 cache state when EL2 is implemented?
2: why it is need do cache flush after change the translation table ? what am I missing ?
the issue seems that CPU think the cache is disabled in EL1, but the translation table content is cached, so when do translation walk, cpu will not fetch content from the cache.
Thanks
Thanks Zhefei's reply.
I have found the root cause of my issue. it is because that HCR_EL2.ID and HCR_EL2.CD bit does not set correctly. after set these two bits to 1. the issue is fix and Kernel can run correctly on my heper-visor.
so i summarize this issue:
1: HCR_EL2.ID HCR_EL2.CD bits are used to enable stage 2 translations cache
2: even the stage 2 translations cache not enabled (HCR_EL2.ID HCR_EL2.CD are all 0), the translation table for stage 1 also will be cached.
Am i right. thanks
Hi lemin,
Did you find this issue when run in guest VM mode for stage2 translation?
For cacheability for stage1 and stage2 you can ref
https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf
page D4-2181: Combining the stage 1 and stage 2 cacheability attributes for Normal memory
So if you disable HCR_EL2.ID HCR_EL2.CD(are all set to 1), it will not cache stage1.
lemin said:after Linux kernel modify the content of translation table (such as called io_remap to map some device memory), it need to flush the related cache, otherwise when accessing the memory space which mapped by io_remap will cause an EL1 data abort.
On armv8 Linux kernel ioremap does not need to flush the cache after the translation table modification, hardware will keep the cache coherency between cpu cores, instead you need invalidate tlb entry for translation table. I guess "the translation table for stage 1 also will be cached" means tlb?
Could you give some details about your hypervisor.
sorry I have a type error on my last message. just correct it:
2: even the stage 2 translations cache not enabled (HCR_EL2.ID HCR_EL2.CD are all 1), the translation table for stage 1 also will be cached.
另外我英语不大好，我用中文解释下：
上个星期我描述的问题是这样子的：
就是当linux kernel修改页表之后，比如说调用io_remap之后，需要手动刷下cache，之后才能访问io_remap的空间。理论上手动刷cache这一步是可以不用的。
后来经过我debug发现，由于我HCR_EL2.ID和HCR_EL2.CD都设置成了1，导致stage 2 translation 被disable 了。虽然stage2 translation 被disable了，但从cache内容来看，整个EL1的页表还是被cache了（就是linux的页表），所以就会出现下面这个状况，导致需要手动刷cache:
------> 当修改页表之后，页表的内容被cache了，没有真正写入到内存当中。这个时候kernel去访问刚刚映射的内容时，由于stage2 translation 被disable了，TLB也没有命中，这个时候kernel去内存中提取映射关系，但是页表内容还在cache中，没有写到内存，所以产生了data abort.
这个问题，在我把HCR_EL2.ID和HCR_EL2.CD 设置成0之后解决了。
所以，我的理解是：
1：即使stage 2 translation cache被disable 了。el1中的页表还是会被cache的。
请问我的理解对不对？另外，我在手册中没有看到关于页表cache行为的描述，所以想请教下各位专家。谢谢
关于我的hypervisor --- github.com/.../mVisor
更正下：
后来经过我debug发现，由于我HCR_EL2.ID和HCR_EL2.CD都设置成了1，导致stage 2 translation 的cache 被disable 了。虽然stage2 translation 的cache被disable了，但从cache内容来看，整个EL1的页表还是被cache了（就是linux的页表），所以就会出现下面这个状况，导致需要手动刷cache:
Hi Lemin,
I got your point. May I know a little details about armv8 platform, is it a juno board? or other develop board. The mvisor is a type-1 hypervisor?
How many vcpus created for the guest VM? Have you tried to create 1 cpu for guest VM.
Is the guest OS ioremap to normal memory for virtual IO devices?
Could you check the translation table in cache for this issue, is it a translation entry guest OS just created?
Don't care about the English, if you ask in English, more arm expert can give you suggestion :)
Hi Steven
May I know a little details about armv8 platform, is it a juno board? or other develop board
---------- the platform I tested is ARM FVP, and i test it on DS5
The mvisor is a type-1 hypervisor?
------------ Yes it is a type-1 hepervisor
------------- I created 4 vcpus, and do not try 1vcpu for guest VM. But this issue is also exist when only 1 vcpu is online
------------ it is remap to the real HW device. but Whether it is mapping normal memory space or device memory space, all need to flush the cache, since the translation table is cached in the cache
----------- Yes, I have checked the content is the guest OS just created. and only the translation table of the guest VM is cached, all other memory are not if I set HCR_EL2.CD to 1.
So, why the translation table will cached even if I disable the stage 2 translation cache, does it is a normal behavior ?
lemin said:So, why the translation table will cached even if I disable the stage 2 translation cache, does it is a normal behavior ?
HCR_EL2.CD
For the Non-secure EL1&0 translation regime, forces all stage 2 translations for data accesses and translation table walks to Normal memory to be Non-cacheable.
Non-cacheable only for translation table walks to Normal memory.
From KVM implementation for aarch64, HCR_EL2.ID, HCR_EL2.CD set to 0.