Within Arm, we work on many different compilers and compiler tools. Regardless of the subject language, all compilers are all similar in operation. They take a program written in a given programming language as input. A static compiler will compiler the input to a binary which can be run directly on the target machine. Other compilers will compile it to an intermediate binary. A runtime provided by the compiler can be used to run intermediate binaries by Just In Time (JIT) compiling method by method to binary format.
When adding support to a compiler, it is important to ensure compiled code produces the correct results, even if the intermediate steps have completely changed. However, compilers are very complicated tools which is why good development testing is invaluable. Ideally the testing tools provided by a compiler quickly test as many cases as possible. The quality of these tools can greatly aid or hinder development.
Consider these examples:
main
Good development tooling can help to mitigate these issues.
For the past year or so in Arm, we have had a small team working on .NET’s code generation component known as RyuJIT. We have written several optimizations, all of which required lots of debugging to get the features functionally correct. In this blog post, I am going to highlight some of the tools available to the developer and how they have aided our work. Everything in this blog is my own personal opinion.
All the code and dumps generated below are produced using a modified version of coreclr from the .NET runtime repository. This has been slightly modified to allow for certain optimizations to be turned off through new configurations options. The code is available on GitHub. The SHA of the HEAD is:
SHA
HEAD
182bee3189ebb38511e3963595bdb2234a310b09
All the examples are produced on an Arm64 box running Linux. However, they should be reproducible on any .NET target but may require minor changes to the command line (for example, to replace any direct references to Arm64).
If you want to follow along, then install the standard build dependencies, do a standard checked build and generate a CORE_ROOT folder:
./build.sh -rc Checked -lc Release -s clr+libs ./src/tests/build.sh generatelayoutonly Checked
Before running SPMI asmdiffs (see below), you need the jitutils tools built. Clone them from here, ensure there is a release version of .NET in your path, then build with:
$ cd jitutils $ ./build.sh -p $ export DOTNET_ROOT=<path to system installed .NET> $ export PATH=$PATH:$(pwd)/bin
The final 2 steps are needed to run the jitutils tools.
For most VM based languages, configuration options are passed to the runtime through command line arguments. With .NET, options are instead passed through environmental arguments. For example:
$ export DOTNET_TieredCompliation=0 $ export DOTNET_JitDisasm=myclass.my*method $ export DOTNET_StressRegs=0x80 $ dotnet ./helloworld.dll
This makes it much easier to instrument an existing setup. For example, if an app is launched from inside a series of bash scripts, then to pass command line options to the app the scripts need modifying. This can be done through hard coding the options at the point the app is called, or adding extra command line options into the script which are then passed down to the app. These changes must be made for every script, and every time the runtime is called within those scripts.
With .NET, the env variables can be set directly before calling the script. This is much easier and reduces script bloat.
The downside to this approach is it is possible to unintentionally leave environment variables turned on. I have accidentally filled a multiple TB disk after leaving on a dump variables. Also, a script may clear the environment, but this rarely happens in practice. These issues are preferable than manually editing scripts, but your milage may vary.
.NET can provide long detailed dumps of what is going on inside RyuJIT. This is enabled by an environment variable set to a regex on the name of each compiled method, and is streamed directly to standard output:
$ export DOTNET_JitDump=myclass.mymethod $ export DOTNET_JitDump=my*class.* $ export DOTNET_JitDump=my*
Or, the hash code for a method can be used (shown at the top of an IR dump):
$ export DOTNET_JitHashDump=0xdb9e6798
To get only the final disassembly of that method, then:
$ export DOTNET_JitDisasm=class.method
The complete dump is useful for seeing what is going on inside the compiler. To really understand what is happening, the dumps are essential as they contain thousands of lines per method, detailing all the transformations RyuJIT performs. At each phase it includes ASCII formatted IR graphs for easy visualization of the current syntax tree that is being operated on. The blog post I wrote about If conversion contains some detailed analysis the output of some of these phases.
The regex allows you to match multiple methods, but I do not recommend against matching too many at once. If multiple methods are compiled at the same time (due to multithreading) then the output becomes interleaved and unreadable. My suggestion here would be for .NET to add the option to save the dumps in individual files based on the method name (maybe 1 day I will write a patch...). In practice this has not been an issue.
.NET has its own testsuite inside src/tests, containing thousands of handwritten files, each containing multiple cases testing for a single condition. As you would expect for a software project, each new piece of functionality or bug fix usually includes several new tests. These tests are crucial in ensuring the stability of .NET and preventing regressions. However, there are 2 main issues with relying on this testsuite. Firstly, despite the best intentions of a developer, there may be cases which are missed. This will be addressed in the fuzzers section later. Secondly, the suite by itself does not address the debugging issues listed at the start of the blog. Once an error occurs, it can be hard to find the exact issue or even failing method. SuperPMI is a tool which aims to address this.
SuperPMI, or sometimes SPMI, can capture all the methods compiled by .NET across an entire run of a program or suite of programs, and then allow a developer to quickly replay all of the compilations. SuperPMI is my personal favorite of all the .NET developer tools as it enables checking the impact of code changes across entire test suites. To do this, it comes in 3 modes.
This mode is used to capture all the compiled methods. Due to the integration of this into the .NET CI, it means that developers rarely run this stage directly.
To run in collect mode, pass a command string to run a .NET program or a test scenario. For example:
$ python ./src/coreclr/scripts/superpmi.py collect “run_my_dotnet_testsuite.sh”
First the script sets some environment variables – namely, a directory to dump files to and the location of the superpmi-shim-collector. For example:
SuperPMIShimLogPath=/tmp/spmi/ SuperPMIShimPath=/home/alahay01/runtime/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so DOTNET_JitName=superpmi-shim-collector.dll
This allows SPMI to hook into a .NET mechanism. Normally when .NET is run, all compilation is handled by RyuJIT which lives inside the library libclrjit. However .NET allows for the compiler to be switched to another, through DOTNET_JitName, as long as it provides the exact same interfaces.
SPMI collect runs the command string specified on the command line. Whenever .NET is invoked, all method compilations will instead be sent to the SPMI shim collector. This is a thin wrapper which will:
This results in a single method context file (with extension .mc) per compiled method. From .NET’s point of view there is no difference in the execution of the .NET application.
Next SPMI collates all the method context files from the log directory into a single collection, referred to as a method context hive file (with extension .mch). If .NET was run multiple times during the test execution (for example, if running the entire .NET testsuite), then there will be lots of duplicate sets of methods. Every invocation is going to run through the same steps prior to main. Most programs are going to end up calling many of the same utility methods. SPMI finishes by removing all duplicate methods with identical meta data from the file.
The upstream CI system does a regular SPMI collection of:
Each of these produces collection files of a few Gb which are saved to a known downloadable location containing the SHA of the current git HEAD.
Replay mode can be invoked very easily:
$ python ./src/coreclr/scripts/superpmi.py replay
First, SPMI checks the git ref the repository was based off, then downloads all the collections related to either the current commit (or, if it has diverged from upstream, then it uses the last forked point). Note that this can take a while to download, but it will be cached for future runs.
SPMI now runs through each method in each collection and recompiles each 1 in turn, spreading out across as many cores that are available. The stored method code and meta data is sufficient so that when passed through to the same RyuJIT it exactly recreates the original compilation.
This is a very powerful way of quickly compiling every single method within the main testsuites without having to run the suite. Depending on hardware, it can take just under 2 minutes to run through the all the downloaded collections. This speed is partially due to the removal of duplicate method contexts, but mostly with not having to execute any of the resulting code or the VM.
If at any point a compilation fails, then this method is saved to a failure directory in an individual file with a unique id. Each failure can then be replayed:
$ superpmi ./spmi/failures/12345.mch
Or even directly from the original collection with the unique method id:
$ superpmi -c 12345 ./spmi/downloads/testsuite.mch
SPMI replay is purely about testing whether the compile succeeds: Segfaults, early exits and asserts are the cases being tested for. It does not care about the final code produced by the compiler, only whether the compiler completed successfully. It cannot be used this to test contents of the compiled code as there are no checks to confirm the method compiles to valid code.
This is where asserts become extra valuable. Not sure about something? Or just want to guarantee a condition that is being assumed? Then just add an assert in the RyuJIT codebase. Sprinkle liberally throughout your code. Once .NET gets compiled in release mode these will all vanish and not incur any costs.
This supercharges development in 2 major ways.
The first is aiding your main development loop. Write a patch, add asserts everywhere, write short test cases, and get the code working with those test cases. Now run SPMI replay, find the smallest failure file (as this is usually the shortest test case), debug it, fix it, then rerun SPMI, repeating until there are no more failures.
Second is the nightmare case: your code causes .NET runtime to fail before reaching the test case. Simply throw it at SPMI replay and debug the failures.
The key to both loops is that when replaying a SPMI failure, it compiles a single method. This may sound not a big deal for developers of static languages (for example, GCC and LLVM developers), but for JIT based languages this is huge. In a jitted language, in order to compile a single method, the entire program must be run in the VM. The user must then find the correct settings in order to get debug information for just the failing method. This is often a messy and frustrating process. Plus, you must then find the exact method that failed, not just the method name, but often the types and parameters it was called with. SPMI removes all this process. In addition, SPMI, allows all the standard coreclr environment variables, and can be run easily through a debugger (such as lldb/gdb). The following is often all that is needed to debug a failure:
$ export DOTNET_JITDUMP=* $ lldb -– superpmi -c 12345 ./spmi/downloads/testsuite.mch
This is shown in action using the example repo.
$ python ./src/coreclr/scripts/superpmi.py replay [15:45:25] ================ Logging to /home/alahay01/dotnet/runtime_bools/artifacts/spmi/superpmi.3.log [15:45:25] Using JIT/EE Version from jiteeversionguid.h: 4bceb905-d550-4a5d-b1eb-276fff68d183 [15:45:25] Using libclrjit.so from Core_Root: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so [15:45:25] Found download cache directory "/home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64" and --force_download not set; skipping download [15:45:25] SuperPMI replay [15:45:25] ------------------------------------------------------------ [15:45:25] Start time: 15:45:25 [15:45:25] JIT Path: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so [15:45:25] Using MCH files: [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries_tests.pmi.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.crossgen2.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_tiered.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.pmi.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_pgo.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/realworld.run.linux.arm64.checked.mch [15:45:25] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/coreclr_tests.run.linux.arm64.checked.mch [15:45:25] Using superpmi from Core_Root: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi [15:45:25] [15:45:25] Temp Location: /tmp/tmp6qmalivp [15:45:25] [15:45:25] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [15:45:25] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/benchmarks.run.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/benchmarks.run.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [15:45:26] Clean SuperPMI replay (32525 contexts processed) [15:45:26] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries_tests.pmi.linux.arm64.checked.mch [15:45:26] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/libraries_tests.pmi.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/libraries_tests.pmi.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries_tests.pmi.linux.arm64.checked.mch [15:45:38] Clean SuperPMI replay (323521 contexts processed) [15:45:38] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.crossgen2.linux.arm64.checked.mch [15:45:38] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/libraries.crossgen2.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/libraries.crossgen2.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.crossgen2.linux.arm64.checked.mch [15:45:43] Clean SuperPMI replay (263312 contexts processed) [15:45:43] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_tiered.linux.arm64.checked.mch [15:45:43] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/benchmarks.run_tiered.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/benchmarks.run_tiered.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_tiered.linux.arm64.checked.mch [15:45:44] Clean SuperPMI replay (58057 contexts processed) [15:45:44] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.pmi.linux.arm64.checked.mch [15:45:44] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/libraries.pmi.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/libraries.pmi.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.pmi.linux.arm64.checked.mch [15:45:51] Clean SuperPMI replay (290559 contexts processed) [15:45:51] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_pgo.linux.arm64.checked.mch [15:45:51] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/benchmarks.run_pgo.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/benchmarks.run_pgo.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_pgo.linux.arm64.checked.mch [15:45:55] Clean SuperPMI replay (146282 contexts processed) [15:45:55] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/realworld.run.linux.arm64.checked.mch [15:45:55] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/realworld.run.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/realworld.run.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/realworld.run.linux.arm64.checked.mch [15:45:57] Clean SuperPMI replay (35342 contexts processed) [15:45:57] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/coreclr_tests.run.linux.arm64.checked.mch [15:45:57] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmp6qmalivp/repro -p -f /tmp/tmp6qmalivp/coreclr_tests.run.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmp6qmalivp/coreclr_tests.run.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/coreclr_tests.run.linux.arm64.checked.mch [15:46:54] SuperPMI encountered missing data for 8 out of 609373 contexts [15:46:54] Replay summary: [15:46:54] All replays clean [15:46:54] Finish time: 15:46:54 [15:46:54] Elapsed time: 0:01:29.420463
A failing case has more value for this example. The example repo has a special configuration option which will cause RyuJIT to assert when compiling a bit cast statement. Enable this and rerun:
$ export DOTNET_JitFailLoweringBitCast=1 $ python ./src/coreclr/scripts/superpmi.py replay [08:34:12] ================ Logging to /home/alahay01/dotnet/runtime_bools/artifacts/spmi/superpmi.4.log [08:34:12] Using JIT/EE Version from jiteeversionguid.h: 4bceb905-d550-4a5d-b1eb-276fff68d183 [08:34:12] Using libclrjit.so from Core_Root: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so [08:34:12] Found download cache directory "/home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64" and --force_download not set; skipping download [08:34:12] SuperPMI replay [08:34:12] ------------------------------------------------------------ [08:34:12] Start time: 08:34:12 [08:34:12] JIT Path: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so [08:34:12] Using MCH files: [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries_tests.pmi.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.crossgen2.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_tiered.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.pmi.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_pgo.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/realworld.run.linux.arm64.checked.mch [08:34:12] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/coreclr_tests.run.linux.arm64.checked.mch [08:34:12] Using superpmi from Core_Root: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi [08:34:12] [08:34:12] Temp Location: /tmp/tmpvavhbaqk [08:34:12] [08:34:12] Running SuperPMI replay of /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [08:34:12] Invoking: /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi -v ewi -r /tmp/tmpvavhbaqk/repro -p -f /tmp/tmpvavhbaqk/benchmarks.run.linux.arm64.checked.mch_fail.mcl -metricsSummary /tmp/tmpvavhbaqk/benchmarks.run.linux.arm64.checked.mch_metrics.csv /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 16193 of size 435 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 23810 of size 336 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1732 of size 359 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 17220 of size 171 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 29957 of size 138 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 6797 of size 63 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 25741 of size 187 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 6798 of size 63 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 11600 of size 101 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 27664 of size 496 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 22038 of size 458 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 26135 of size 68 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 8408 of size 174 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 22232 of size 628 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 857 of size 378 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 23769 of size 658 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 7130 of size 459 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ISSUE: <ASSERT> #16193 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'BenchmarkDotNet.Engines.RunResults+<GetWorkloadResultMeasurements>d__18:MoveNext():bool:this' during 'Lowering nodeinfo' (IL size 435; hash 0x1d57393c; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #23810 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Single:SinPi(float):float' during 'Lowering nodeinfo' (IL size 336; hash 0x26bee518; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1732 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Single:CosPi(float):float' during 'Lowering nodeinfo' (IL size 359; hash 0x7eae59f3; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #17220 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Math:ScaleB(double,int):double' during 'Lowering nodeinfo' (IL size 171; hash 0x8e4c327a; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #29957 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.MathF:IEEERemainder(float,float):float' during 'Lowering nodeinfo' (IL size 138; hash 0x40f07886; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #6797 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'Newtonsoft.Json.JsonConvert:EnsureDecimalPlace(double,System.String):System.String' during 'Lowering nodeinfo' (IL size 63; hash 0xd85183a7; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #25741 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.MathBenchmarks.Double:MaxTest()' during 'Lowering nodeinfo' (IL size 187; hash 0x58e484f8; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #6798 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'Newtonsoft.Json.JsonConvert:EnsureFloatFormat(double,System.String,int,ushort,bool):System.String' during 'Lowering nodeinfo' (IL size 63; hash 0x93e5fc32; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #11600 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.MathF:ILogB(float):int' during 'Lowering nodeinfo' (IL size 101; hash 0xeb1a7455; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #27664 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Double:SinPi(double):double' during 'Lowering nodeinfo' (IL size 496; hash 0xa0c6d997; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #22038 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Reflection.Metadata.BlobWriterImpl:WriteConstant(System.Reflection.Metadata.BlobBuilder,System.Object)' during 'Lowering nodeinfo' (IL size 458; hash 0x783b8db9; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #26135 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Text.Json.Utf8JsonWriter:WriteNumberValue(double):this' during 'Lowering nodeinfo' (IL size 68; hash 0xb4bf1d24; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #8408 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number+Grisu3:TryRunSingle(float,int,byref):bool' during 'Lowering nodeinfo' (IL size 174; hash 0xf28546e5; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #22232 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Double:Hypot(double,double):double' during 'Lowering nodeinfo' (IL size 628; hash 0x2c99e8f9; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #857 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:FormatDouble[ushort](byref,double,System.ReadOnlySpan`1[ushort],System.Globalization.NumberFormatInfo):System.String' during 'Lowering nodeinfo' (IL size 378; hash 0x798faca9; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #23769 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Decimal+DecCalc:VarDecFromR8(double,byref)' during 'Lowering nodeinfo' (IL size 658; hash 0x03f5d7f2; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #7130 /home/alahay01/dotnet/runtime_boERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 6684 of size 426 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 861 of size 152 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 6685 of size 144 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 15262 of size 159 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 6688 of size 123 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1635 of size 618 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 4835 of size 410 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 12839 of size 586 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 15143 of size 173 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1064 of size 261 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 17832 of size 146 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 12650 of size 521 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 2222 of size 203 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 2862 of size 1472 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 21678 of size 318 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ols/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Net.Http.HttpConnectionPoolManager:.ctor(System.Net.Http.HttpConnectionSettings):this' during 'Lowering nodeinfo' (IL size 459; hash 0x27d9f7ec; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #6684 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:FormatSingle[ushort](byref,float,System.ReadOnlySpan`1[ushort],System.Globalization.NumberFormatInfo):System.String' during 'Lowering nodeinfo' (IL size 426; hash 0x81a3a903; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #861 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:Dragon4Double(double,int,bool,byref)' during 'Lowering nodeinfo' (IL size 152; hash 0xd8772c88; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #6685 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:Dragon4Single(float,int,bool,byref)' during 'Lowering nodeinfo' (IL size 144; hash 0x9b80fc82; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #15262 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.MathBenchmarks.Single:MaxTest()' during 'Lowering nodeinfo' (IL size 159; hash 0x57cb17f7; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #6688 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:NumberToFloat[float](byref):float' during 'Lowering nodeinfo' (IL size 123; hash 0x3e0d39c8; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1635 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Reflection.MdConstant:GetValue(System.Reflection.MetadataImport,int,System.RuntimeTypeHandle,bool):System.Object' during 'Lowering nodeinfo' (IL size 618; hash 0x8002a434; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #4835 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Single:TanPi(float):float' during 'Lowering nodeinfo' (IL size 410; hash 0x7f8a6cf7; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #12839 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Double:TanPi(double):double' during 'Lowering nodeinfo' (IL size 586; hash 0x60d166f8; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #15143 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Formats.Cbor.CborReader:ReadDouble():double:this' during 'Lowering nodeinfo' (IL size 173; hash 0x28a9f034; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1064 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Threading.ProcessorIdCache:ProcessorNumberSpeedCheck():bool' during 'Lowering nodeinfo' (IL size 261; hash 0xcbc92f02; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #17832 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Half:op_Explicit(double):System.Half' during 'Lowering nodeinfo' (IL size 146; hash 0x6b757127; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #12650 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Double:CosPi(double):double' during 'Lowering nodeinfo' (IL size 521; hash 0x30691ffc; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #2222 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Numerics.Tests.Constructor:.ctor():this' during 'Lowering nodeinfo' (IL size 203; hash 0x95938738; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #2862 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Threading.PortableThreadPool+HillClimbing:Update(int,double,int):System.ValueTuple`2[int,int]:this' during 'Lowering nodeinfo' (IL size 1472; hash 0x1578c075; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #21678 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'Microsoft.CodeAnalysis.PEModule:GetConstantValueOrThrow(System.Reflection.Metadata.ConstantHandle):Microsoft.CodeAnalysis.ConstantValue:this' during 'Lowering nodeinfo' (IL size 318; hash 0xcde605ff; FullERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 7023 of size 166 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 22319 of size 115 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1137 of size 123 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 7025 of size 131 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1138 of size 356 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 21236 of size 276 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 7032 of size 333 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 30585 of size 139 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 1021 of size 178 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] ERROR: Exception thrown: DebugBreak or AV Exception 123 [08:34:14] ERROR: Method 22910 of size 109 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). [08:34:14] Opts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #7023 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Half:op_Explicit(float):System.Half' during 'Lowering nodeinfo' (IL size 166; hash 0xe1080bc2; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #22319 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.MathF:ScaleB(float,int):float' during 'Lowering nodeinfo' (IL size 115; hash 0x37104ffc; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1137 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:NumberToFloat[double](byref):double' during 'Lowering nodeinfo' (IL size 123; hash 0x324bc468; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #7025 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Half:op_Explicit(System.Half):float' during 'Lowering nodeinfo' (IL size 131; hash 0x7795c702; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1138 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number:NumberToFloatingPointBits[double](byref):ulong' during 'Lowering nodeinfo' (IL size 356; hash 0x388813e2; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #21236 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'Microsoft.CodeAnalysis.CodeGen.ILBuilder:EmitConstantValue(Microsoft.CodeAnalysis.ConstantValue):this' during 'Lowering nodeinfo' (IL size 276; hash 0xaf0acddc; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #7032 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'BilinearTest:BilinearInterpol(double[],double[],double,double,double[],double,double,double):double[]' during 'Lowering nodeinfo' (IL size 333; hash 0x718e717c; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #30585 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Single:Hypot(float,float):float' during 'Lowering nodeinfo' (IL size 139; hash 0xbe8dfed3; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #1021 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Number+Grisu3:TryRunDouble(double,int,byref):bool' during 'Lowering nodeinfo' (IL size 178; hash 0xe6db4b8f; FullOpts) [08:34:14] [08:34:14] ISSUE: <ASSERT> #22910 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Math:ILogB(double):int' during 'Lowering nodeinfo' (IL size 109; hash 0xe1da5a76; FullOpts) [08:34:14] [08:34:14] Compilation failures [08:34:14] Method numbers with compilation failures: [08:34:14] 857 [08:34:14] 861 [08:34:14] 1021 [08:34:14] 1064 [08:34:14] 1137 [08:34:14] 1138 [08:34:14] 1635 [08:34:14] 1732 [08:34:14] 2222 [08:34:14] 2862 [08:34:14] 4835 [08:34:14] 6684 [08:34:14] 6685 [08:34:14] 6688 [08:34:14] 6797 [08:34:14] 6798 [08:34:14] 7023 [08:34:14] 7025 [08:34:14] 7032 [08:34:14] 7130 [08:34:14] 8408 [08:34:14] 11600 [08:34:14] 12650 [08:34:14] 12839 [08:34:14] 15143 [08:34:14] 15262 [08:34:14] 16193 [08:34:14] 17220 [08:34:14] 17832 [08:34:14] 21236 [08:34:14] 21678 [08:34:14] 22038 [08:34:14] 22232 [08:34:14] 22319 [08:34:14] 22910 [08:34:14] 23769 [08:34:14] 23810 [08:34:14] 25741 [08:34:14] 26135 [08:34:14] 27664 [08:34:14] 29957 [08:34:14] 30585 [08:34:14] Copying /tmp/tmpvavhbaqk/repro-7032.mc -> /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.benchmarks.run.linux.arm64.checked [08:34:14] Copying /tmp/tmpvavhbaqk/repro-26135.mc -> /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.benchmarks.run.linux.arm64.checked [08:34:14] Copying /tmp/tmpvavhbaqk/repro-17832.mc -> /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.benchmarks.run.linux.arm64.checked [08:34:14] Copying /tmp/tmpvavhbaqk/repro-1138.mc -> /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.benchmarks.run.linux.arm64.checked <SNIP> [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-123949.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-88667.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-93751.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-182641.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-207182.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-455637.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-57213.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-5947.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-488833.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-207949.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-27952.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-196983.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-231816.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-451621.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-475742.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-559144.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-33700.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-4123.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-225974.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-488186.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-444255.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-111038.mc [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-9805.mc [08:35:46] [08:35:46] To run a specific failure (replace JIT path and .mc filename as needed): [08:35:46] [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/xxxxx.mc [08:35:46] [08:35:46] Replay summary: [08:35:46] Replay failures in 8 MCH files: [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries_tests.pmi.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.crossgen2.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_tiered.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/libraries.pmi.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/benchmarks.run_pgo.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/realworld.run.linux.arm64.checked.mch [08:35:46] /home/alahay01/dotnet/runtime_bools/artifacts/spmi/mch/4bceb905-d550-4a5d-b1eb-276fff68d183.linux.arm64/coreclr_tests.run.linux.arm64.checked.mch [08:35:46] Finish time: 08:35:46 [08:35:46] Elapsed time: 0:01:33.261144
Note how SPMI gives you the exact commands to replay 1 of the failures. To debug, find the smallest failure:
$ ls -lS artifacts/spmi/repro.libraries.crossgen2.linux.arm64.checked | tail -2 -rw-r--r-- 1 alahay01 alahay01 2224 Aug 23 10:13 repro-16069.mc -rw-r--r-- 1 alahay01 alahay01 2204 Aug 23 10:13 repro-17255.mc
And rerun it:
$ /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.libraries.crossgen2.linux.arm64.checked/repro-16069.mc Using jit(/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so) with input (/home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.libraries.crossgen2.linux.arm64.checked/repro-16069.mc) indexCount=-1 () Jit startup took 2.520801ms ISSUE: <ASSERT> #1 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Single:IsNegative(float):bool' during 'Lowering nodeinfo' (IL size 10; hash 0x5274da32; FullOpts) ERROR: Exception thrown: DebugBreak or AV Exception 123 ERROR: Method 1 of size 10 failed to load and compile correctly (/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so). Loaded 1 Jitted 1 FailedCompile 1 Excluded 0 Missing 0 Total time: 5.688003ms
This can quickly be debugged by enabling dumps and running in a debugger. Use the flag --boa to enable break on assert.
$ export DOTNET_JitDump=* ❯ lldb -- /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi /home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so /home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-17255.mc -boa Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'lldb.embedded_interpreter' Error: Failed to find runtime directory (lldb) target create "/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi" Current executable set to '/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi' (aarch64). (lldb) settings set -- target.run-args "/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so" "/home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-17255.mc" "-boa" (lldb) run Process 101097 launched: '/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/superpmi' (aarch64) Using jit(/home/alahay01/dotnet/runtime_bools/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/libclrjit.so) with input (/home/alahay01/dotnet/runtime_bools/artifacts/spmi/repro.coreclr_tests.run.linux.arm64.checked/repro-17255.mc) indexCount=-1 () Jit startup took 199.123192ms ****** START compiling System.Text.Json.Serialization.Tests.ValueTests:SingleToInt32Bits(float):int (MethodHash=da4eedf3) Generating code for Unix arm64 OPTIONS: compCodeOpt = BLENDED_CODE OPTIONS: compDbgCode = false OPTIONS: compDbgInfo = true OPTIONS: compDbgEnC = false OPTIONS: compProcedureSplitting = false OPTIONS: compProcedureSplittingEH = false OPTIONS: optimizer should use profile data IL to import: IL_0000 02 ldarg.0 IL_0001 28 05 0d 00 0a call 0xA000D05 IL_0006 2a ret ....SNIP..... *************** Starting PHASE Do 'simple' lowering *************** Finishing PHASE Do 'simple' lowering [no changes] *************** Starting PHASE Lowering nodeinfo ISSUE: <ASSERT> #1 /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp (531) - Assertion failed 'false' in 'System.Text.Json.Serialization.Tests.ValueTests:SingleToInt32Bits(float):int' during 'Lowering nodeinfo' (IL size 7; hash 0xda4eedf3; FullOpts) ERROR: SuperPMI: Assert Failure (PID 101097, Thread 101097/18ae9) Assertion failed 'false' in 'System.Text.Json.Serialization.Tests.ValueTests:SingleToInt32Bits(float):int' during 'Lowering nodeinfo' (IL size 7; hash 0xda4eedf3; FullOpts) /home/alahay01/dotnet/runtime_bools/src/coreclr/jit/lower.cpp, Line: 531 superpmi was compiled with optimization - stepping may behave oddly; variables may not be available. Process 101097 stopped * thread #1, name = 'superpmi', stop reason = signal SIGTRAP frame #0: 0x0000aaaaaaba5244 superpmi`DebugBreakorAV(val=123) at spmiutil.cpp:43:13 [opt] 40 if (val == 0) 41 __debugbreak(); 42 if (BreakOnDebugBreakorAV()) -> 43 __debugbreak(); 44 } 45 46 int exception_code = EXCEPTIONCODE_DebugBreakorAV + val; (lldb) frame select 3 libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available. frame #3: 0x0000fffff708d068 libclrjit.so`Lowering::LowerNode(this=0x0000aaaaaacd0ae0, node=0x0000aaaaaaccb038) at lower.cpp:531:13 [opt] 528 #if defined(DEBUG) 529 if (JitConfig.JitFailLoweringBitCast() != 0) 530 { -> 531 assert(false); 532 } 533 #endif 534 ContainCheckBitCast(node); (lldb)
Some basic debugging shows RyuJIT erroring at the assert enabled with DOTNET_JitFailLoweringBitCast.
That is everything for part 1. In the next, and final part, I run through how you can use SPMI to quickly see changes in the generated code. Then I move onto fuzzing using Fuzzlyn.
Part 2