It seems the flash programming algorithm is a piece of code that is downloaded into the SRAM and executes from there.
But how does it get started if there is nothing yet in the vector table at 0? If the flash is erased then everything in the vector table at zero will be 0xFFFFFFFF, so how does it start executing the algorithm in RAM? (which is at 0x20000000 for example)
I want to know because I want to be able to boot from RAM as well.
Starting a device via JTAG is a messy process. If the processor is starting from a blank flash it's going to fault very quickly. Code can be injected into RAM, PC/SP set up, breakpoints/watchpoints set, it doesn't need to be reset again. JTAG can modify the internal state of the processor.
Most devices have an Internal ROM which can be mapped at zero.
-> Mine doesn't have that BOOT0/1 feature <-
What M3 MCU do you use?
It's an M3 embedded in a much larger ASIC, not a stand-alone MCU chip