in C51, if I use overlap optimize, my project have problem.( routine R1 changed routine R2's auto variable). if I do not use overlap optimize. the problem gone. Is this Keil's problem? I guess so. How dose Keil do overlap optimize?
Are you sure it's a problem with the optimization, or a problem with the overlay analysis? There are a number of ways that you can wind up with a call tree that is incorrect, generally involving function pointers. If the call tree is not correct, then the assignment of auto variables to memory locations could overlap inappropriately and cause that symptom. Sometimes you have to tweak the the overlay tree manually to get the right results. The manual has a discussion of the overlay processing that you might want to study; there's also an app note. (And old discussions on this board, of course.)