Effect of changing Optimization Level

You can crank it up to 11 if you first enable linker code packing.

Is there any side effect if we use optimization level 9?

It depends on your code. Code should be written in such a way that the program runs correctly regardless of compiler optimization level. Sadly, this is not always the case. For example, if you use a simple 'for' loop to provide a precise delay, you may find that the delay depends on compiler optimization level (as well as compiler version). Another example would be the use of the keyword 'volatile'. Typically, all variables are treated as volatile at low optimization levels, but an incorrectly written program can break at higher optimization levels because compiler caches non-volatile variables in CPU registers. There can be other examples related to pointer aliasing and so on.
So the answer to your question is: it depends on your code.

One side-effect of optimisation is always to reduce "debugability" - as it gets harder to see which machine instructions relate to which source line(s).

But, before applying compiler optimisations, you need to ensure that your source is optimum in the first place; for C51, see: http://www.keil.com/support/man/docs/c51/c51_xc.htm

Also, if you're needing the highest compiler optimisations, it may be an indication that you should at least be starting to think about a re-design to give you more code space...

You can crank it up to 11 if you first enable linker code packing.

If you search the forum you should be able to find a thread that goes into some detail about optimisation levels. The conclusion seemed to be that code size generally grows above level 9.

That conclusion (especially the "in general" part) may or may not have been valid (haven't checked), but it is certainly true that the amount of "gain" from the higher optimisations does depend upon the nature of the application, and some may actually grow.

Again, there is no substitute for writing the application "optimally" in the first place...

That conclusion (especially the "in general" part) may or may not have been valid (haven't checked), but it is certainly true that the amount of "gain" from the higher optimisations does depend upon the nature of the application, and some may actually grow.

I've tested this on a few projects and found it to be the case in all those I tried, I'm pretty sure one or two others found similar results. I agree it is hardly a statistically conclusive test though.

Again, there is no substitute for writing the application "optimally" in the first place...

I agree that if code size or speed are important it is worth making the effort from the beginning where it is obvious that a benefit can be achieved, but if not the focus should be on clarity and maintainability. Unfortunately knowing whether micro optimisations are likely to be of benefit or even necessary is something that only comes with experience of the specific platform and toolchain. General rules such as 'you shouldn't use floating point in an embedded application' are daft - if there is no negative impact on the project then it makes sense to use it if it simplifies coding or reduces development time. There is the additional proviso with that particular example that you need to understand the sources of potential error, of course.

It is a bit strange to have 11 different optimization levels. That somehow gives an indication that each optimization has a natural order or at least cost in relation to every other optimization.

Most compilers settles for maybe two or three optimization levels (little, medium and much or maybe fast, medium and expensive). This then complemented with another switch to inform the compiler if it should prioritize speed or size. And finally - for the control freaks - a full set of optimization flags to allow full customization of what optimization steps to use.

One thing interesting here is the note that the total size tends to grow at optimization levels above 9, since the documentation for lvl 10 and 11 says:

10 Rearrange Code (Linker Optimization): When
detecting common block subroutines, code is rearranged
to obtain larger recurring sequences.

11 Reuse of Common Exit Code (Linker Optimization):
Identical exit sequences are reused. This may reduce
the size of common block subroutines even further.
This optimization level generates the most compact
program code possible.

They are performed by the linker, so it would have been reasonable to think that the linker only performed these steps for the cases where the linker sees an actual benefit.

It is a bit strange to have 11 different optimization levels.

I suspect the marketing people were involved:

"Most compilers only have three optimisation levels. The XXXX toolchain offers eleven providing a near fourfold increase in code efficiency over lesser systems".