Simple math problem

I'm guessing this is a really basic understanding problem on my part but I am trying to set a variable as:

unsigned char randomnumber = 2^(0x03);

Now, in my head, I expect 2^3 = 8 (0x08) but the program is giving me 0x0001; I've tried manipulating it all the way down to:

unsigned char randomnumber = ((int)2)^((unsigned char)(0x03));

but all I'm getting is 0x01 rather than the previous 0x0001. I'm guessing this has to do with a fundamental misconception on how this is actually processing the data. Can anyone help me understand what it is I'm doing wrong?

Any help would be appreciated. Thanks, guys!