unable to use fread() function

fseek() and fread() does return results, but you never check them.
So most probably, you don't check the result of your memory allocations either.

By the way - why do you think you need to perform a seek between each read? Isn't your fread call moving the file pointer forward the required amount?

But then again - why do you do a seek specifying the value 0 when your first read is 51+3 bytes into the file?

And since you are skipping the image header - how do you even know that the image have 3 bytes for each pixel?

And tmp is a unsigned character, having size 1. So why do you try to read 3 bytes in your fread? Where do you want to place the remaining two bytes? Haven't you considered the result of reading more data than tmp can store? What may happen to other data stored directly after tmp on the stack? Would that be good? Would that even make your program work as expected?

You say the code works with GNU C. But haven't you considered what happens if a different compiler moves some variables into registers, instead of keeping them on the stack? Are C programs built using different compilers really expected to behave identically when you do buffer overflow attacks on the variables?

fseek() and fread() does return results, but you never check them.
So most probably, you don't check the result of your memory allocations either.

By the way - why do you think you need to perform a seek between each read? Isn't your fread call moving the file pointer forward the required amount?

But then again - why do you do a seek specifying the value 0 when your first read is 51+3 bytes into the file?

And since you are skipping the image header - how do you even know that the image have 3 bytes for each pixel?

And tmp is a unsigned character, having size 1. So why do you try to read 3 bytes in your fread? Where do you want to place the remaining two bytes? Haven't you considered the result of reading more data than tmp can store? What may happen to other data stored directly after tmp on the stack? Would that be good? Would that even make your program work as expected?

You say the code works with GNU C. But haven't you considered what happens if a different compiler moves some variables into registers, instead of keeping them on the stack? Are C programs built using different compilers really expected to behave identically when you do buffer overflow attacks on the variables?

So the original code you posted with "tmp" variable wasn't really real code, since tmp was a char while your _real_ tmp was RGB? Never show "almost" real code.

So what goes wrong with your allocation? Does it exit after a failure? Or does it return, but you are unhappy with contents of Matrix? This time too, you don't seem to show full code. No return type shown. And you don't show how the function hands over Matrix at the end.

Do you get an error printout?

Note that exit() isn't a perfect function to use in an embedded application, since there are no operating system to return to. There are neither a desktop nor a command line. Depending on tools, you might get the code stuck in an infinite loop somewhere. Or maybe the application restarts by calling main again. Or maybe you get a watchdog reset. exit() may not even give the processor time to print any pending error message.

Note that RGB contains 3 bytes, but what is sizeof(RGB) with your compiler?

How much memory do you need? How much memory have you set up as heap? How far do you get in the allocation before it fails? How have you verified that your height and width really have the values you think they have?

So what goes wrong with your allocation? Does it exit after a failure? Or does it return

It exits after failure.

sizeof(RGB) in my code is 3 bytes. In my application I am trying to read an image with 320 x 240 size. In my startup code heap is set as Heap_Size EQU 0x00000800 and if I try to change these values .\Obj\Memory.axf: Error: L6406W: No space in execution regions with .ANY selector matching Section

If I am taking an image with 10x8 dimensions it is working but beyond that it does n't.

How have you verified that your height and width really have the values you think they have?

Yes I have have verified. They have actual values (I mean height and width of input image)

Stop fooling around. 320*200*3 = 64000*3.

The processor don't have enough internal RAM to handle it.

The processor doesn't even have RAM memory regions that are neighbour to each other to allow a single memory allocation of the full size, since the 64kB of main RAM is not neighbour to USB or Ethernet RAM or to the two 64kB external memory banks. And the two 64kB external memory banks are not neighbour either.

Do you have external RAM connected? How much? If yes - how have you convinced the RTL to place 128kB heap in external RAM and 64kB heap in internal RAM?