Looks like the last command is failing because it is trying to link against host libraries (from /usr/lib64) which are probably not compiled for ARM architecture. If you were compiling on your target then this would not be a problem. As we are cross compiling here, things are slightly tougher.
The steps taken to build libarmidillo.so from this post look promising http://stackoverflow.com/questions/21263427/cross-compiling-armadillo-linear-algebra-library
To help link your applications the gcc compiler in DS-5 provides two differently built libc system root’s, hardfloat (hf) and softfloat, but ideally you’d want to link against the system root built for your device. To include headers you could point to DS-5/sw/gcc/arm-linux-gnueabihf/libc/usr/include/; and select the appropriate system libraries from DS-5/sw/gcc/arm-linux-gnueabihf/libc/usr/lib.
As long as you are building for an ABI that is compatible with your target filesystem ABI then your application should run (presuming the C libraries are compatible; older versions are normally compatible with newer ones). You can specify the ABI by selecting a combination of the appropriate –mfloat-abi and –mfpu switches, but I think the default for is hardfloat for arm-linux-gnueabihf.
Hope this helps