vfp problem

Using vfp when not in user mode is troublesome. This bit of documentation on its use in the Linux kernel should give you an idea of the problems

https://www.kernel.org/doc/Documentation/arm/kernel_mode_neon.txt

The Cortex-M series have a special mechanism to do lazy save and restore of the registers which are not preserved by a called program which makes it much easier to use floating point in an interrupt.

ARM Information Center - Cortex-M4(F) Lazy Stacking and Context Switching

Using vfp when not in user mode is troublesome. This bit of documentation on its use in the Linux kernel should give you an idea of the problems

https://www.kernel.org/doc/Documentation/arm/kernel_mode_neon.txt

The Cortex-M series have a special mechanism to do lazy save and restore of the registers which are not preserved by a called program which makes it much easier to use floating point in an interrupt.

ARM Information Center - Cortex-M4(F) Lazy Stacking and Context Switching

If the kernel didn't support kernel_neon_begin(), what can I do??

There's probably some facility for this in whatever operating system you're using so it'll depend on that. Which one is it? If desperate one could enable the floating point save all the registers, or at least the ones which aren't callee preserved, plus the control and status registers, call your routine to do the work and then restore the registers and disable the floating point again but a routine to do this would save on  some of the work when it doesn't need to be done.