It looks like I solved it myself. The problem is, that the numericalConstants of the shader also have to be set manually!
So I tried this
this.m_context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, this.m_constants);
I got the constants with this:
this.m_constants = fragmentRegisterMap.numericalConstants.values;
Yes, this is exactly the work that ProgramConstantsHelper does for you. If you look at AGAL, there are no built in constants so all constant values need to be passed in.
I simply didnt thought about it, so all I got was strange behavior of the shaders ;-) Now everything works fine and fast