Is there a way to install an executable into the extension folder (which I can get using CSXSInterface.getInstance().getSystemPath(APPLICATION))?
Yep, the answer depends on how you're packaging your extension.
If you're using Extension Builder, you just need the executable to be automatically copied to the bin-debug folder during the build process. The simplest way to achieve this is to put the executable underneath the src/ folder. Another option is to make your own resources/ directory and add it to the project source path (see the section "Adding resource folders to the project source path" here: http://help.adobe.com/en_US/Flex/4.0/UsingFlashBuilder/WS6f97d7caa66ef6eb1e63e3d11b6c4ce74 9-7fd8.html). Once you've got it copying to the bin-debug folder you're set, you can launch or export as usual and your executable will end up in the extension folder.
If you're using ucf.jar, just make sure that your executable is in the extension folder you're packaging.
You shouldn't need to create a hybrid extension in order to have a CS extension that uses NativeProcess.