You can certainly use FlexUnit. Add the libraries to your project and test away.
What we normally do, depending on the overall project requirements, is put as much of the core/logic as we can in a library. Then we write both an Air app and an Extension that are intended to execute unit tests against that library. The Air app tests the functionality that is not dependent on a host application (PS, AI, ID...).
Depending on your project, you may or may not be able to use an Air app for testing (CSXS calls are not supported from Air).
Is it planned to handle Creative Suite Extension project in FlexUnit?
Another option is to abstract the API objects into interfaces, only adding the functionality you need, and mocking those with something like mockolate to do your tests. It doesn't work for everything, but it covers a lot more than the CSEB independent stuff.
As an example, I have IApplication, with an activeDocument getter that returns IDocument. The implementations are handled by a service locator, so my extension uses actual API objects, and my test project uses mocks - easy!