My problem is that I don't know how to exchange variables between the cocoa code and my code in c++.
Rename your .cpp files to .mm and you can mix obj-c and c++ freely. If you have to pass data to from your obj-c code to non-obj-c, the Core Foundation equivalents are quite helpful. If there is a specific case you need help with, post here and we can help you out.
I haven't look at the SDK in a bit, but there should be some examples of non ADM dialogs. Is the Twirl Filter example still there?
Probably someone has managed to write a program with a user interface (a simple dialog with a button or two, that would be good) without using AMD.
I've built cross platform (non-ADM) user interfaces with the Creative Suite SDK (CS5+) using PlugPlug to launch it from C++ code. However, you obviously cannot share data (variables) between C++ code and ActionScript, and using CSXS events to communicate and pass data between the two can be an unpleasant experience. There are a few examples in the SDK.
You will have to use FlashBuilder / Extesion builder to create an Illustrator Extension.
This will be the UI.
To make it works with your plugin, you will have to use csxs event to make them communicate.
It is not so complicated, and works pretty well.
You can take a look at StrokeFilter and StrokeFilterUI to see how you will have to do.