Creation of custom brushes is not currently possible other than using the Capture/Brush App. Once a brush is available in CC libraries, you should be able to use it. But neither Animate nor Illustrator allows it at the moment.
You can request for this feature via the wish-form if you feel it's important!
This is ridiculous. If I make a vector brush in illustrator and save it to CC library I can't use it in Animate? Your own site documentation is very vague to this fact....
From Creative Cloud (CC) Library
You can import new Art Brushes to your Animate document using the CC Libraries in addition to the Art brush presets available by default.
To add a new Art Brush, open the CC libraries panel and just click on any of the supported brushes. (Animate currently supports only Vector Brushes from CC Libraries).
It should state that it only supports vector brushes made in Capture CC. I wasted half a work day trying to figure out why I wasn't able to import a vector brush from the CC library.
Is this still true? That is really unfortunate if so.
I've made custom vector brushes in Illustrator (Love it.)... drag and drop them onto the CC library (Easy as pie!), and then it won't sync into Animate (What the...?)?
That seems like an obvious feature to have built in.
Using Capture seems like a more difficult, less precise method to get the art you want out of the software. Photos of brushes are clearly going to be less accurate than the artwork you can create in Illustrator.
You can make custom brushes now. They work pretty well actually.