No, the checkmarks are not customizable at all.
You don't mention what version you're on, and from your question I presume it's a pre-CC version.
In CC, the checkmarks have turned into what is essentially checkboxes. Have a look at the screenshot here http://www.freelancebookdesign.com/scripts/display-line-numbers-in-indesign to see what it looks like in CC.
Hi Ariel, I am actually going to use the interface with a script for FrameMaker 11. I posted the question here because the community is bigger and the ScriptUI controls should be basically the same regardless of the product. I do see what you mean about the look being different in InDesign CC. Thanks for your reply. --Rick