Specifics: Screenshots, system info, crash log, effect settings.
In addition to what Mylenium specified, step-by-step of what you're doing, exact point version number of AE (12.2.1? 11.0.4?), and the format(s) and codec(s) of any assets in your comp would also be useful.
I'll bet the Photoshop file is CYMK. Convert it to RGB.
Well lucky you didnt put money on that because you would have lost the bet. Its rgb. I didnt use the PS file anyway, I used a jpeg. Is it ok to use a jpeg?
The shatter map should be the same size as your comp and RGB, sounds like you have done that. Without more details it's impossible to say. I made some dots using a shape layer and two repeaters, added a solid, set up a shatter map and it's working just fine for me. Try downloading this project and replace the shape layer with your polka dots. Saved as a CS6 file just in case.
I enjoy Shatter tremendously. Along with Wave World and Vegas, it is one of the more under-used and under-appreciated effects systems. When Shatter was first released, Mr Maffit was specific about how to create custom maps. He insists that Colorama, set to 8 hard colors, is the best way to create a map his Shatter engine will know how to use. Suggest you create your map in a comp, apply Colorama, and save it out as a frame, reimport.
What mght be causing AE to crash on YOUR map is impossible to decipher from here. Try other graphics files. If they continue to cause problems, keep changing formats and sizes and color spaces till you find the cause. But I will give you the answer we love to give and hate to read: That's never happened to me, not once in all the years I've been using Shatter, back when it was an Evolution plugin.
There's always the manual (and the implication of another bet). In addition to the resources listed here, Brian has some of the best stuff about Shatter available somewhere online: his original Atomic training videos for Evolution plugs.
Creating a custom shatter map
All layers in After Effects are represented as an RGBA image, including black-and-white images. The Shatter effect calculates the luminance threshold of each channel to create a custom shatter map. Shatter calculates the 50% luminance threshold of each channel, creating an image composed of only eight colors: red, green, blue, yellow, magenta, cyan, white, and black. These eight colors become possible combinations of the channels set either all the way on (255) or all the way off (0). The shatter layer splits along the edges of these different colored sections.
When designing custom shatter maps, you can find it useful to manually set a threshold for each channel of the image at 50% (you can use the Curves effect to do so). When you set the threshold, you can see how the image will be broken into pieces. Alternatively, you can create custom shatter maps by drawing an image using only the eight colors listed above, with no intermediate shades or anti-aliasing.
Note: Use the Colorama effect to posterize the colors of an image to these eight colors. Turn off Interpolate Palette in the Colorama effect controls. See Colorama effect.
The alpha channel determines whether a shattered piece exists. A white alpha channel value results in a shattered piece, and a black alpha channel value results in no piece. Using an alpha channel, you can make a tile map with holes in it or generate simple 3D models like extruded text.
Note: The custom shatter map determines the shapes of the pieces a layer shatters into, but not when the pieces shatter. The timing is determined by the Force controls and can be further controlled by a gradient layer.
The Toolfarm website provides a tutorial that shows how to create a custom shatter map for some real-world bricks.
Online resources about the Shatter effect
Aharon Rabinowitz provides a video tutorial on the Creative COW website that shows how make 3D extruded text in After Effects using the Shatter effect.
Rick Janusz provides a video tutorial on the Forging Fire website that shows how to use the Shatter effect to create an extruded 3D object from a 2D element.
Chris and Trish Meyer provide an article on the ProVideo Coalition website that explains how to use a simple set of expressions to orient a layer with the Shatter effect so that it integrates with other 3D layers in a composition.
Andrew Kramer provides a video tutorial on his Video Copilot website that demonstrates the use of the Shatter effect to make a planet explode. Lloyd Alvarez provides a video tutorial on the AETUTS+ website that demonstrates the use of the Shatter effect with a custom shatter map to
cause a person to disintegrate.