I seem to be missing something I do not understand what your describing with " When I press M in the filter window for the new filter I do not see any mask." What or where is the filter window. For me the Filter mask seems to be common to all smart filters added to the smart object layer.
I your mean Filter Gallery window. All effects will also go through the common filter mask. However Filter Gallery will only show as a single smart filter.
Here I selected the left half if the image so there would be an active selection when the Filter mask is added and not be a white show all mask. In the filter gallery I could see each effect added was only to the left half. I commit three effects. so the filter would show with the filter mask in the layers palette. I the reopened the smart filter Filter Gallery. When I reopened it the active selection not being active any more showed the whole image as being effected however the filter mask protected the right side of the smart object layer as you can clearly see in the image window.
Again when I originally applied the smart Filter Gallery the was an active selection so there would be something in the filter mask and the select would be visible in the filter gallery. Here is what I saw.
If this is not the first smart filter added to the layer and there was already A filter mask to see its effect in any new added smart filter I would think you would first have to load the filter mask as an active selection.
Not only did I have to load the filter mask as a selection I had to turn off the other smart filters visibility.
I convert my layer to a smart object. I then go to filter blur gallery and create a field blur using several points. I exit the gallery and to to the filter mask and mask out the person in the picture so that the only the background is blurred.
I click again on the smart object and add another (second blur gallery filter). This time I create a new control point and add some bokeh with the bokeh sliders. However it seems to ignore the filter mask and the effect while I am in the filter gallery screen is not masked by the filter mask. Now if I click okay and go back to the layers panel it seems as if the mask was applied to the bokeh effect but I cannot see the effect of the mask when I am inside the blur gallery while I am manipulating the sliders.
Did you load the Filter Mask as a selection before reopening the smart filter gallery filter to do a re-adjusting of the filter or adding a second smart filter. Like I showed above.
The only preview is in the filter gallery window and it filter the whole smart object unless there is an active selection for the preview.
Remember the smart object layers pixels are not altered at all only the layer composite rendering is effected and the filter mask will be used during the rendering process for the layers composite view.
The smart object layers pixels rendered for the object for the layer by Photoshop are not altered they are just used during the rendering process as the starting point the associated transform is done then each filter applied in order and the filter mask is used over the smart object layers pixels transformed pixels while rendering the layer's composite view and shown scaled down in the layer palette layer content icon.