Take care that in your document set up you have facing pages. So the left page (or even page) will take the left master pager, the right (or odd page) will take the right master page.
What if there are random pages to insert? or some pages need to modify its' pagenum?
For ex, when a book is almost finished, suddenly a page must insert before page4, then page 5 6 7 8 ....etc must be re-applied master page again.
How to make these pages auto arrange their pagenum position?
If you have facing pages and you allow to shuffle pages page number 5 will become page number 6 when you insert a page before. Then this page will take the left master page automatically. You have to use facing pages.
What is the problem here?
Adding single pages at random in a facing pages document, especially if there are master page objects that were overridden and /or the margins are not equal inside and outside, can cause unwanted master objects to appear behind overridden objects, or placed objects to no longer be properly aligned on a page. This happens whether the document is part of a book or not.