You are generating Responsive HTML5, right? Can you open one of the generated ASPX topics with notepad? In the first line, which of the following do you see:
<%@ Page Language="C#" masterpagefile="~masterurl/custom.master"
I have tested this issue and on RoboHelp 11, the SharePoint native generation for Responsive HTML5 simply doesn't work. (It generates regular topics instead of ASPX files only with the extension ASPX.) This is a bug so please report it at https://www.adobe.com/cfusion/mmform/index.cfm?name=wishform&product=38
Please note that my recommendation that this is a bug applies ONLY to Responsive HTML5. Multiscreen HTML5 still creates ASP pages, though there could be a separate issue there.
Please post back if you are using Multiscreen HTML5.
We are using Multiscreen HTML5, and with our settings of layout (System Master Page) tag in .aspx looks like:
<%@ Page Language="C#" masterpagefile="~masterurl/default.master" title="Title in Cyrillic" %>
I think we solve the problem. The title of project (and automatically layout) were contained "" (). After I renamed it in simple name, the aspx start open. So Sharepoint couldn't recognize the title of page with quotas.
Thanks for your answer, because it pushed me to decision.