Welcome to the forum.
I have moved your post to a different category as your question is about the source project, not the fact that you generate to RoboServer.
There is no specified maximum, it's about what works on your PC. If your project is opening, working and closing at an acceptable speed without any problems, that's the criteria. At the levels you are talking about, typically people do report problems of one sort or another that go away when the project size is reduced.
How is the size reduced? Merged help. There is an article on my site about merged help but and splitting projects but it does not cover outputting to RoboServer. As and when you get to that point, there are others here who will help you on that aspect.
See www.grainge.org for RoboHelp and Authoring tips
There are three main criteria that you should be concerned with:
1) The amount of physical RAM that is installed in the user computer.
2) The total addressable RAM, which is determined by whether the operating system is 32-bit or 64-bit.
3) The variable types used within the software (i.e., RoboHelp) that are used during your compilation process.
For points (1) and (2), a big issue these past few years has been the barrier of addressable RAM available on 32-bit systems. To summarize in simple terms, 32-bits allows the system to address approximately 4.2 GB of RAM, though a portion of this address space must be allocated to the video card, plus there might be restrictions according to the operating system. In other words, if you are distributing a help file that simply uses more resources than are available on user computers, then they will be unable to open the file.
Point (3) has to do with the range of numeric values that variables within a software application (i.e., RoboHelp) are capable manipulating behind the scenes. Again, this is related to 32-bit versus 64-bit, where generally (integer) variables are used to store 32-bit values in the range of +/- 2.1 billion (or unsigned 0..4.2 billion), though it is possible to use 64-bit integers. In practice, you probably won't encounter this as an issue, though it would manifest in the form of the software either reaching unexpected conclusions, or shutting down without warning.
[ As an example of point (3), back in the '90s I had an encyclopedia set that came on several install CDs, which to be installed on a hard drive with ~60 GB capacity. At the time, there was an old Windows system call that couldn't report accurate hard drive capacity over ~2 GB. If the hard drive in question exceeded ~2GB, then the value by the system call would be "mod" 2GB (i.e., the value would "roll over" multiple times until some value less than 2GB was achieved). As it turned out, the installer for this encyclopedia used this system call, and it would check available space *after* each CD-worth of content was installed. So after 2-3 discs were installed, the installer refused to continue because it determined that insufficient space remained on the hard drive. ]
Anyhoo, aside from the technical aspects, here's a tidbit that might help you bring the size of your individual help files down. This assumes that you don't need *all* of the project files to be available within each discrete help file.
1. In RoboHelp, create a conditional build flag (perhaps called) "Exclude from Output"
2. In the Topic List pane, select *all* of the project files.
3. Right-click the selection and choose Properties.
4. On the Advanced tab, tick your Exclude from Output flag.
5. Click OK (and then go for a break whilst the files are processed).
6. Open the Table of Contents for the help file that you intend to compile.
7. For the top-level tree of the TOC, right-click and choose Apply Conditional Build Tag >> New / Multiple.
(If the TOC has multiple top-level trees, then repeat this for each tree)
8. The Conditional Build Tags dialog will open.
9. In the Topics column, clear the Exclude from Output checkbox.
10. Click OK.
11. As part of the build conditions for your project, set the compilation to ignore files that have the Exclude from Output flag.
At this point, compile the project. What you'll find is that *only* the files that pertain to the TOC are included, and files belonging to other projects won't be included (i.e., files that user could previously find using the Search tab). This has benefit of hiding files that given users shouldn't have access to, and bringing the overall size down significantly for each help file.