I'm working on a big project with over 50 screens. You have two screenshots attached. I used a non-conventional approach. I design in PS, I export each artboard and I import it into XD. Then I create transparent rectangles and place them over my buttons to create the actual prototyping. I understand this is not the way XD is supposed to work, this is not the question here. Just pointing this out because I assume having less layers is less demanding on the program.
I have my back buttons set on over 20 screens and I was creating the rest, yet it seems after a certain point XD stopped working correctly. I go through the motions to create the 'previous artboard' action, yet after the link should be made, it remains at its initial state. I switched to another smaller project and there it worked without issues (while having the big project opened).
It may also be worth mentioning that in the big project, the program hesitates whenever I try to create a new link. I click on the blue arrow and the program stops for a second before it shows me my options (target, transition, etc). In the smaller project this doesn't happen. I tried killing all programs assuming this is a RAM issue, same thing.
I'm using the latest XD at the time of this post (20th of May), on a Windows 10 machine that just got updated yesterday.
i7 4790 with 16 Gb RAM and 850 Pro 1 Tb SSD.
Are you able to share this file with us confidentially so we can test it? If so, please send me a link at pflynn (at adobe.com).