Website is slow to load when someone first connects

Greetings all. I have a strange problem. I recently installed ColdFusion 11 on a fresh test server running Windows Server 2008 and IIS7. Since this is a test server there isn't really any traffic going to it, so it probably doesn't serve pages but when I connect to it.

Now if I try to access a page after having not done anything in a while the browser will sit for about 15-20 seconds before the page loads. After that though every page loads just as fast as I'd expect. If I stop browsing the site and wait a while (not exactly sure how long, 10+ minutes?) it does the same thing.

This is only happening to the sites I have set up as ColdFusion sites. I have a test site that isn't configured via the Coldfusion tool and only has html content and it loads just fine.

I have followed the ColdFusion connector tuning found here: ColdFusion 11 IIS Connector Tuning — Adobe ColdFusion Blog and it hasn't changed the problem.

Anyone else run into this or know how I can fix it?

Thanks!