Have now narrowed this down to a problem in the a previous line that exports a page to JPG.
document.Export 1246775072, c:\image.jpg
The image file is created correctly, however if i do any sort of floating point arithmetic (x = 1 / 2 for example) after this call then i get the overflow error.
If i comment this line i have no problems.
If anyone is willing to look at this problem this is a minimal script that reproduces it.
Set InDesignApp = CreateObject("InDesignServer.Application")
Set IndDoc = InDesignApp.Open("C:\overflow.indd")
InDesignApp.JPEGExportPreferences.JPEGExportRange = 1785742674
InDesignApp.JPEGExportPreferences.PageString = IndDoc.Spreads(1).Pages(1).Name
x = 1 / 2
IndDoc.Export 1246775072, "C:\overflow.jpg"
x = 1 / 2
You will need to create a document called overflow.indd in C. I can pass you mine if i could find a way to attach a file to this post.
You should get the overflow error in the last line of code.
To my surprise the error persists!
The only solution is to restart the server.