Hi,
we are having some issues with the new disk cache.
writing diskcache seems to take extremely long. we did some test and it came down to the fact, that a single 8 bit 1080p frame (which has a diskcache size of around 8 MB) takes about 3-4 seconds to write.
we don't have ssd-drives in our machines yet, but our drives are pretty fast. and yes, we use extra drives for the diskcache.
once the diskcache is written, everything works great and fast. it's just that writing a 10 sec. hd-sequence to disk takes minutes, which make the feature kinda useless (not even talking about writing cache to disk right with the ram preview).
anything we are missing here?
cheers,
Ronny
Make sure that you're not using the system disk for disk cache. Make sure you've got plenty of extra room. Make sure you're using a fast drive. SSD would be fastest.
If you don't have these options, then limit the size, empty often or turn it off.
Read this for more information.
the only thing we dont have (yet) is an ssd. everything else should be fine. and again, i am trying this on a very simple example. comp with one layer which is just a clip.
deactivating this is not really an option as this was the primary reason we switched to cs6.
i doubt this will work that much faster with an ssd.
anybody from the developers around to comment on this?
sorry, my bad
so the machine is a dell precision t7500, 2xquad xeon, 24gb ram, quadro fx 3800. it has a system drive plus a data drive. both are sata drives (i think they are both wd raptors). both drives have around 140 MB/s read/write performance (thats what the atto benchmark tells me).
there is a similar system here with 2x six core xeons, 24 gb ram, quadro fx 4000. rest is the same. the system has the same issues with disk cache.
there is a server, which delivers all source images which reads/writes at around 120 MB/s.
our test was taking hd footage from the server (tif sequence, 8 bit) and putting it into a comp. no fx. ram preview --> about 5 sec. of footage are done in a couple of seconds.
data disk with the diskcacge folder starts to make noise(writes). writes a couple of minutes.
so we purged everything, deleted all the diskcache and did a background render diskcache for the comp. same thing here. writes a couple of minutes which takes the the ram preview only seconds.
checked the cache files. they are just 8 MB a frame and there seems to be the right amount of cache frame (for 5 sec. of footage)
is there anything special about how ae cs6 writes the cache files - like doing it in tiny blocks or so?
we have no problem getting some ssd drives, but we want to make sure, that it works as advertised.
diskcache goes on data drive - forgot to mention.
btw. once the cache is written, reading it back into ram is superfast.
I just ran a test on a 3D text layer extruded with 2 lights. The comp is 6 seconds. It took about a minute to render. The disk cache went from 0 KB to 1.2 GB in the time it took to do the Ram Preview. IOW, I didn't see any delay in writing to disk cache.
I wanted to to make sure you realized that when you render to disk cacke the frames are rendered. (blue line). It takes the same amount of time to render to disk cache as it takes to render to ram preview.
Have you seen this demo of caching from Brian Maffitt? You should be able to see the same kind of results.
so we did a couple more tests and figured out, that the write performance of our drives for smaller blocksizes is pretty weak (allthough the drives are fast in general). this is true for pretty much all the machines we checked (around four). so it seems, aftereffect does write the cache files in chunks smaller than 128 kb (probalby even smaller), which causes the bottleneck for us - every file is more than 8 MB in size.
funny sidenote: we did a test, writing the diskcache onto the server which seems to be a lot quicker with writing small junks and the performance is a lot better, almost as expected!
so we just ordered an intel ssd to see how much this changes the performance. I'll keep you posted about that (will take a couple of days).
btw. did you do your tests with an ssd drive?
ps: i knew some of the presentations for the diskcache features, thats why we figured out that something was wrong. thanks for pointing me at the video though.
Hi Rick,
here are some updates to our situation with the diskcache:
we got ourselves an intel ssd and plugged it in one of our systems. the atto benchmark show much higher throughput with small blocksizes but in AE the situation has not changed. its as slow as before.
Is there a chance that you or anybody who is reading this could do a test on a windows7 machine?
the last thing we could try is to get an ssd pci card which is not connected to the sata-port of the machine.
thanks,
Ronny
Just to clarify your problem I want to make sure that it is taking much longer to write to disk cache than it takes to render a ram preview. I spent last week working on a windows machine and I gave it a test by setting up a simple and a complex comp. I set the resolution of both to full and ran a ram preview. 5 seconds on the simple comp, 40 seconds on the complex (tons of particles, motion blur and layers). Then I cleared all cache and purged all memory and ran the first test, Cache Work Area in background. The bg render took the about same about of time with both comps. I then Cleared all cache, purged all memory and ran another Ram Preview. Times were just about identical as the first time. I then, with the Ram Preview still in memory, I selected Cache Work Area in background and, as expected, because the only thing that should have happened would have been a file copy, the BG process only took about a second for both comps. I then Purged All Memory. This left only the blue line (disk cache) for both comps. A ram preview was nearly instantaneous for both comps telling me the Disk Cache was used.
If you're seeing huge differences in render times for ram preview and Cache Work Area in the Background then please let us know.
The machine was not set to render multiple frames simultaneously because most of the plug-ins used in the project I was working on were not MP aware and I was also bouncing back and forth to PPro all the time. In every test, and in all of my work, as soon as the work areas were rendered they were on the disk. The only way to change the write to disk time was to change the project.
yes this the case and yes, you completly understood my problem.
your test is indeed very good.
you just need to clarify on thing:
you said the bg-render took the same amount of time for both comps. do you mean bg render for the simple comp took as long as the bg render for the heavy
or
do you mean bg render of the simple comp took as long as ram preview of the simple comp and and bg render of heavy comp took as long as ram previe of heavy comp (you probably meant that)?
ok so i did another little test myself.
i have a comp with 50 frames (HD, 8bit):
ram preview(no multiprocessing): 2-3 sec
bg-render (from cached ram preview): 2 min 26 sec. (!)
additional test
copying the cache directory (it's 50 files, around 8 MB each) to an external usb 2.0 hdd (not a very fast drive) takes 18 sec.
copying the files back onto the internal aftereffect diskcache disk takes 14 sec. (which is certainly limited by the throughput of the usb drive). so in my world, aftereffects should at least be able to write the diskcache within 14 secseconds.
so to answer you question, yes, i see huge differences between ram preview and cache workarea in background.
my guess would be, that aftereffects is using a strange methode to write the cache to disk (which be only problematic with certain hardware, for example all our dell workstations).
writing to none sata drives, being networkmapped drives or usb seems to works better but is limited by the bandwidth of the interface (1gbit for networks and around 20MB/s for the usb 2.0 hdd we used)
i have just ordered a superfast pci ssd card which is pretty much the last thing i could think of (and the most expensive). i'll let you know how that works.
Hi Rick,
so we got ourselves an ultrafast internal pci ssd card (read/write about 1.4 GB/sec) and now everything works as expected (actually even better - aftereffects is pretty much realtime now).
still the problem remains, that with everything that was connected to the sata controller of our machines, the performance was really bad. we talked to our system administrator about the issue, but he wouldn't know of anything that could fix the problem.
my guess would be, that it is a very specific problem with the combination of dell workstations (and their sata controller) and the way aftereffects writes the diskcache. is there any way to bring this to the attention of the developers, because i'm sure (and i have already read on other forums) that other people are having this issue and its certainly not an option for everyone to just plug an ssd card into their machines (though it's really sweet :-).
So this issue is not fixed, we just got ourselves a pretty expensive workaround.
Anyways, thanks for your help and your patience.
cheers,
Ronny
ronny-anz wrote:
is there any way to bring this to the attention of the developers?
Yes there is: file a bug report. Include as much info as you can. Link them to this thread. They will greatly appreciate it.
North America
Europe, Middle East and Africa
Asia Pacific