we are having some issues with the new disk cache.
writing diskcache seems to take extremely long. we did some test and it came down to the fact, that a single 8 bit 1080p frame (which has a diskcache size of around 8 MB) takes about 3-4 seconds to write.
we don't have ssd-drives in our machines yet, but our drives are pretty fast. and yes, we use extra drives for the diskcache.
once the diskcache is written, everything works great and fast. it's just that writing a 10 sec. hd-sequence to disk takes minutes, which make the feature kinda useless (not even talking about writing cache to disk right with the ram preview).
anything we are missing here?
Make sure that you're not using the system disk for disk cache. Make sure you've got plenty of extra room. Make sure you're using a fast drive. SSD would be fastest.
If you don't have these options, then limit the size, empty often or turn it off.
Read this for more information.
the only thing we dont have (yet) is an ssd. everything else should be fine. and again, i am trying this on a very simple example. comp with one layer which is just a clip.
deactivating this is not really an option as this was the primary reason we switched to cs6.
i doubt this will work that much faster with an ssd.
anybody from the developers around to comment on this?
You didn't say if your cache was on the boot disk. You didn't say if the cache disk was a portable or external drive. If the drive is external and it's not a USB3 or Thunderbolt you're going to be looking at slow read write times.
A little more about you system would help a lot.
sorry, my bad
so the machine is a dell precision t7500, 2xquad xeon, 24gb ram, quadro fx 3800. it has a system drive plus a data drive. both are sata drives (i think they are both wd raptors). both drives have around 140 MB/s read/write performance (thats what the atto benchmark tells me).
there is a similar system here with 2x six core xeons, 24 gb ram, quadro fx 4000. rest is the same. the system has the same issues with disk cache.
there is a server, which delivers all source images which reads/writes at around 120 MB/s.
our test was taking hd footage from the server (tif sequence, 8 bit) and putting it into a comp. no fx. ram preview --> about 5 sec. of footage are done in a couple of seconds.
data disk with the diskcacge folder starts to make noise(writes). writes a couple of minutes.
so we purged everything, deleted all the diskcache and did a background render diskcache for the comp. same thing here. writes a couple of minutes which takes the the ram preview only seconds.
checked the cache files. they are just 8 MB a frame and there seems to be the right amount of cache frame (for 5 sec. of footage)
is there anything special about how ae cs6 writes the cache files - like doing it in tiny blocks or so?
we have no problem getting some ssd drives, but we want to make sure, that it works as advertised.
diskcache goes on data drive - forgot to mention.
btw. once the cache is written, reading it back into ram is superfast.
I think the bottleneck is reading and writing the source files through the network. Try copying the source files to a local data disk and see if this improves performance. If it does, then talk to your IT folks about the network.
i doubt that. the same problem occures if i take a solid, apply a fast blur with some keyframes (there is no i/o to render that) and create the cache. writing cacheframes is just slow. normal rendering is fine (even to the networkdrive).
we dont cache to the networkdrive!
I just ran a test on a 3D text layer extruded with 2 lights. The comp is 6 seconds. It took about a minute to render. The disk cache went from 0 KB to 1.2 GB in the time it took to do the Ram Preview. IOW, I didn't see any delay in writing to disk cache.
I wanted to to make sure you realized that when you render to disk cacke the frames are rendered. (blue line). It takes the same amount of time to render to disk cache as it takes to render to ram preview.
Have you seen this demo of caching from Brian Maffitt? You should be able to see the same kind of results.
so we did a couple more tests and figured out, that the write performance of our drives for smaller blocksizes is pretty weak (allthough the drives are fast in general). this is true for pretty much all the machines we checked (around four). so it seems, aftereffect does write the cache files in chunks smaller than 128 kb (probalby even smaller), which causes the bottleneck for us - every file is more than 8 MB in size.
funny sidenote: we did a test, writing the diskcache onto the server which seems to be a lot quicker with writing small junks and the performance is a lot better, almost as expected!
so we just ordered an intel ssd to see how much this changes the performance. I'll keep you posted about that (will take a couple of days).
btw. did you do your tests with an ssd drive?
ps: i knew some of the presentations for the diskcache features, thats why we figured out that something was wrong. thanks for pointing me at the video though.
... btw. did you do your tests with an ssd drive?
A USB 3 Toshiba 1TB portable drive attached to my MacBook Pro. As soon as the frame was rendered (blue line appeared) it was on the drive.
here are some updates to our situation with the diskcache:
we got ourselves an intel ssd and plugged it in one of our systems. the atto benchmark show much higher throughput with small blocksizes but in AE the situation has not changed. its as slow as before.
Is there a chance that you or anybody who is reading this could do a test on a windows7 machine?
the last thing we could try is to get an ssd pci card which is not connected to the sata-port of the machine.
Just to clarify your problem I want to make sure that it is taking much longer to write to disk cache than it takes to render a ram preview. I spent last week working on a windows machine and I gave it a test by setting up a simple and a complex comp. I set the resolution of both to full and ran a ram preview. 5 seconds on the simple comp, 40 seconds on the complex (tons of particles, motion blur and layers). Then I cleared all cache and purged all memory and ran the first test, Cache Work Area in background. The bg render took the about same about of time with both comps. I then Cleared all cache, purged all memory and ran another Ram Preview. Times were just about identical as the first time. I then, with the Ram Preview still in memory, I selected Cache Work Area in background and, as expected, because the only thing that should have happened would have been a file copy, the BG process only took about a second for both comps. I then Purged All Memory. This left only the blue line (disk cache) for both comps. A ram preview was nearly instantaneous for both comps telling me the Disk Cache was used.
If you're seeing huge differences in render times for ram preview and Cache Work Area in the Background then please let us know.
The machine was not set to render multiple frames simultaneously because most of the plug-ins used in the project I was working on were not MP aware and I was also bouncing back and forth to PPro all the time. In every test, and in all of my work, as soon as the work areas were rendered they were on the disk. The only way to change the write to disk time was to change the project.
yes this the case and yes, you completly understood my problem.
your test is indeed very good.
you just need to clarify on thing:
you said the bg-render took the same amount of time for both comps. do you mean bg render for the simple comp took as long as the bg render for the heavy
do you mean bg render of the simple comp took as long as ram preview of the simple comp and and bg render of heavy comp took as long as ram previe of heavy comp (you probably meant that)?
ok so i did another little test myself.
i have a comp with 50 frames (HD, 8bit):
ram preview(no multiprocessing): 2-3 sec
bg-render (from cached ram preview): 2 min 26 sec. (!)
copying the cache directory (it's 50 files, around 8 MB each) to an external usb 2.0 hdd (not a very fast drive) takes 18 sec.
copying the files back onto the internal aftereffect diskcache disk takes 14 sec. (which is certainly limited by the throughput of the usb drive). so in my world, aftereffects should at least be able to write the diskcache within 14 secseconds.
so to answer you question, yes, i see huge differences between ram preview and cache workarea in background.
my guess would be, that aftereffects is using a strange methode to write the cache to disk (which be only problematic with certain hardware, for example all our dell workstations).
writing to none sata drives, being networkmapped drives or usb seems to works better but is limited by the bandwidth of the interface (1gbit for networks and around 20MB/s for the usb 2.0 hdd we used)
i have just ordered a superfast pci ssd card which is pretty much the last thing i could think of (and the most expensive). i'll let you know how that works.
so we got ourselves an ultrafast internal pci ssd card (read/write about 1.4 GB/sec) and now everything works as expected (actually even better - aftereffects is pretty much realtime now).
still the problem remains, that with everything that was connected to the sata controller of our machines, the performance was really bad. we talked to our system administrator about the issue, but he wouldn't know of anything that could fix the problem.
my guess would be, that it is a very specific problem with the combination of dell workstations (and their sata controller) and the way aftereffects writes the diskcache. is there any way to bring this to the attention of the developers, because i'm sure (and i have already read on other forums) that other people are having this issue and its certainly not an option for everyone to just plug an ssd card into their machines (though it's really sweet :-).
So this issue is not fixed, we just got ourselves a pretty expensive workaround.
Anyways, thanks for your help and your patience.
is there any way to bring this to the attention of the developers?
Yes there is: file a bug report. Include as much info as you can. Link them to this thread. They will greatly appreciate it.
Check your MP settings. It sounds like you're using MP rendering and background cacheing isn't going to work very well with that. Turn off MP and give it a check.
Rendering to the disk cache is entirely a different thing than writing to disk cache.