Re: Lots of very high res textures

[Colin Doncaster <colin_doncaster(at)yahoo.ca>]

Hey there -

My apologies, new timer on the list.  Also, I haven't done as much  
rendering with MR as I have with Renderman but here are some thoughts  
based on general rendering architecture.

First off, have you brought up a top or something similar to see if  
it's the memory that is maxing out vs. the hard disk thrashing due to  
texture reads?  Be aware the drive can start to thrash both because of  
texture reads and virtual ram, so if the mem is maxing out that will  
be another concern as they'll be fighting to disk usage.  You can help  
this out by setting your virtual drive to something other than your  
local drive ( assuming you're reading textures locally ).

Are the textures stored locally or somewhere remotely on the network?  
if so your bottleneck may be ethernet ( Gig E or whatever you have ),  
in the past I've used scripts that copies all of the needed textures  
locally and this offers incredible speed ups when accessing large  
textures remotely ( same should be done for any other data ).  This  
also helps when you're rendering 100 frames of animation and each  
frame is trying to access the same texture, you really want each job  
to access it's own texture pool.

The MR docs talk about setting the textures as "local", it's in the  
memory mapped textures section.

This also seems to be the case for the memory caching, it only works  
for textures defined as "local", I'm not too sure if this is on by  
default in XSI - or how to turn it on.

Are you using any raytracing, i'd say you'd probably want to get it  
working with the rasterizer rendering option.

Changing the tile order and tile size should also help, if the local  
and caching is working correctly.  The smaller the tiles the more  
likely MR will throw out unused texture information, the order forces  
it to evaluate the tiles differently possibly triggering unused  
textures earlier. There only seems to be two order options though and  
they don't seem to be ideal for chucking textures away.

Last but not least is render slicing, you'll have to write a script to  
parse the .mi files but you can set up a system where each frame is  
broken in to x, y or x/y slices and then re-assembled. ( adjust the  
window option on the camera ) This would be similar to your render  
region and give the images more of a chance to succeed.  This is  
pretty much used exclusively on a project I'm on now via renderman.

I guess you have to wonder if you really need the 8k textures to begin  
with, you'd either have to be really really close to something and/or  
be rendering a massive frame.

Just some thoughts, again, I apologies if this is all a given as XSI  
with MR is more of an unknown to me.

Cheers.

On 3/01/2008, at 4:58 PM, Andy Jones wrote:

> Okay, let me preface this by saying I know it's kind of ridiculous  
> to try to render with a bunch of 8K 32-bit floating point textures.   
> I have an asset that I got from another vendor that came with  
> multiple 8K maps for different parts of the shading (diffuse1,  
> diffuse2, spec1, spec2, etc. -- like 8 maps per surface total).  And  
> there are about 5 different shaders on the asset.  So, all told, I'm  
> looking at about 25GB of texture data (when uncompressed).  And,  
> after converting the files to pyramidal .map files, I'm up to 53  
> GB.  Obviously, that's WAY overkill for the single frame of the  
> asset at a sort of far distance that I have to render.  But it's not  
> that common that I end up with an asset this heavy on textures, so I  
> figured I'd have a go at getting it to actually render.
>
> I'm still experimenting, but so far, I've noticed some pretty  
> interesting things.
>
> 1) I'm taking it for granted that I must use pyramidal .map files.   
> I didn't even bother trying this without doing that, but I'm pretty  
> sure it's true.
> 2) In conjunction with pyramidal .map files, elliptical filtering  
> works at a reasonable speed for lookups on a single texture, and  
> didn't seem terribly slower than disabling elliptical filtering  
> (though I found that a little surprising).  So, most of my tests  
> were done with elliptical filtering (20, 4, .3).
> 3) For those of us like me who often lazily don't set a Mental Ray  
> memory limit, you'll need to do that.  Otherwise, MR has an  
> increased tendency to choke on memory allocation.
> 4) If I draw a region, the render speed increases dramatically when  
> I refresh.  For example, an experimental "first round" render time  
> was about 1m31s, whereas refreshing the region took only 16s.  My  
> hypothesis for this is that the hard disk's cache is simply doing  
> its job.
> 5) If I enable more than one texture per sample (for example, if I'm  
> using both a spec map and a separate diffuse map), I experience  
> dramatically increased render times.  For example, 3m12s vs 14m17s.  
> My hypothesis for this is that the hard disk is having to seek back  
> and forth between the two textures which live very far apart from  
> each other on the disk (since they are 8K).
>
> Together, 4 and 5 suggest a possible optimization whereby one would  
> benefit from simply rendering each texture individually to an empty  
> buffer, then re-rendering the multi-textured surface samples  
> (presumably pulling textures straight out of the cache the second  
> time around).  Or, you could render in multiple passes, restricting  
> each pass to a single texture (though this gets really really  
> complicated, if not impossible, when you get to things like  
> reflections, where you're casting multiple rays per sample).
>
> 6) For these renders, my CPU utilization is generally down to about  
> 6%, for the obvious reason that all the time is being spent doing  
> lookups on textures.  What I would really like is a way to manually  
> "kill off" perceived texture resolution in my .map files, so that MR  
> effectively ignores a specified number of the highest resolution  
> layers of the pyramid, so that I could force myself back to renders  
> with more reasonable CPU utilization.  While there's an obvious way  
> to do this (down-res the texture and regenerate the .map), it's a  
> lot of extra work that doesn't seem like it should really be  
> necessary, since the resulting textures are already a subset of the  
> existing ones.  I think there are ways to sort of do this, and I'm  
> particularly interested in any information people have about that.   
> For example, I know that in the elliptical filtering settings, you  
> can specify the maximum number of texels covered by the minor radius  
> in the highest resolution pyramid.  My standard has aways been 4,  
> but for example, I was able to decrease the render above that took  
> 3m12s down to 1m37s by taking this value down to 2.  That's an  
> improvement, but nowhere near what would be needed to get back to  
> more reasonable render times.  I've already been over this a good  
> while back, but it's pretty annoying that MR reads the texture  
> filter bias (the multi-resolution texture blurring parameter on the  
> clip) from the .map file (basically because they chose to implement  
> on-disk memory mapping for the entire image structure, instead of  
> just the data portion of it, as they should have).
>
> Anyway, I'm not holding my breath for any major revelations about  
> how to optimize a scene with lots of gigantic textures (unless  
> someone's got some?), but I just thought this was a good opportunity  
> to push the envelope a bit, do a few tests and think about possible  
> improvements for the future.  As storage gets more and more  
> affordable, 8K textures and "taller" pyramidal maps are going to  
> become more and more common.  Not so much because you really need  
> the higher resolution, but because you "could" need it on a close-up  
> render of an asset, and it's often easier to just paint one high  
> resolution texture and have it be dealt with dynamically.  I think  
> that at some point, MR needs a re-write of the direct-from-disk  
> texturing method.  A multi-channel format would be a good start, to  
> optimize situation #5 above.  A new format should ideally also be  
> able to completely emulate the behavior of lower resolution textures  
> with as little performance loss as possible.  And on the  
> implementation side, it should be optimized as much as possible for  
> read-ahead.
>
> But honestly, I'm still kind of impressed that it's even stable (32- 
> bit Windows).
>
> Now, I'll just shut up and down-rez all my textures to 1/4th  
> resolution.
>
> - Andy

---
Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
unsubscribe xsi