Re: 64 vs. 32 bit with map files?

[Andy Jones <andy(at)thefront.com>]

Problems with it (some of which you might have easy solutions for?):

1) If I turn it on, it writes the files to a subdirectory within the 
directory the texture is in.  So I have to write scripts to cache the 
textures to a local drive anyway.  Otherwise I'll kill my network.
2) The problems with architecture persist when using .mi2 files to, say, 
send stuff to a 3rd party farm.
3) While verifying items on this list, XSI 6.0 crashed while trying to 
open the preferences :(
4) When I change the filtering settings, it doesn't force an update to 
the .map file.  I.e., if I generate it without filtering on, it is 
forever stuck that way.  So, if I want elliptical filtering to work 
efficiently, I also have to turn on  multi-resolution texture on the 
clip.  So, what do I do if I want an image node that does elliptical 
filtering and another image node that does no filtering whatsoever?  
Answer -- make 2 copies of the texture.
5) The filtering level is hard-coded in the file format because .map 
files contain an entire texture struct instead of just the data part of 
it.  So I can't use the same texture file with multiple mult-resolution 
texture biases.

Currently, we work around these limitations by generating our own .map 
files to a local directory, always forcing pyramidal filtering and 
exclusively using elliptical filtering.  It works okay for us, but it's 
less than ideal.  We moaned about it about 1.75 years ago, but 
eventually just let it go, since it basically worked okay.  To me, this 
architecture thing is a final straw, making it time to finally deal with 
some of this stuff.

And, in case it hasn't been clear, I still feel like the solution to 
this should be coming from Mental Images -- not you guys.  That way .mi2 
files can work nicely too.  But of course, I appreciate all of your 
efforts to fix it yourselves when Mental Images slacks off.

To me, the ideal solution is to implement something like a .map file 
that maps a more pure data blob to disk in a directly readable way, them 
implement a texture struct that reads its data out of these blobs.  
Then, a locally overridable local flag would be included to optionally 
map all, none or some textures to this format before the render (obeying 
the syncing rules XSI uses, but also looking at anything in the mi2 file 
that might change the data encoded).  And the files would go to a 
subfolder of the texture's location, or a specified location that can be 
set in the .mi2 file.  Ideally, these blobby things could be used by 
shaders as well, to allow for big arrays of data without taking up a ton 
of memory.

So, .map version 1.0 (the current imlementation is version 0.6).

- Andy

Halfdan Ingvarsson wrote:

> Like I said earlier, XSI already has a facility to do this for you. Just make sure it's not enabled on the artists' workstations but only on the render nodes (you can force it in the preferences). It automatically converts any texture to .map files based on timestamps and a size threshold.
>
> See: "mental ray Render Options->Optimization->Convert Textures to Memory Maps." and "Preferences->Rendering->Images->Map Conversion"
>
> - ½
>
> -----Original Message-----
> From: owner-xsi(at)Softimage.COM [mailto:owner-xsi(at)Softimage.COM] On Behalf Of Andy Jones
> Sent: 28-Mar-07 10:29
> To: XSI(at)Softimage.COM
> Subject: Re: 64 vs. 32 bit with map files?
>
> So, we're in agreement then.  Productions need a tool that manages .map 
> files for them and makes them transparent to the user.  The way I see 
> it, there are two options:
>
> Option 1:  Everyone hires me, Alan or some other Jones.
> Option 2:  XSI and/or Mental Images provides a solution.
>
> As tempting as option 1 is to me in the short term, I'd rather it be 
> option 2 so that in the long run XSI doesn't get replaced by something 
> more user-friendly and my expertise beyond .map files is still useful.
>
> - Andy
>
> Halfdan Ingvarsson wrote:
>
>  
>
> > Don't forget problems of endianness, or even non-IEEE754 systems, where your scheme will completely break down. Bear in mind that mental ray works on more than just x86 chips. Also, user-friendliness has never really been their forte ;)
> >
> > I dealt with it in production way back when and I never thought it much of a problem. I just did the local conversion on the render node and used that to sync up with the server, much like what Andy describes. The map files were never ever used by the artist at any point and neither should they.
> >
> > You kids today have it too easy :-)
> >
> > - ½ 
> >
> > -----Original Message-----
> > From: owner-xsi(at)Softimage.COM [mailto:owner-xsi(at)Softimage.COM] On Behalf Of Alan Jones
> > Sent: 28-Mar-2007 06:05
> > To: XSI(at)Softimage.COM
> > Subject: Re: 64 vs. 32 bit with map files?
> >
> > Halfy and Luc-Eric,
> >
> > The problem isn't our understanding how it works internally or the reasons why it is so. We're dealing with how to use it in production and while yes it's possible for us to do things to make it work, such a basic feature should work without any user intervention.
> >
> > I don't see the reason why are format which does work as a memory dump couldn't be made to work on both 32bit and 64bit - 100 megabytes of 32bit floats in a row would be accessed identically on either platform. I'm sure the map file is more complex than that, but it should be possible to make it work.
> >
> > Cheers,
> >
> > Alan.
> >
> > On 3/28/07, Andy Jones <andy(at)thefront.com> wrote:
> >
> >
> >    
> >
> > > Instead of using XSI's automatic generation, we generate them as 
> > > needed for a scene using a script, then we basically rsync them into a 
> > > cache on local machines from the render script.  This seems to be 
> > > faster for us than regenerating them every time, which can take quite 
> > > a while.  Plus we have more control over how the .maps are generated.  
> > > I suppose we could also modify our scripts to have them do the rsync 
> > > on the original textures, then regenerate the .map files on the local 
> > > machines, but I don't know that this would necessarily be that 
> > > faster...  So, yes, we're perfectly capable of coding a solution that handles .map files elegantly.
> > >
> > > But it's exactly my point that it's possible to work around the issue 
> > > of .map files that are generated for the wrong architecture by 
> > > trashing them, regenerating them, translating them, and so on.  So, 
> > > the basic memory-footprint-on-disk functionality of the .map files 
> > > should ideally be encapsulated within some sort of data structure and 
> > > algorithms that provide a more seamless solution.  I'm thinking this 
> > > behavior should reside in Mental Ray, so that I can, say, send a scene 
> > > and a bunch of .map textures to a renderfarm somewhere and not have to 
> > > worry about whether or not they have the same architecture as me.  To 
> > > me, .map files still seem like a bit of a hack-job.  I.e., they work, 
> > > but haven't been integrated nicely with the renderer.
> > >
> > > Whatever the solution is, it should just work in a production without 
> > > help from people like me.  And ideally, it should just work with 
> > > mental ray standalone as well.  As genius as these files are, they're 
> > > ultimately just textures to the end user.  It shouldn't be complicated 
> > > to work with them.  Also, header information that is not intrinsically 
> > > tied to the data blob on the disk shouldn't live in the file.
> > >
> > > - Andy
> > >
> > > Halfdan Ingvarsson wrote:
> > >
> > >   
> > >
> > >      
> > >
> > > > The thing is, map files should be thought of as completely disposable.
> > > > They're very handy when you have to read the same texture over and 
> > > > over again, especially in the tiled version, but a complete dog as 
> > > > any form of interchange.
> > > >
> > > > Ideally they should only be generated at the point of render and 
> > > > then only on the machine that intends to use it. XSI does provide a 
> > > > system for automatically doing that and maybe it should be spelled 
> > > > out more clearly in the manual.
> > > >
> > > > - 1/2
> > > >
> > > > --------------------------------------------------------------------
> > > > ----
> > > > *From:* owner-xsi(at)Softimage.COM on behalf of Andy Jones
> > > > *Sent:* Tue 27-Mar-07 18:36
> > > > *To:* XSI(at)Softimage.COM
> > > > *Subject:* Re: 64 vs. 32 bit with map files?
> > > >
> > > > Shouldn't it be possible to encode data about what architecture the 
> > > > file was mapped with originally, so that an alternate encoding can 
> > > > be generated if there's a mismatch?  Clearly, workarounds for this 
> > > > problem.  It's my opinion that these "workarounds" should be 
> > > > incorporated intrinsically into the way Mental Ray (or, failing 
> > > > that,
> > > > XSI) deals with .map files (or some new superior format that 
> > > > replaces them).  As an example of one such workaround, you could 
> > > > theoretically write an "architecture switch" node for the 
> > > > rendertree, then put a bunch of differently encoded .maps behind it.  
> > > > That'd be a stupid, clunky way to solve the problem, but it would work, unless I'm missing something.
> > > > Right?
> > > >
> > > > - Andy
> > > >
> > > > Luc-Eric Rousseau wrote:
> > > >
> > > >     
> > > >
> > > >        
> > > >
> > > > > The goal of .map file to be memory-mapped directly without 
> > > > > transformation, so they are by their very nature 
> > > > > architecture-specific.  They're whole benefit is that they are 
> > > > > directly in the memory format.  If they were not, then they would 
> > > > > not have any benefits over normal file formats.
> > > > >
> > > > >
> > > > >
> > > > >       
> > > > >
> > > > >          
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Alan Jones
> > > > > >
> > > > > > ouch - architecture specific image format - who was the genius 
> > > > > > behind that idea?
> > > > > >
> > > > > > On 3/27/07, Halfdan Ingvarsson <hingvars(at)softimage.com> wrote:
> > > > > >
> > > > > >
> > > > > >         
> > > > > >
> > > > > >            
> > > > > >
> > > > > > > Yes. Map files are essentially a memory dump of the
> > > > > > >
> > > > > > >
> > > > > > >           
> > > > > > >
> > > > > > >              
> > > > > > internal image format that mental ray uses.
> > > > > >
> > > > > >
> > > > > >         
> > > > > >
> > > > > >            
> > > > > >
> > > > > > > mr will happily load them from other platforms but doesn't
> > > > > > >
> > > > > > >
> > > > > > >           
> > > > > > >
> > > > > > >              
> > > > > > make any guarantees about their usefulness in that case.
> > > > > >
> > > > > >
> > > > > >         
> > > > > >
> > > > > >            
> > > > > ---
> > > > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text 
> > > > > in
> > > > >       
> > > > >
> > > > >          
> > > > body:
> > > >     
> > > >
> > > >        
> > > >
> > > > > unsubscribe xsi
> > > > >
> > > > >
> > > > >
> > > > >       
> > > > >
> > > > >          
> > > > ---
> > > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > > > unsubscribe xsi
> > > >
> > > >     
> > > >
> > > >        
> > > ---
> > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > > unsubscribe xsi
> > >
> > >   
> > >
> > >      
> > ---
> > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > unsubscribe xsi
> >
> > ---
> > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > unsubscribe xsi
> >
> >
> >
> >    
>
>
> ---
> Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> unsubscribe xsi
>
> ---
> Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> unsubscribe xsi
>
>  


---
Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
unsubscribe xsi