Re: 64 vs. 32 bit with map files?

["peter boeykens" <peter_b(at)skynet.be>]

to add my 2 cents,

I dislike using a seperate gui for the conversion, but also dislike the 
invisible "all or nothing" approach of the render preferences..
Also once in a while .map's give problems, so I prefer to decide on a case 
by case basis what to use, and which options to include.
I prefer to have the .maps next to the editable image, with the same 
filename. (actually, I keep a .psd, a .pic and a .map at the same location 
and same name for important textures)
This way I can switch back and forth by just changing the .pic extension to 
.map whenever I want. I do this in the gui, but this could be done in the 
.scntoc too I think for batch rendering purposes?

I would really like to have two paths in a clip.
One to the image in a editable format like the pic or tiff or so, and 
another to the .map file, with a toggle for which one to use. (very much 
like model resolutions) and perhaps a "generate .map file" button in the 
clip ppg, with checkboxes for filtering and all, that would create the .map 
on command.
And while we're at it, I'd like a third path, to the viewport display 
picture. For very big files, say 10k, I use a downsized version.
Adding this in the rendertree and specifying it in the material node works 
fine, but it could be more streamlined.

By having multiple versions and resolutions in one clip, the amount of clips 
exisiting in a scene could be greatly reduced, and performance could be 
greatly optimised by choosing the format that fits the task (interaction, 
quick n dirty preview, local rendering, remote rendering...)





----- Original Message ----- 
From: "Andy Jones" <andy(at)thefront.com>
To: <XSI(at)Softimage.COM>
Sent: Wednesday, March 28, 2007 5:48 PM
Subject: Re: 64 vs. 32 bit with map files?


Problems with it (some of which you might have easy solutions for?):

1) If I turn it on, it writes the files to a subdirectory within the
directory the texture is in.  So I have to write scripts to cache the
textures to a local drive anyway.  Otherwise I'll kill my network.
2) The problems with architecture persist when using .mi2 files to, say,
send stuff to a 3rd party farm.
3) While verifying items on this list, XSI 6.0 crashed while trying to
open the preferences :(
4) When I change the filtering settings, it doesn't force an update to
the .map file.  I.e., if I generate it without filtering on, it is
forever stuck that way.  So, if I want elliptical filtering to work
efficiently, I also have to turn on  multi-resolution texture on the
clip.  So, what do I do if I want an image node that does elliptical
filtering and another image node that does no filtering whatsoever?
Answer -- make 2 copies of the texture.
5) The filtering level is hard-coded in the file format because .map
files contain an entire texture struct instead of just the data part of
it.  So I can't use the same texture file with multiple mult-resolution
texture biases.

Currently, we work around these limitations by generating our own .map
files to a local directory, always forcing pyramidal filtering and
exclusively using elliptical filtering.  It works okay for us, but it's
less than ideal.  We moaned about it about 1.75 years ago, but
eventually just let it go, since it basically worked okay.  To me, this
architecture thing is a final straw, making it time to finally deal with
some of this stuff.

And, in case it hasn't been clear, I still feel like the solution to
this should be coming from Mental Images -- not you guys.  That way .mi2
files can work nicely too.  But of course, I appreciate all of your
efforts to fix it yourselves when Mental Images slacks off.

To me, the ideal solution is to implement something like a .map file
that maps a more pure data blob to disk in a directly readable way, them
implement a texture struct that reads its data out of these blobs.
Then, a locally overridable local flag would be included to optionally
map all, none or some textures to this format before the render (obeying
the syncing rules XSI uses, but also looking at anything in the mi2 file
that might change the data encoded).  And the files would go to a
subfolder of the texture's location, or a specified location that can be
set in the .mi2 file.  Ideally, these blobby things could be used by
shaders as well, to allow for big arrays of data without taking up a ton
of memory.

So, .map version 1.0 (the current imlementation is version 0.6).

- Andy

Halfdan Ingvarsson wrote:

> Like I said earlier, XSI already has a facility to do this for you. Just 
> make sure it's not enabled on the artists' workstations but only on the 
> render nodes (you can force it in the preferences). It automatically 
> converts any texture to .map files based on timestamps and a size 
> threshold.
>
> See: "mental ray Render Options->Optimization->Convert Textures to Memory 
> Maps." and "Preferences->Rendering->Images->Map Conversion"
>
> - ½
>
> -----Original Message-----
> From: owner-xsi(at)Softimage.COM [mailto:owner-xsi(at)Softimage.COM] On Behalf Of 
> Andy Jones
> Sent: 28-Mar-07 10:29
> To: XSI(at)Softimage.COM
> Subject: Re: 64 vs. 32 bit with map files?
>
> So, we're in agreement then.  Productions need a tool that manages .map 
> files for them and makes them transparent to the user.  The way I see it, 
> there are two options:
>
> Option 1:  Everyone hires me, Alan or some other Jones.
> Option 2:  XSI and/or Mental Images provides a solution.
>
> As tempting as option 1 is to me in the short term, I'd rather it be option 
> 2 so that in the long run XSI doesn't get replaced by something more 
> user-friendly and my expertise beyond .map files is still useful.
>
> - Andy
>
> Halfdan Ingvarsson wrote:
>
>
> > Don't forget problems of endianness, or even non-IEEE754 systems, where 
> > your scheme will completely break down. Bear in mind that mental ray works 
> > on more than just x86 chips. Also, user-friendliness has never really been 
> > their forte ;)
> >
> > I dealt with it in production way back when and I never thought it much of 
> > a problem. I just did the local conversion on the render node and used 
> > that to sync up with the server, much like what Andy describes. The map 
> > files were never ever used by the artist at any point and neither should 
> > they.
> >
> > You kids today have it too easy :-)
> >
> > - ½
> > -----Original Message-----
> > From: owner-xsi(at)Softimage.COM [mailto:owner-xsi(at)Softimage.COM] On Behalf 
> > Of Alan Jones
> > Sent: 28-Mar-2007 06:05
> > To: XSI(at)Softimage.COM
> > Subject: Re: 64 vs. 32 bit with map files?
> >
> > Halfy and Luc-Eric,
> >
> > The problem isn't our understanding how it works internally or the reasons 
> > why it is so. We're dealing with how to use it in production and while yes 
> > it's possible for us to do things to make it work, such a basic feature 
> > should work without any user intervention.
> >
> > I don't see the reason why are format which does work as a memory dump 
> > couldn't be made to work on both 32bit and 64bit - 100 megabytes of 32bit 
> > floats in a row would be accessed identically on either platform. I'm sure 
> > the map file is more complex than that, but it should be possible to make 
> > it work.
> >
> > Cheers,
> >
> > Alan.
> >
> > On 3/28/07, Andy Jones <andy(at)thefront.com> wrote:
> >
> >
> > > Instead of using XSI's automatic generation, we generate them as needed 
> > > for a scene using a script, then we basically rsync them into a cache on 
> > > local machines from the render script.  This seems to be faster for us 
> > > than regenerating them every time, which can take quite a while.  Plus we 
> > > have more control over how the .maps are generated.  I suppose we could 
> > > also modify our scripts to have them do the rsync on the original 
> > > textures, then regenerate the .map files on the local machines, but I 
> > > don't know that this would necessarily be that faster...  So, yes, we're 
> > > perfectly capable of coding a solution that handles .map files elegantly.
> > >
> > > But it's exactly my point that it's possible to work around the issue of 
> > > .map files that are generated for the wrong architecture by trashing 
> > > them, regenerating them, translating them, and so on.  So, the basic 
> > > memory-footprint-on-disk functionality of the .map files should ideally 
> > > be encapsulated within some sort of data structure and algorithms that 
> > > provide a more seamless solution.  I'm thinking this behavior should 
> > > reside in Mental Ray, so that I can, say, send a scene and a bunch of 
> > > .map textures to a renderfarm somewhere and not have to worry about 
> > > whether or not they have the same architecture as me.  To me, .map files 
> > > still seem like a bit of a hack-job.  I.e., they work, but haven't been 
> > > integrated nicely with the renderer.
> > >
> > > Whatever the solution is, it should just work in a production without 
> > > help from people like me.  And ideally, it should just work with mental 
> > > ray standalone as well.  As genius as these files are, they're ultimately 
> > > just textures to the end user.  It shouldn't be complicated to work with 
> > > them.  Also, header information that is not intrinsically tied to the 
> > > data blob on the disk shouldn't live in the file.
> > >
> > > - Andy
> > >
> > > Halfdan Ingvarsson wrote:
> > >
> > >
> > >
> > > > The thing is, map files should be thought of as completely disposable.
> > > > They're very handy when you have to read the same texture over and over 
> > > > again, especially in the tiled version, but a complete dog as any form 
> > > > of interchange.
> > > >
> > > > Ideally they should only be generated at the point of render and then 
> > > > only on the machine that intends to use it. XSI does provide a system 
> > > > for automatically doing that and maybe it should be spelled out more 
> > > > clearly in the manual.
> > > >
> > > > - 1/2
> > > >
> > > > --------------------------------------------------------------------
> > > > ----
> > > > *From:* owner-xsi(at)Softimage.COM on behalf of Andy Jones
> > > > *Sent:* Tue 27-Mar-07 18:36
> > > > *To:* XSI(at)Softimage.COM
> > > > *Subject:* Re: 64 vs. 32 bit with map files?
> > > >
> > > > Shouldn't it be possible to encode data about what architecture the file 
> > > > was mapped with originally, so that an alternate encoding can be 
> > > > generated if there's a mismatch?  Clearly, workarounds for this problem. 
> > > > It's my opinion that these "workarounds" should be incorporated 
> > > > intrinsically into the way Mental Ray (or, failing that,
> > > > XSI) deals with .map files (or some new superior format that replaces 
> > > > them).  As an example of one such workaround, you could theoretically 
> > > > write an "architecture switch" node for the rendertree, then put a bunch 
> > > > of differently encoded .maps behind it.  That'd be a stupid, clunky way 
> > > > to solve the problem, but it would work, unless I'm missing something.
> > > > Right?
> > > >
> > > > - Andy
> > > >
> > > > Luc-Eric Rousseau wrote:
> > > >
> > > >
> > > >
> > > > > The goal of .map file to be memory-mapped directly without 
> > > > > transformation, so they are by their very nature architecture-specific. 
> > > > > They're whole benefit is that they are directly in the memory format. 
> > > > > If they were not, then they would not have any benefits over normal 
> > > > > file formats.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Alan Jones
> > > > > >
> > > > > > ouch - architecture specific image format - who was the genius behind 
> > > > > > that idea?
> > > > > >
> > > > > > On 3/27/07, Halfdan Ingvarsson <hingvars(at)softimage.com> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Yes. Map files are essentially a memory dump of the
> > > > > > internal image format that mental ray uses.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > mr will happily load them from other platforms but doesn't
> > > > > > make any guarantees about their usefulness in that case.
> > > > > ---
> > > > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in
> > > > body:
> > > >
> > > >
> > > > > unsubscribe xsi
> > > > ---
> > > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in 
> > > > body:
> > > > unsubscribe xsi
> > > ---
> > > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in 
> > > body:
> > > unsubscribe xsi
> > ---
> > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > unsubscribe xsi
> >
> > ---
> > Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> > unsubscribe xsi
>
>
> ---
> Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> unsubscribe xsi
>
> ---
> Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
> unsubscribe xsi


---
Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
unsubscribe xsi 


---
Unsubscribe? Mail Majordomo(at)Softimage.COM with the following text in body:
unsubscribe xsi