Molly Rocket

[icastano]

Sean, your lecture was pretty good, the best I've seen so far at GDC this year. I'm sorry I had to leave in the middle of it, but I had to catch my train back home. I didn't get the chance to ask you some questions, but here it goes:

- What extensions to the graphics pipeline/api would you propose to accelerate this application?

- Something that would come to mind is a mechanism to find out what texture data you need, and avoid the CPU readback. Do you have any suggestion of how that mechanism should look like?

[sean]

[sean]

The other thing that harware should always do is be orthogonal.

For example, as I said it would be nice if we could just directly get at the analytic derivatives of the texture coordinates, assuming they're computed that way. This is mostly a performance orthogonality; we can compute them by hand, but if that uses finite differencing it's probably slower and less accurate. Is that really worth burning transistors on making this orthogonal (allowing that data to be forwarded from the texture unit)? Maybe not.

But here's an orthogonality issue that I've encountered twice while working on SVT: Mipmapping beyond 1x1. (Uh, assuming hardware hasn't started doing this, or been doing it all along, and I just never found out.)

Now wait, you say, how can you mipmap beyond 1x1? And if you could, is that really orthogonality?

The mipmap level above a 1x1 mipmap level would be 1x1. This generalizes the practice already visible from non-square textures, whose mipmaps go:

• 16 x 4
  
• 8 x 2
  
• 4 x 1
  
• 2 x 1
  
• 1 x 1
  

Either coordinate, if it drops to only one texel wide, simply stays 1 texel wide for the remaining mipmaps. So the "orthogonality" here is in having that just always be true, even if you hit 1x1.

The two cases where I've encountered it: with a page size of, say, 128, the coarsest mipmap your SVT can have is not 1x1, but 128x128... because the coarsest page table you can have is 1x1. If you could have additional "more coarse" tables above 1x1, you could use those as page table entries referencing additional pages, which would be incomplete. (E.g. the next page would only use a 64x64 region of the 128x128 page, etc.)

Another application for this is using a mipmap to compute a trilinear factor, by storing the mip level in the texture and using trilinear. If you need to support 10 mip levels, this requires a 1K x 1K texture, which is wasteful (and inefficient on the fine levels), and allowing many more mip levels becomes quickly infeasible. Imagine if it was 1k x 1k, but once you hit the 1x1 mip level, you could keep going "up" with coarser mip levels, each of which is still 1x1 but that's fine because its only job is to store the mip level to be trilineared. In fact, forget 1K x 1K; you'd just make EVERY level be 1x1, and the thing would use teensy amounts of memory.

(Currently I actually use one of these textures in the optimized shader that I ran on my laptop for the first pass (pre-readback) to compute the needed mipmap level.)

This is just a way of computing a function per mip level. You could also allow just passing the mip-level computation from the texture coordinates straight into the shader, and do an arbitrary function on that. That works for this case, but is less general and less flexible.

And of course you can just use partials to compute the expected mip level by hand if you have partials available (which is what I do in GLSL), so in some sense this is irrelevant to future hardware, but I wanted to point out that it COULD be running better on the older hardware if they had just bothered to make this feature orthogonal instead of assuming 'well, why would you ever want to go up past 1x1'? Thinking that you wouldn't is absolutely failing to think about your texture+shader as a general computation unit, and falling into thinking about them

[Won]

Well, it seems that alot of the complexity would go away if GPUs supported virtual texture memory (as Carmack has wanted for some time). Sean alludes to an important difference when he says that, unlike normal virtual memory, you can fudge things with placeholders, which suggests that what you need is some smart kind of non-blocking faulting behavior.

What would be cool is if you could tell the GPU to use some massive virtual texture, and as it rendered it, it used the best MIP level available. Then, you could query the hardware for the texture tiles that "faulted" and the texture tiles that "idled" and leave it to software to deal with the replacement/upload or even perform prefetch.

[sean]

Yeah, in my original draft I mentioned DX10 Virtual Textures, which appear to just be Textures using virtual memory, where the GPU memory is backed by the CPU memory (compare VM where the CPU memory is backed by disk).

That's fine (although I guess irrelevant on consoles or 3d cards that have unified memory), but it's just the most trivial thing and isn't particularly streaming/procedural generation friendly. If you tried to stream with such a system, you'd still have to chop your stuff up into textures that could fit in the CPU.

Also, I'm not sure why this needs to be "in" DX, I guess they added something to the driver model. Presumably hardware in OpenGL at least could have been doing this all along.

[Won]

I think there is an important distinction that SVTM has over straightforward virtual texture memory, and that is: faults don't block rendering. And yes, you want things to be streaming/procedurally friendly, too.

I suppose with virtual texture memory you could have something like a per-tile LOD clamp, but you would still have no way of knowing what tiles were resident since you don't actually have control over that. Which comes back the idea that faulting (and idling) textures should be queryable, so you could let the app manage the bandwidth to the card.

[sean]

Oh yeah, I forgot to comment on that.

I agree this would be cool. It seems like a giant new architecture explicitly for this specific thing, so I don't know, maybe it doesn't make sense, to be able to stream this extra data out... I'd hate to add something specifically for this technology since it's not clear how useful it will turn out to be in practice. (Maybe people use it for three years and abandon it.)

Ok, here you go: what you need is support for multiple render targets, with different targets having different levels of AA (dunno if that exists yet).

Now, here's how it works. We'll combine the passes that render the actual frame and render the data-to-read-back. In one render target, we do our regular rendering, with whatever data is available (subbing mipmaps etc). In the other render target, we'll write out what page we wanted to access.

This is pretty much exactly what you're asking for, we're just using existing facilities to implement it.

Of course, we incur a one-frame latency on the readback. That's fine for moving but it sucks for mouselook rotation. But maybe that's just the price of doing business.

The only problem is that it's actually BETTER if we render it in a separate pass because we can decouple what we render in the two cases.

Here are some ways a decoupled readback render is useful:

• If we decide we want to actually predictively render the scene a few frames in advance of the camera, we can't do that anymore. This might be crucial for disk streaming.
  
• It won't help for mouselook, but we could help rotation a little by actually rendering with a different camera, with a wider FOV, so we're always downloading stuff 20-30 degrees outside the FOV, meaning we can handle a rotation rate of 20 degrees/frame (say, at 30 hz, that's 600 degrees/second) without visible loss.
  

If we do it all in one pass, we pay a frame of latency and give up the ability to do the above decoupling. Now, if that frame of latency got us from 30hz to 60hz, that would be totally worth it in general. But if it gets us from 30hz to 60hz, but it turns out we need the predictive/extra-spread camera for visual quality or disk streaming, it sucks.

Still, it seems reasonable to try.

[TomF]

[Won]

Yeah, the more I think about it, it is definitely the logical place for it, and I think I envisioned the same async query interface. It would also help to know which resident tiles were not used (or a per tile fragment counter) so that you could do something reasonable for replacement. Although this count would necessarily be delayed by a frame, so it might make sense to keep this info on the card, and have tiles automatically replace tiles with the lowest fragment counts, which gives you a LFU-like replacement policy. But that's starting to sound very specialized.

Thinking in terms of queries, there is also NVX_conditional_render, which allows you to have associate a rendering "callback" predicated on the result of a query. This might be useful, if you are doing something like procedurally generating SVTM GPU-side, or if you want to do some kind of processing or filtering on the queries before feeding them back to the CPU.

[TomF]

[Won]

Well, to clarify: the query interface would be nice and general. It was just the auto-LFU business that I thought was too specific.

[bengarney]

[sean]

Because it depends on the partials of both coordinates. However, you can probably get away with two 1024x1 1D textures (well, one that you sample from twice, maybe that's what you meant), and take a min or a max or something, if the way the actual hardware mip computation combines them is min or max... which I think it is. I did mention that somewhere... probably in the rough draft and it got cut. :)

[bengarney]

[sean]

Oh, hey, you may be right. I was thinking you meant a 1D texture, but if you do a 2D texture that's 1024x1, yeah, I guess that just works. Nice!

If it's not about hardware/api performance stuff, yeah, we should move it elsewhere. If you have a bunch of questions, maybe just start a general question thread.