Renegade/Code/ww3d2/DX8 Status.txt

Now we've got WW3D up and running with the skeleton app.  Here is a somewhat more detailed/organized
list of what is left to do:

LIMITATIONS/POTENTIAL PROBLEMS:
- Skins need to use a limited set of rendering features?
- Polys going into the alpha system must use a limited set of rendering features.
- Additional procedural material passes cannot be applied to alpha blended meshes (no multi-pass alpha)
- Meshes only get two vertex color arrays.  Their vertex materials must be configured properly to enable them.
- HY: Simplescene supports only four global lights
- HY: WW3D Spoltlights have an approximate inverse linear attentuation instead of max linear attentuation


UPDATED TODO LIST:
agg_def:
	- Texture replacement feature needs to be implemented
assetmgr:
	- Texture cache, 
	- Dazzle loader, 
bmp2d
	- works. Code for setting aspect ratio was added but not used due to legacy - HY
boxrobj
	- Alpha blending support needed
	- Vis rendering code needed
camera
	- *** CameraClass projection convention changed to -1<x<1... Fix all code in commando and E&B!
	- Also convert all uses of CameraClass::Un_Project!
cullablematpass
	- collapsed into matpass
dazzle
	- Fully commented out
decalmsh
	- works
decalsys
	- DistAlphaVP to be converted (renegade)
dx8wrapper
	- Toggle_Windowed function commented out
	- One-time init functions commented out (TextureLoader,SegmentedLine,Dazzle)
dynamesh
	- works except for alpha support
font3d
	- works
intersec
	- Compiles but not tested, 
	- Probably contains a lot of un-used functions because all intersect calls are now routing through Cast_Ray!
line3d
	- works except for alpha sorting
mapper
	- All mappers need to properly handle the u-v array index
	- Animating1DMapper commented out
	- Axial mapper commented out
	- Silhouette mapper commented out
matrixmapper
	- All mappers need to properly handle the u-v array index
matinfo
	- Texture reduction factor access commented out
matpass
	- works except for skins and culling
mesh
	- static sort list code commented out
	- vis rendering code commented out
	- ORIENTED and ALIGNED features broken
	- dependencies code commented out
meshmdl
	- shadow rendering
	- vis rendering
	- get_deformed_screenspace_vertices commented out (Can we delete this? skins are working...)
	- get_deformed_eyespace_vertices commented out (Can we delete this? skins are working...)
	- Render_Prelit
	- needs to use the dynamic index buffer when rendering cullable material passes
meshmdlio
	- animated texture loading not implemented
	- use mesh TWOSIDED flag to set shaders on loading (and take out Jani's temp code to do this on the fly)
metalmap
	- fully commented out
pointgr
	- works except for alpha support
r3dobj
	- fully commented out - may be able to DELETE
render2d
	- works
rendobj
	- texture_reduction feature commented out - may not be needed (only have a global texture reduction factor?)
rinfo
	- surrender-specific vis code
ringobj
	- works
	- no alpha support
	- vis rendering code commented out
scene
	- static sort list code commented out
segline
	- material handling code commented out
	- rendering code commented out
shattersystem
	- works, but might not for multi-texturing.
	- Naty check if dynamesh supports multi-texturing. you changed the interface
	- a bit. HY
sphereobj
	- works
	- no alpha support
	- no viz
	- might have bug in sphere generation when sphere is too big
sr_util
	- DELETED!
statistics
	- fully commented out
texfcach
	- fully commented out
texproject
	- works except for the second stage stuff (is this generally broken?).
textdraw	
	- works. This is a legacy font system to be removed from Renegade.
texture
	- lots of miscellaneous missing features here
	- NoLOD flag to be supported for UI textures
texturefile
	- fully commented out
textureloader
	- fully commented out, this is the background loading system
txt.cpp
	- works. This is a legacy font system to be removed from E&B.
txt2d.cpp
	- works. This is a legacy font system to be removed from E&B.
vertmaterial
	- all uses of VertexMaterialClass may need to configure the UV-array source index.
	- all users of VMC must properly enable/disable lighting
	- all users of VMC must configure the diffuse and emissive color source if they want per-vertex arrays...
	- changed so the default is visible - HY
	- TODO: Make Get_Preset const so no one can modify the static presets
ww3d	
	- TextureLoader support commented out of Begin_Render
	- Statistics code commented out
	- Read_Gerd_Render_Device_Description commented out (?)		
	- Set/Get_Texture_Reduction commented out
	- Flush_Texture_Cache function commented out
	- Detect_Drivers commented out
	- On_Activate/Deactivate_App commented out
	- Update_Pixel_Center commented out
General:
	- Add support for multiple texture stages
	- Create sort system
	- Add support for exclusion masks. The way this works is that we support exclusion masks differently at the top-level object level and the low-level rendering object level (note that this means that the mid-levels will get skipped over unless we collapse the container bits together with the low-level object bits. However we probably cannot do this, since the subobjects bits may be more permissive than the container's bits. Should we worry about this?). At the top level, when the light environment is constructed, we filter the light's exclusion bits vs. the light environments' exclusion bits (passed into the construct call, stored in the light environment at creation, or something). If it does not pass, it is not inserted into the light environment. When we insert a light into the light environment, we also store its exclusion bits (perhaps we only do this if "light aggregation" is turned off - see below). Then the low-level code  for applying the current light environment receives the low-level robj's exclusion bits, compares them, and decides whether to actually set the light into D3D. For this to work properly, we need a flag to prevent lights which are rejected from the light environment because it is full of more-important lights from contributing to the ambient.
	- Change the "Get_Texture_Array" functions in MeshMatDescClass to private, change name to PEEK!
	- have texture category sorting functions take account of both texture stages

NOTE: We have three enums/#defines/consts for the number of texture stages:
RENDERER_MAX_TEX_STAGES, MeshMatDescClass::MAX_TEX_STAGES, MeshBuilderClass::MAX_STAGES.
We are even explicitly comparing two of them and asserting if they are different in
dx8renderer.cpp. There is a similar situation with DX8FVFCategoryContainer::MAX_PASSES,
MeshMatDescClass::MAX_PASSES and MeshBuilderClass::MAX_PASSES. We need to clean this up!

Features we are removing support for (will need application changes):
	- indirect textures (we will provide a (slow) function to swap textures)


Ok, the WW3D2 skeleton program compiles and links now with no references to SR.  Some files I took more care on, others I blocked out the entire file for later.  Every piece of code is wrapped with #ifdef WW3D_DX8 with a matching #endif and comment, there should be no typo's in the #ifdef's because I used a macro to insert them all (so we can do searches and safely assume that we see all of them).  
Here are some (raw) notes I took while commenting out all references to SR:
 
X - VertexMaterialClass - need to separate from sr (SOON)
X - Basic texture functionality (loading from files, etc)
- More advanced texture handling code needs to be re-implemented, but try to maintain our current interfaces as much as possible.  Many issues here: background thread loader, texture file cache, texture LOD, indirect textures ?... (SOON)
X - ShaderClass - just convert to our own, mimicing the bitfield that SR used for now (SOON)
- Exclusion masks - this is not supported but some application code is probably using it. We need to decide what to do here.
- Camera aligned and camera oriented meshes - the way we used to do them required the mesh push transform code to change the camera matrix, which doesn't work well with our current scheme. We need to think how to support this stuff.
X - MaterialInfo Class - can un-comment when we have the above material systems in place, evaluate where this is used?
- sr_util - all of these functions un-needed?  (no more As_srVector3 :-)
- agg_def.cpp - all texture features commented out
X - box render objects completely commented out - materials, gerd
- texturefcache - commented out
- TextureFileClass - commented out
X - DynaMeshClass - commented out, need materials, gerd replacement, new rendering interface before we can bring this back.  A *lot* of stuff uses DynaMeshClass...  (shatter system, text systems, etc)
- Dazzles - commented out, gerd, materials
- streaming textures - commented out, new texture code needed
- texture creation functions in assetmanager commented out
X - Bitmap2DObjClass - commented out, do we need this any more? - depends on dynamesh
X - BW Render, used for shadow rendering, uses some srVectorProcessor stuff
X - Camera - internals commented out
X - MatPass, CullableMatPass - all procedural material passes commented out, design new, low-level rendering interface, then port these to it.  
X - passdata.h - no sure what this is for, it is commented out
X - meshmodel - all material functions commented out, will also probably re-write the guts of meshmodel to store "sub-meshes" rather than material arrays.  depends on design of new low-level rendering system. (SOON)
X - decal system - all material handling commented out, rendering commented out, depends on new low-level rendering system
X - matrix mapper - this is used to generate u-v coordinates in a surrender vertex processor, new system needs to support this functionality, then we can port this (well, it will use DX8 features to do the same thing in HW...)
X - intersec.h - commented out, Eric Cosky's collision code, do we need this any more?
X - dx8renderer - uses srVectorProcessor
X - dx8wrapper - uses Gerd
X - fog.cpp - all fog objects commented out, probably won't implement fog with a render object, just have fog settings be stored in the scene
X - font3d - surface loading!  textures
X - lightclass - hmmmm, re-consider lighting...
X - lightenv - uses lightclass as input.
X - line3d - uses srModeller, srMeshModel.  Do we need this any more?
X - mapper.cpp - srVertexProcessor, need to convert to HW form (DELETED)
X - mesh - materials, gerd transforms, meshmodel
X - meshgeometry - uses srVectorProcessor in a couple of places
X - meshmdl - lots of material and gerd stuff, re-design?
- metalmap - uses vectorprocessor, textures
X - pointgroups - surrender types, gerd rendering, materials
X - particle buffer - needs point group
X - particle emitter - materials, particle buffer
X - particle loader - needs particle classes, materials
X - polyinfo - don't know what this is?
X - projector - built on top of matrix mapper, used for projected textures
X - r3dobj - image3dobj, can we delete this?
X - render2d - textures, gerd calls
X - rendertype - lots of GERD stuff, Jani is this an "obsolete" optimization?
- rinfo - material passes, gerd stuff, vis commands which use ext-functions for the surrender software rasterizer.
X - ring render object - materials, gerd rendering calls
X - scene - fog settings, don't use FogClass any more, just plug fog settings into the new low-level interface...
- scene - ensure that the scene ambient light value gets added into the light environment's effective ambient (E&B uses the scene ambient and currently it doesn't do anything)
- segline - custom gerd pipeline, materials
X - shatter system - dynamic meshes, materials.
X - sphere - materials, gerd rendering calls
- statistics - Jani's texture stats, new texture system must support this.
- texfcache - completely commented out
X - texproject - needs procedural material pass interface, vertex processor...
X - textdraw - textures, dynamic meshes
- texture loader - is this the backround thread loader?  commented out
X - txt.cpp - uses textures
X - txt2d.cpp - uses txt, dynameshes
X - vertexbuffer.cpp - not sure what this is, uses srVectorProcessor
- ww3d - lots of gerd calls, surrender types, screen capture, movie capture, texture reduction, statistics
- Investigate cost of BeginScene-EndScene, set viewport has to occur right after beginScene... Should we minimize our viewport changes?   If the cost is high, we might need to separate the viewport definition from the camera...
- Silhouette mapper un-implemented
- Axial mapper un-implemented
- How should handle the extra render states for dealing with TexGen?  If we apply these states, there is nothing to remove them!
 -- HY render states for texgen and texmaterial are encapsulated in vertex material & mapper
 -- if the vertex material detects a null mapper the defaults are set.
- Make Dynamesh support Cast_Ray so that the E&B intersection code works!


Shaders have been converted to Shader->Apply() and the state is being tracked by the shader class.
Materials have been converted to Material->Apply() which calls Mapper->Apply()

Now, some big issues we need to think about are (in no particular order):
 
SOME GOALS -   Maintain as many existing class interfaces as we can so that our apps don't have much re-coding!
                           Input data is existing W3D files, no format changes or new tools!  (for now anyway :-)
                           Incorporate the ideas in Jani's DX8 code, grouping small meshes together in larger VBs, etc
 
GERD Replacement - our new low-level rendering interface needs to be the replacement for everywhere we are using the GERD.  It needs to supply all of the features that we are using in the GERD currently while allowing us to use Jani's mesh optimizations.  There is a lot to think about here and almost everything depends on us doing this right.  It also might need to serve as an abstraction layer so that I can implement a software VIS-renderer behind it.  
SOLUTION: No GERD replacement, no abstraction layer.  DX8Wrapper will be a thin insulation layer though.

VIS - I'll have to re-implement the rendering code for VIS.  It will be a 32bit, solid fill, zbuffering software rasterizer.  Not too hard but still a bit of work.  Needs to be done in a way that all render objects can be rendered through it easily.
SOLUTION: Vis rendering will branch at the render object level to a new custom vis rasterizer possibly based on the bw shadow code

TEXTURES - there are many features in the textures system, the "file cache", background loading, LOD, tga loading, global texture reduction factor, etc.  
SOLUTION: Naty is working on this system.
 
MESHMODEL - I think we'll need to do some load-time conversion of all of our mesh-models into a form similar to Jani's dx8 mesh code.  More thought needs to be put into this.
 
ALPHA SORTING - This issue becomes especially complex when you think about mixing different types of render objects.  This issue will influence how we design the low level rendering interface.  
SOLUTION: There will be a custom alpha sorting/rendering system.  RenderObjects will be allowed to talk to either DX8 or this system.

UNIX - It looks like Neal has put some effort into making ww3d compile under Unix.  We will probably make life very difficult for him if there are DX8 calls all over the place :-)
SOLUTION: UNIX version of E&B will not use WW3D2.  It only needs collision detection so we'll implement a custom solution.


Optimizations:
==============
* Currently we search a linear list for textures with the same name every time we ask
the asset manager for a new texture. We should probably use the hashtable template
for this instead.
* The VectorProcessorClass stuff is completely unoptimized - we might want to
optimize this in future.
 

********* NEW STUFF ************
DX8 Features which we currently do not support and might want to in future:
* Different modes than clamp and wrap - DX8 exposes all kinds of cool modes like mirror, mirror once - do we expose these in our tools? How well are they supported in hardware? If they are not universally supported then we probably don't want to expose them...
* Cube and volume textures - support via inheriting from shared base class with TextureClass?
* Anisotropic filtering: currently trilinear is enabled in the TextureClass but not in our tools. There are issues to think about for both trilinear and anisotropic like: not all hardware supports it, some hardware supports it if there is just one texture but not otherwise, or with a slowdown etc. - this is probably not something we want the artist to specify directly: perhaps the artist can mark those meshes which would benefit from "advanced filtering", and we turn it on selectively depending on hardware?

We can do fancy texture matrix operations "for free" now. (Only on Hardware T&L cards!)