Collection of bookmarked posts related to shader hacking

In light of the fact that I'm no longer able to dedicate any significant amount of time to working on fixes (and, unfortunately, I'm not the only recent 'retiree'), and that we seem to have several new eager and capable individuals looking to learn the tricks of the trade of shaderhacking, I want to be able to provide whatever support I could for the transition to this generation.

There's no denying the fact that in order to learn the fundamentals of shader hacking that there's a LOT to read, learn, and understand. I, nor anyone else can change that, but what I can do is help provide a listing of many useful posts that I had uncovered while I had lots of time on my hands. I spent hours scouring the two largest threads on the Geforce 3DVision forum for this information, and now I'll share it all so that you don't have to!

Mind you, this is all supplementary reading for AFTER you've completed and understood the basics that are laid out and taught through bo3b's School for Shaderhacking. Even though it teaches the use of Helixmod on DX9 titles, and most newly release games are DX11 and require the use of 3DMigoto, which uses an entirely different programming language (HLSL, which is essentially the same as C++ in many regards), those lessons will provide the foundation on which all of this other information builds upon.

I've grouped and categorized the bookmarks to help with organization and navigation. But first I'll start with the holy grail; the one single post that probably taught me more than all the rest combined! (well, again, this was after learning and understanding the basics. Won't make any sense until then):

Fixing shadows, various transformation stages, and fixes for each

... and now on to the rest.

Fix examples

And lastly, I provided a lengthy explanation in response to a PM I received from an inquiring soul regarding fixing HUDs, that goes over some basics in some easy to understand language. Since I can't provide a bookmark to the PMs, I'll copy and past the discussion here:

PM Received:

If you have time could you do me a favour and take a quick look at the shader below, and if you can point me in the right direction for some example code to stereoize it. It is just the HUD shader which id like to push into depth a little bit. I always learn best by comparing examples, so it would be a great help if you could spend a couple of minutes, I'd really appreciate it.


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

cbuffer cb0 : register(b0)

{

  float4 cb0[4];

}









// 3Dmigoto declarations

#define cmp -

Texture1D IniParams : register(t120);

Texture2D StereoParams : register(t125);





void main( 

  float2 v0 : POSITION0,

  float2 v1 : TEXCOORD0,

  float4 v2 : COLOR0,

  float2 v3 : TEXCOORD1,

  out float4 o0 : SV_POSITION0,

  out float2 o1 : TEXCOORD0,

  out float2 p1 : TEXCOORD1,

  out float4 o2 : COLOR0)

{

  float4 r0;

  uint4 bitmask, uiDest;

  float4 fDest;



  r0.xyzw = cb0[1].xyzw * v0.yyyy;

  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;

  o0.xyzw = cb0[3].xyzw + r0.xyzw;

  o1.xy = v1.xy;

  p1.xy = v3.xy;

  o2.xyzw = v2.xyzw;

  return;

}



/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

//

// Generated by Microsoft (R) D3D Shader Disassembler

//

//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

//

//

// Input signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// POSITION                 0   xy          0     NONE   float   xy  

// TEXCOORD                 0   xy          1     NONE   float   xy  

// COLOR                    0   xyzw        2     NONE   float   xyzw

// TEXCOORD                 1   xy          3     NONE   float   xy  

//

//

// Output signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// SV_POSITION              0   xyzw        0      POS   float   xyzw

// TEXCOORD                 0   xy          1     NONE   float   xy  

// TEXCOORD                 1     zw        1     NONE   float     zw

// COLOR                    0   xyzw        2     NONE   float   xyzw

//

vs_5_0

dcl_globalFlags refactoringAllowed

dcl_constantbuffer cb0[4], immediateIndexed

dcl_input v0.xy

dcl_input v1.xy

dcl_input v2.xyzw

dcl_input v3.xy

dcl_output_siv o0.xyzw, position

dcl_output o1.xy

dcl_output o1.zw

dcl_output o2.xyzw

dcl_temps 1

mul r0.xyzw, v0.yyyy, cb0[1].xyzw

mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw

add o0.xyzw, r0.xyzw, cb0[3].xyzw

mov o1.xy, v1.xyxx

mov o1.zw, v3.xxxy

mov o2.xyzw, v2.xyzw

ret 

// Approximately 0 instruction slots used



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

... and my response:

So, first off, let's cover one of the basic things, which is the input/output declarations that comes right after the "void main(" line

float2 v0 : POSITION0,
float2 v1 : TEXCOORD0,
float4 v2 : COLOR0,
float2 v3 : TEXCOORD1,
out float4 o0 : SV_POSITION0,
out float2 o1 : TEXCOORD0,
out float2 p1 : TEXCOORD1,
out float4 o2 : COLOR0

In pretty much all PS and VS shaders, inputs begin with the letter 'v' and outputs begin with the letter 'o' (or, as in this case, sometimes 'p' as well, but I've really never come across a case where I've had to modify those ones). Inputs are usually coordinates or data that has been passed into this shader for it's use to calculate something, and outputs are what that shader has calculated and are passing out to the next stage in the rendering pipeline.

Often, there will be more than one type of input and/or output, and they have different properties of what kind of data they hold. This shader uses most of the common types: SV_POSITION0 contains the position of the vertex on the screen, TEXCOORD# will contain data on the texture(s) that is being mapped/layered into the vertex, and COLOR#, as you've probably guessed, contains some colour data (although that we usually don't need to bother with, since we're really only ever concerned with correcting the positioning of things for stereo purposes).

In this case, it's a vertex shader, so this shader is responsible for calculating where all of vertexes related to this particular shader will appear on the screen. If this was for an actual 3D model of some sort, the inputs would be carrying data related to where the object exists in the game world, and the vertex shader would be converting that into which pixels on the screen they will appear on. In this case, though, it's a HUD shader so it's a bit more simplified, so the inputs are probably just static values, which is why there is not a lot of calculations going on, and you see a lot of "output = input" stuff happening.

Generally speaking, when we want to stereoize something that is not in stereo at all, we will want to add some amount of separation value to the position of the vertex (which, if we've been paying attention, in this shader is o0 because it was declared as SV_POSITION).

If the HUD items are completely 2D, then they have 0 depth (or 0 separation) and are considered at the front of the screen. Think of "1 separation" representing the entire range of depth all the way to "infinity", or otherwise the back of the screen. So if we added the full value of separation to the output, it would shift from being at the very front of the screen to the very back of the screen now. That usually would not be ideal, so instead we might add only 0.5 separation (half), which would effectively put the HUD items at 50% depth, or halfway into the screen, or maybe just 0.25 at 25%, or 0.10 for 10%, etc. (or even better, we might set up a button that will change the values on the fly, but that's a little more advanced and we'll get to that later).

So now that I've covered all the theory, here's how you actually DO it. Thankfully, since this is a DX11 game, it's a lot easier since HLSL code is a lot easier to work with than ASM code.

On line 32, we see the output position gets it's value's calculated: o0.xyzw = cb0[3].xyzw + r0.xyzw;
Since we simply want to add some separation to it (we're not actually "fixing" something that's broken), that's all perfectly fine. At any point after line 32, but before line 36 (the "return;" function, which is essentially the equivalent of "the end" in code) we need to add our separation.

Now, another quick thing to note is that o0 has 4 different values:
x - the coordinate along the horizontal left/right axis
y - the coordinate along the vertical up/down axis
z - the coordinate along the depth in/out of the screen axis
w - kinda the same thing as z, but during certain calculations

Now, we might think that we would want to be adjusting the z or w axis in order to push things further back, but that's not correct. We are almost always adjusting the x coordinate instead (because separation is a left/right thing). So, in this case, we want to add some separation to the o0.x coordinate.

Now, before we can add separation, we first need to create a variable (aka a container) that will contain the value of separation. Fortunately, 3DMigoto automatically provides us with a value for separation (and even convergence too, but that's not necessary for this operation), we simply need to 'declare' that we are going to use it. To declare a variable, at any point inside the main operation (between lines 25 and 36), we need to add the following line of code:

float4 stereo = StereoParams.Load(0);

"stereo" now holds 4 different pieces of data, although you will probably only ever be concerned with 2 of them:

stereo.x - contains separation
stereo.y - contains convergence

Some people, like bo3b, encourage taking an additional step for the sake of clarity, which would be adding the next 2 lines of code:

float separation = stereo.x;
float convergence = stereo.y;

So now you actually have 2 variables named "separation" and "convergence" instead of having to use "stereo.x" and "stereo.y" which might be confusing for you. I, personally, don't bother with this extra step, so if you ever review the code I use in my fixes, you'll see I simply use "stereo.x" for separation. I'll provide you with both examples in just a moment.

Ok, so now we've declared our variable for separation, we are now able to add that to the output. So, again, any time after line 32, but before line 36, we will do this. For the sake of simplicity we will be adding 0.5 separation to push it 50% into the screen, which should be a noticeable enough shift to visibly see, without being too far in that it will probably be too far. We do this with the following line of code:

o0.x = o0.x + separation * 0.5;

or

o0.x = o0.x + stereo.x * 0.5;

In actuality, though, this can be simplified a little bit to be:

o0.x += separation * 0.5;

or

o0.x += stereo.x * 0.5;

Because the += operator means to add whatever this value is to the existing value. There are equivalent operators -= *= and even /=.

So, finally, lets round this all up. Here are 2 perfectly equivalent ways of doing this:

Easy to understand way:


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

cbuffer cb0 : register(b0)

{

  float4 cb0[4];

}









// 3Dmigoto declarations

#define cmp -

Texture1D IniParams : register(t120);

Texture2D StereoParams : register(t125);





void main( 

  float2 v0 : POSITION0,

  float2 v1 : TEXCOORD0,

  float4 v2 : COLOR0,

  float2 v3 : TEXCOORD1,

  out float4 o0 : SV_POSITION0,

  out float2 o1 : TEXCOORD0,

  out float2 p1 : TEXCOORD1,

  out float4 o2 : COLOR0)

{

  float4 r0;

  uint4 bitmask, uiDest;

  float4 fDest;



  r0.xyzw = cb0[1].xyzw * v0.yyyy;

  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;

  o0.xyzw = cb0[3].xyzw + r0.xyzw;

  o1.xy = v1.xy;

  p1.xy = v3.xy;

  o2.xyzw = v2.xyzw;



float4 stereo = StereoParams.Load(0);

float separation = stereo.x;

float convergence = stereo.y;



o0.x = o0.x + separation * 0.5;



  return;

}



/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

//

// Generated by Microsoft (R) D3D Shader Disassembler

//

//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

//

//

// Input signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// POSITION                 0   xy          0     NONE   float   xy  

// TEXCOORD                 0   xy          1     NONE   float   xy  

// COLOR                    0   xyzw        2     NONE   float   xyzw

// TEXCOORD                 1   xy          3     NONE   float   xy  

//

//

// Output signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// SV_POSITION              0   xyzw        0      POS   float   xyzw

// TEXCOORD                 0   xy          1     NONE   float   xy  

// TEXCOORD                 1     zw        1     NONE   float     zw

// COLOR                    0   xyzw        2     NONE   float   xyzw

//

vs_5_0

dcl_globalFlags refactoringAllowed

dcl_constantbuffer cb0[4], immediateIndexed

dcl_input v0.xy

dcl_input v1.xy

dcl_input v2.xyzw

dcl_input v3.xy

dcl_output_siv o0.xyzw, position

dcl_output o1.xy

dcl_output o1.zw

dcl_output o2.xyzw

dcl_temps 1

mul r0.xyzw, v0.yyyy, cb0[1].xyzw

mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw

add o0.xyzw, r0.xyzw, cb0[3].xyzw

mov o1.xy, v1.xyxx

mov o1.zw, v3.xxxy

mov o2.xyzw, v2.xyzw

ret 

// Approximately 0 instruction slots used



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

and here's how my code might look like:


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

cbuffer cb0 : register(b0)

{

  float4 cb0[4];

}









// 3Dmigoto declarations

#define cmp -

Texture1D IniParams : register(t120);

Texture2D StereoParams : register(t125);





void main( 

  float2 v0 : POSITION0,

  float2 v1 : TEXCOORD0,

  float4 v2 : COLOR0,

  float2 v3 : TEXCOORD1,

  out float4 o0 : SV_POSITION0,

  out float2 o1 : TEXCOORD0,

  out float2 p1 : TEXCOORD1,

  out float4 o2 : COLOR0)

{

  float4 r0;

  uint4 bitmask, uiDest;

  float4 fDest;



  r0.xyzw = cb0[1].xyzw * v0.yyyy;

  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;

  o0.xyzw = cb0[3].xyzw + r0.xyzw;



float4 stereo = StereoParams.Load(0);

o0.x += stereo.x * 0.5;



  o1.xy = v1.xy;

  p1.xy = v3.xy;

  o2.xyzw = v2.xyzw;

  return;

}



/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

//

// Generated by Microsoft (R) D3D Shader Disassembler

//

//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016

//

//

// Input signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// POSITION                 0   xy          0     NONE   float   xy  

// TEXCOORD                 0   xy          1     NONE   float   xy  

// COLOR                    0   xyzw        2     NONE   float   xyzw

// TEXCOORD                 1   xy          3     NONE   float   xy  

//

//

// Output signature:

//

// Name                 Index   Mask Register SysValue  Format   Used

// -------------------- ----- ------ -------- -------- ------- ------

// SV_POSITION              0   xyzw        0      POS   float   xyzw

// TEXCOORD                 0   xy          1     NONE   float   xy  

// TEXCOORD                 1     zw        1     NONE   float     zw

// COLOR                    0   xyzw        2     NONE   float   xyzw

//

vs_5_0

dcl_globalFlags refactoringAllowed

dcl_constantbuffer cb0[4], immediateIndexed

dcl_input v0.xy

dcl_input v1.xy

dcl_input v2.xyzw

dcl_input v3.xy

dcl_output_siv o0.xyzw, position

dcl_output o1.xy

dcl_output o1.zw

dcl_output o2.xyzw

dcl_temps 1

mul r0.xyzw, v0.yyyy, cb0[1].xyzw

mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw

add o0.xyzw, r0.xyzw, cb0[3].xyzw

mov o1.xy, v1.xyxx

mov o1.zw, v3.xxxy

mov o2.xyzw, v2.xyzw

ret 

// Approximately 0 instruction slots used



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

And that should effectively teach you everything you need to know to add a static value to the HUD. Let me know if that works (and maybe play around with the amount of separation to find what is a good amount), and if you understand all that then we can discuss adding a button to cycle values, or other types of fixes.

... and some Q&A related to frame analysis:

Q: I might need to pick your brain a little bit over that frame analysis stuff. I got it going and did a dump, but I am not sure how to identify which shader it is

A: Each file that gets dumped will have have the shaders that are used in the filename, for example:

000122-o0=!MU!=957b6ea2-vs=b229b3faa24a4aac-ps=ef7eaa7c9cab3eb3.jps

can be broken down as

000122 - the draw call number

o0 - which output from the vertex shader that this draw call is pertaining to. Sometimes it will be o1, o2, etc

!MU! - flags that may indicate an error in the texture hash to follow. Not important until you start looking at doing advanced things like texture filtering.

957b6ea2 - texture hash

vs=b229b3faa24a4aac - vertex shader b229b3faa24a4aac is being used

ps=ef7eaa7c9cab3eb3 - pixel shader ef7eaa7c9cab3eb3 is being used

Q: Is there a option to dump all shaders?

A: Yes, although I'm not sure if you are referring to dump EVERY shader a game uses, or simply all the shaders currently being used in a scene. There are ways for each.

To dump every shader a game uses, change the following options in the d3dx.ini file to:

export_fixed=1
export_shaders=1
export_hlsl=2
dump_usage=1

Then all the shaders will be dumped into a folder called ShaderCache when you first launch it (which might take a minute or two). Some games, however, will only dump shaders as they are generated, rather than everything all at once, which is a pain.

To dump all the shaders in a scene, well, there is not a built in method to do that, but you can manually just cycle through every shader and dump each one. Of course that is tedious, so what I suggest is to create a macro of some sort (I use Razer Synapse since I have a Razer Naga mouse, but I could also have used MadCatz software for my keyboard, or Logitech software, or even AutoHotkey) that simply alternates between numpad 2 and 3 to dump all pixel shaders, and numpad 5 and 6 to dump all vertex shaders. Just be sure to make note of the timestamp on the files generated to know which ones you've dumped, or simply remove all other files out of the ShaderFixes folder temporarily before you start dumping.

Q: I tried to do so manually but it seems I can never successfully hunt down the shadow in particular it's like it doesn't exists.

A: Yeah, dumping manually for certain types of effects is next to impossible. Back in DX9, shadows used to not be so bad, but in DX11 I'm finding more and more that they are not obvious to dump (hence why I now rely on frame analysis instead). One thing that might help is to try changing the option 'marking_mode' to 'pink' which causes the selected shader to appear as pink, or sometimes disappear (and in some cases will make things that didn't seem to be affected by 'skip' to react).

Xanz Mods

Search This Blog