Need some help with code

[Max Murtazin]

So, I've been writing some basic LDRaw "unpacker" for some recent time, and have encountered a problem.

I'm really not sure why, but my code for transforming line type 1 makes model fall apart. It handles fine transforming all other line types, but when it comes to 1s everything breaks unless they have identity matrices for their transformations.

Checked everything, but can't find the issue. Can anybody help me with the thing?

Link to the code: https://pastebin.com/tcy91ih2

[Jonathan N]

The code will be more efficient and easier to understand if you represent points as a Vector3 and use something like Vector3.Transform instead of making everything a matrix. Linear algebra libraries have different conventions for whether matrices are row-major or column-major, which also influences the order of multiplication. I would recommend decomposing your accumulated matrices to make sure the translation, rotation, and scale matches what you would expect. It's easy to transpose or flip the order somewhere by mistake.

[Max Murtazin]

The code will be more efficient and easier to understand if you represent points as a Vector3 and use something like Vector3.Transform instead of making everything a matrix. Linear algebra libraries have different conventions for whether matrices are row-major or column-major, which also influences the order of multiplication. I would recommend decomposing your accumulated matrices to make sure the translation, rotation, and scale matches what you would expect. It's easy to transpose or flip the order somewhere by mistake.

I do use both Vector3 and Matrix4x4 already

[Jonathan N]

I do use both Vector3 and Matrix4x4 already

You also create matrices with what looks like a single vector filled in (tmp) and then extract the vector from the multiplied result.

It's also not clear from the code how you accumulate the matrices. I would just keep a function parameter for the current accumulated transform. Each line type 1 would be a recursive call that passes the current transform multiplied by that line's transform to the function. When you see a vertex, multiply it by the current transform parameter.

It's possible to use a method that modifies the matrix field or property like what you have, but it's easy to forget to "reset" the transform when leaving a subfile. It helps to draw a tree on paper or work with a simple file with only a few pieces. A parent file's transforms affect its children, but the child transforms should not affect other files at the same depth in the tree. The function recursion will handle the case where a file has multiple subfile references.

[Max Murtazin]

You also create matrices with what looks like a single vector filled in (tmp) and then extract the vector from the multiplied result.

It's also not clear from the code how you accumulate the matrices. I would just keep a function parameter for the current accumulated transform. Each line type 1 would be a recursive call that passes the current transform multiplied by that line's transform to the function. When you see a vertex, multiply it by the current transform parameter.

It's possible to use a method that modifies the matrix field or property like what you have, but it's easy to forget to "reset" the transform when leaving a subfile. It helps to draw a tree on paper or work with a simple file with only a few pieces. A parent file's transforms affect its children, but the child transforms should not affect other files at the same depth in the tree. The function recursion will handle the case where a file has multiple subfile references.

I don't have any single-vector matrices. Each matrix I have is info of some type 1 line. 
All the tmp stuff, just checked, is some old unused code I accidentally left in (thought I removed it) — the function actually in use is ParseLine1, not ReplaceLine1

For accumulating transformation — each time I encounter a type 1 line, I tranform it using matrix from current function call, and then pass that transformed line as an argument to call of the function right after, thus transformation is, or at least meant to recursively accumulate — and then each time it gives to the parent call transformed data

What is really a problem, as I mentioned in first post, is that for whatever reason line type 1s are not being properly transformed properly, while 2 through 5 work fine. 
Maybe there really is something wrong with just using matrix multiplication, but that simply makes no sense to me — Vector3's transformation function should be doing exactly the same — matrix multiplication

[Jonathan N]

I don't have any single-vector matrices. Each matrix I have is info of some type 1 line. 
All the tmp stuff, just checked, is some old unused code I accidentally left in (thought I removed it) — the function actually in use is ParseLine1, not ReplaceLine1

For accumulating transformation — each time I encounter a type 1 line, I tranform it using matrix from current function call, and then pass that transformed line as an argument to call of the function right after, thus transformation is, or at least meant to recursively accumulate — and then each time it gives to the parent call transformed data

What is really a problem, as I mentioned in first post, is that for whatever reason line type 1s are not being properly transformed properly, while 2 through 5 work fine. 
Maybe there really is something wrong with just using matrix multiplication, but that simply makes no sense to me — Vector3's transformation function should be doing exactly the same — matrix multiplication

I would step through it with a debugger or use print statements. Matrix decomposition is your friend for these types of problems. Apparently there's a Matrix4x4.Decompose function you can use. The overall process you described sounds right. Note that for matrices A,B it's not true in general that A*B == B*A. I would also check for needing to transpose or not transpose the results. This will be easy to detect if you decompose the matrix and check the translation vector. You can also decompose each line 1 transform to make sure it's initialized properly.