Adding normals to the library

[Nicola]

So as I previously said about normals: no other single file format i've seen leave out normals to be computed by the user. There's absolutely no reasons not to include them. They're integral part of a geometry definition. Why forcing every single developer to reinvent the wheel, to spend countless hour on a custom smoothing algorithm? Algorithms that are already usually complicated, but more so for ldraw where definitions are sparse in multiple files, and one have to take into consideration hard and cond lines. I'm not a newbie of 3d programming (not a genius either by all means), and yet i've spent days trying to implement this algorithm and have yet to succeed. Of course i could spend some other weeks and do it, but why? And even if i did, we'll end up with many different programs, each one with its own algorithm, all slightly different, all buggy in a different way and with no consistency at all.

Updating all the library in not an easy task, but i think it can be done in a way that doesn't break compatibility and permits automatic data entry.

First off, i propose to add a new comment/command, with a standard sintax as follow:

0 NORMALS v1x v1y v1z v2x v2y v2z v3x v3y v3z [v4x v4y v4z]

It will be prepended to all quads and triangles, and will have three or four vertex definitions. Of course each one is the normal at the corresponding vertex of the face, in a per-vertex, per-face paradigm that is usually the standard.
Being a new comment, it will not cause any problem with existing software. Editors working with parts will not have any problem. Editors that write tri and quads may break the file but should not crash or misbehave themselves, and the user should be able to act on the problem. Hard and conditional lines can (and must) stay there of course.
New software developers can take advantage of the normals to achieve an uniform look across different renderers and saving weeks of head banging.

Now the delicate part: this solution assumes that any face have the same normals regardless of where it appears, or in other words that each single subpart/primitive always assume the same normals. But it may be that the same subpart can have different normals depending on the rest of the part it is attached to. If this is indeed the case, than it derives that it's impossible to add normals to a subpart as it can vary.
I'm not sure about this, intuitively each 3d shape should have its normals regardless of what it has around. Much of this maybe depend on the granularity of the subparts, how much down they break a solid. I remember that the primitives breask down even to 2d figures, which will need to be adressed somehow. In the worst case, the primitives would need to be rethinked with normals in mind.

How to popolate the library with normals

Ok, the problem of course is to inject all the normals in the already existing library. As the previous post, my solution is to take an existing smoothing algorithm, elevate it as the standard, and apply it to the whole library.

Basically it would work on a per-part basic. It will load the part and all the subpart into a canonical 3d model data structure, but mantaining a correlation between each face and the file/line that generates it. It will then apply the smoothing algorithm to the part, then traverse all faces and use the correlation to write them back in the source text. Of course some subparts will already have normals calculated (since they're shared), in this case the subpart can be skipped (or take advantage of the occasion to check if the would be calculated the same way).

Ok course normals are useful only if they're available on the whole part library (if a developer need to implement a smoothing algorithm for a single part, he may as well do it for the whole ), this of course means that the library will grow in size, but i hope this is not an issue. Only subparts with actual faces will grow, but most of them should be reusing already defined chunks so will not grow. Eventually some optimization can be implemented (such as inserting a single vector instead of 3/4 if the face has uniform normals).
I'd love here some feedback for this humble proposal

[Philippe Hurbain]

It certainly makes sense, and is backwards compatible for sure. One problem, how to make sure the normals comment remains attached to its triangle/quad? Maybe this could be solved by dumping and regenerating normals after each edit...

But I don't clearly see how you make it work with joined primitives. An example: http://www.ldraw.org/library/official/p/bump5000.dat, made of 3 cone primitives and currently smoothed by condlines.

[Nicola]

One problem, how to make sure the normals comment remains attached to its triangle/quad? Maybe this could be solved by dumping and regenerating normals after each edit...

You mean during the part design? well either the author is aware of normals and work with them, or they could be wiped and recalculated each time for the whole part.

But I don't clearly see how you make it work with joined primitives. An example: http://www.ldraw.org/library/official/p/bump5000.dat, made of 3 cone primitives and currently smoothed by condlines.

[/quote]

Here's an example of the case i speculated above. The problem is not smoothing, as the part as a whole will be smoothed correctly, the problem is that the same cone primitives are used for both cones and spherical surfaces, and will need a different set of normals in different cases. Sadly this breaks the assumption i made. The only solution here is to change the primitives to keep normals in mind, ie clarifiing when a surface is intended to be flat or is an approximation of a smooth surface.

[Philippe Hurbain]

Yeah, we could inline primitives in this case... Could be done automatically when there is a condline next to a surface belonging to a primitive. One step further towards ditching primitives...

[Nicola]

Yes, inlining primitives is one way to go, even if i'm not sure that the check about the condline is enought to cover all cases.

About ditching primitives, i'm all for it. I understand their usefulness for some things, like avoiding replicate definitions of studs, pins and other well defined functional element (hinges, clips etc), but it should end there. I don't know what's the gain of having all possible combinations of cube faces or portion of cylinder etc, when you can put the required quads where you need them. Also, i know nothing about part design, but i think it would be easier to define a part as a whole instead of having to look up for different cutout of boxes and emisphere to adapt (i may be wrong here).

The idea to cover all possible shapes with primitives maybe was good at the beginning, but now lego bricks have all kind of complex shapes (just think of hair pieces or wings, etc), it's impossible. Much better to have a single file with his faces inside.

The only things about (curve) primitives is that they let you replace them with high definition with ease. I don't know if nowadays it is still useful or if we can stick with a single definition

[Merlijn Wissink]

I also always wondered if there was really a use for a lot of primitives. Mind you, I only very recently got interested in part authoring and the workings behind LDraw. The previous years I was a user and nothing more than that.

I do understand the logic behind primitives like studs, common joints etc. etc. Because, first off, they're common and secondly (as far as I know), they are also used for automatic part snapping/connection in software like SR3D Builder or LDCad.

But, primiteves like boxes and things like that are (in my eyes) not that useful. They may save the author maybe a 10 minutes or so of work. And, if you still want the time saving, you can also just inline the primitive.
Especially the primitives based on primitives I find quite strange... For example, there are primitives for a bunch of stud configurations, like 1x4, 1x6, 1x8. They literally just contain the stud primitive 4, 6 or 8 times. It's nothing more than that.

Again, I'm still quite new to all of this. But, that might also be useful. I'm maybe looking at it from a different perspective than the veterans here

[Philippe Hurbain]

Again, I'm still quite new to all of this. But, that might also be useful. I'm maybe looking at it from a different perspective than the veterans here

New blood and ideas are always welcome

[Orion Pobursky]

Let's put this into a non-3D graphics professional/programmer like myself.
I still haven't learned normals to any great extent beyond knowing they exist. How hard is this concept? How much extra math overhead would be needed in order to author a part? Is this something easily automated? How hard would it be to back convert the entire library? My concern is that if parts authoring is even more cumbersome than it is now then part submissions will plummet even more than they already have.

[Roland Melkert]

The things I like about the LDraw format: it's very clean (in it's initial state) and the recursive nature of it makes it very efficient.

So I can't help having mixed feelings adding normals to the library files for a couple of reasons:

1. Not all end user software needs it (e.g. POVRay)
   
2. Normals are trivial to calculate when doing flat shading, so adding one normal per triangle/quad is very wasteful.
   
3. Adding normals for smooth meshes will break the recursive / reusable nature of LDraw like you noted yourself.
   
4. Smoothing isn't that hard when using modern bfc-ed parts (I agree with your notes on fixing the library itself). And if slow or something software is free to pre calculate/cache them.
   

Given the above I feel it's unfair to burden the part authors (often working with little more then notepad etc) with the normal issue, while it's only a couple of days work to implement a decent smoothing algorithm in software (done once).

Maybe we could improve things for the view/edit software programmers by adding a reference smoothing algorithm to the LDraw documentation in pseudo code.

[Nicola]

If you want, i can try to explain briefly what normals are and how they work.

i don't know well how part authoring work, but for a normal 3d application, such as blender, you can define your mesh first, and then you can edit normals in many ways. The easiest is let the program compute them, for example by using a smooth shading command on the whole object. You can specify an arbitrary angle for "hard" edges, and you end up with something like this



This is a one click solution that works for most cases.
Or you can manually select a number of faces that you want to smooth and fine control it. Or by specifying edges to be smoother around.

So, there's no math to do, they're usually calculated by the 3d program and eventually tuned.

For part authoring, i think it can be completely automated, like it is right now after all. Only, instead of calculating it when you display the file in LDView, you calculate it right after you author the part, and put the data in it.

[Nicola]

The things I like about the LDraw format: it's very clean (in it's initial state) and the recursive nature of it makes it very efficient.

So I can't help having mixed feelings adding normals to the library files for a couple of reasons:

1. Not all end user software needs it (e.g. POVRay)
   
2. Normals are trivial to calculate when doing flat shading, so adding one normal per triangle/quad is very wasteful.
   
3. Adding normals for smooth meshes will break the recursive / reusable nature of LDraw like you noted yourself.
   
4. Smoothing isn't that hard when using modern bfc-ed parts (I agree with your notes on fixing the library itself). And if slow or something software is free to pre calculate/cache them.
   

Given the above I feel it's unfair to burden the part authors (often working with little more then notepad etc) with the normal issue, while it's only a couple of days work to implement a decent smoothing algorithm in software (done once).

Maybe we could improve things for the view/edit software programmers by adding a reference smoothing algorithm to the LDraw documentation in pseudo code.

Thanks for your answer, here's my considerations:

1. Povray is a special type of renderer, it uses raytracing, which is a different way to render things than the usual (that's why it takes 10 hours to build a frame). The standard rendering tecnique in the 3d world is using normals.
   
2. Of course the flat surfaces are easy the problem are the curved ones. As i said, optimizations can be made for flat surfaces to make the thing less verbose, and we can discuss on how wasteful it is to fill a couple more MB today.
   
3. You are right, but this is not an universal rule, it's only becouse of how we decided to subdivide things into subparts/primitives.
   
4. First off, it may be impossible to cache them, if the software doesn't know in advance what he has to render. And yes, they're expensive to calculate for a big model. Given that MLCad, LDView etc still have bugs on their renderer, i guess it's not that easy to implement either.
   

Have you ever stopped for a minute to consider WHY our part authors are working with notepad? Do you think there is another branch in the whole 3d world, say videogames, industrial design, movie effect, 3d printing, ANY field where authors are using notepad to design? No, there is not.
Why can't our part authors just fire up Blender, Maya or whatever, design the part with modern tools, autocalculate most of it and export it in a minute? Just like the rest of the world does? Becouse they're stuck with invertnexts, condlines, and a bunch of other "technology" that are unheard of in the rest of the world. And they're unheard becouse they're not necessary.

I don't want to be rude, but they looks like medioeval meticulous miniaturist in a world where everybody else uses electronic printers. And then you guys complain that there are less and less active part authors? Of course there are. But ask yourselves what you are requiring them to do. Time is a precious resource for everybody.

Sorry for the polemic tone, but reading about authors using notepad really made it evident to me how dramatically off the road we are. As i said elsewhere, alternatives exists and are coming out. LDraw already lost a lot of users to LDD. It's easy to imagine how things will go when a serious alternative will be available.

[Roland Melkert]

it may be impossible to cache them

You could cache them per part it's no different from when the source files where per part meshes.

Have you ever stopped for a minute to consider WHY our part authors are working with notepad?

I know full well how different things are with LDraw compared to the rest of the 3D world, but have you considered some people actually prefer working with the text based format.

If the majority of the currently active part authors want normals, and less recursion I'm all for it but I'm not willing to change the format if they don't want/need it.

So in the end, imho, it's up to the part authors to choose (maybe a poll?) and not the software engineers.

[Travis Cobbs]

To the best of my knowledge, the people who have implemented smooth shading recently have done so using color 24 type 2 lines ("edge lines") to indicate creases, and have smoothed geometry that wasn't separated by edge lines. LDView's standard view does something different, which is to use the presense of color 24 type 5 lines ("conditional line") to indicate non-creases.

I came to the conclusion that LDView's way of doing things was the wrong way to go, and when I implemented my POV exporter, I went with the "new" way, and that's what I have advised others to do. I've never gotten around to implementing the other algorithm for LDView's 3D view, and since it is in fact useful for detecting missing conditional lines, I would never get rid of it entirely, simply make it the non-default smoothing option.

The "Smooth Shading and Edge Split" algorithm from Nicola's post is relatively straightforward. The problem is that it works rather poorly with some LDraw geometry. It relies on a threshold angle to decide if things should be smooth shaded or not, and LDraw files have situations where a relatively sharp angle is supposed to be smooth along with other situations where a relatively shallow angle is supposed to have a crease. The edge line-aware algorithm I mentioned above will probably also include a threshold angle, but the threshold will be set to a fairly sharp angle (far sharper than such algorithms normally use) to accomodate LDraw.

Unfortunately, it's not easy to figure out if the place where two arbitrary polygons join has a co-linear edge line. So the edge line-based algorithm is relatively complicated, and also relatively slow.

[Orion Pobursky]

So Ive been seriously wondering if we're all clinging to a sinking ship out of nostalgia and maybe we should convert the library and move on...

[Philippe Hurbain]

I know full well how different things are with LDraw compared to the rest of the 3D world, but have you considered some people actually prefer working with the text based format.

Maybe it's the other way round... I weakly tried working with Blender but gave up in front of the relatively steep learning curve, since the result didn't fit well the (efficient but complex) structure of LDraw files. Making the square Blender peg fit round (primitive ) LDraw hole seemed more work than going on with MLCad/text/special tools.

And we might possibly loose some seasoned LDraw parts authors but attract young blood accustomed to modern modeling tools...

[Nicola]

The "Smooth Shading and Edge Split" algorithm from Nicola's post is relatively straightforward. The problem is that it works rather poorly with some LDraw geometry.

You're right, but i was just talking about normals in general, i was not not suggesting to use this tecnique to mass-add normals to the ldraw library. To do that i think one of the two methods you mentioned are the way to go.

[Scott Wardlaw]

For what little it worth, I completely agree with this! I think this is a necessary step to advance LDraw into more realistic graphics (and help slow all of the other programs / libraries from passing us by).