LDraw.org Discussion Forums
Introducing a second HL library? - Printable Version

+- LDraw.org Discussion Forums (https://forums.ldraw.org)
+-- Forum: General (https://forums.ldraw.org/forum-12.html)
+--- Forum: Official File Specifications/Standards (https://forums.ldraw.org/forum-32.html)
+--- Thread: Introducing a second HL library? (/thread-14432.html)

Pages: 1 2

Re: Introducing a second library? - Roland Melkert - 2015-02-19

Also you probably want to list all unique positions, texcords, normals, colors in single big arrays followed by triangle and quad listings using those arrays. But instead like OpenGL wants (uniqe combo's of those elements), permit separate indexes per element.

And to get any real advantage from the format the data probably needs to flattened (e.g. whole recursive 3001.dat).

Re: Introducing a second library? - Steffen - 2015-02-22

this is a perfect example for how much more compact our current syntax is over JSON.

Re: Introducing a second library? - Roland Melkert - 2015-02-22

Steffen Wrote:this is a perfect example for how much more compact our current syntax is over JSON.

I agree JSON is pretty useless for just replacing the current (offiicial) .dat's.

But when you start flattening things JSON will be the better format as it allows for grouping / indexing the vertices so normals and texture info starts to make more sense.

Re: Introducing a second library? - Ben Supnik - 2015-02-23

Steffen Wrote:this is a perfect example for how much more compact our current syntax is over JSON.

I think that if the -only- thing on the table here is JSON vs. .dat/.ldr syntax, then it's sort of "who-cares"...I can write a program to convert from one to the other and then back losslessly - it is a re-encoding of the same information.

The real question of a second library is whether it is -semantically- different...for example, a "second" library could be:
- Flattened - every triangle in a part is in the part file and not in primitives.
- Have no conditional lines.
- Have quads reduced to pairs of triangles.
- Have the shared vertices pre-indexed.
- Have normals pre-computed for apparently-smooth surfaces.
- Have UV coordinates pre-computed for textured parts (from the spec'd projection).

These are changes such that conversions between the two formats would be "lossy". If a second library isn't semantically different, it doesn't provide substantial benefits to apps.

From what I've read in this thread, Roland's proposal was very much "second library is a derived product of the first" - e.g. some of the potentially expensive and/or annoying tasks of processing the library could be removed. UV mapping and normals could be pre-computed once in a tool and the library flattened, BFC fixed and the result would leave viewers with a much simpler rendering format.

Such a "compiled" library probably needs -no- approval from any kind of oversight committee; if I wanted this for BrickSmith I could code it private to BrickSmith and call it a day. There may be good leverage in several programs sharing a compiled format, but if everyone else says this is stupid, one developer can go do it.

But then others have jumped on "this would be a way to get around backward compatibility." But that's a very different notion; if the -new- format is the master and the old format is the derived product and they're not semantically the same, then this represents fnudamental changes in (1) what the library is and (2) what programs can actually edit the master data. That would need real oversight and approval, and it's a much taller order.

Cheers
Ben

Re: Introducing a second library? - Gerald Lasser - 2015-02-23

The only advantage I see is that a vertice is defined and used for multiple objects then (quad, triangles, lines). So (manually) changing a vertice is pretty easy. If you do this manually now, it can be quite a hassle to find all occurrences of the vertice you want to change.

Re: Introducing a second library? - Nicola - 2015-02-23

Ben Supnik Wrote:I think that if the -only- thing on the table here is JSON vs. .dat/.ldr syntax, then it's sort of "who-cares"...I can write a program to convert from one to the other and then back losslessly - it is a re-encoding of the same information.

The real question of a second library is whether it is -semantically- different...for example, a "second" library could be:
- Flattened - every triangle in a part is in the part file and not in primitives.
- Have no conditional lines.
- Have quads reduced to pairs of triangles.
- Have the shared vertices pre-indexed.
- Have normals pre-computed for apparently-smooth surfaces.
- Have UV coordinates pre-computed for textured parts (from the spec'd projection).

These are changes such that conversions between the two formats would be "lossy". If a second library isn't semantically different, it doesn't provide substantial benefits to apps.

Agreed! The shape of the format is irrelevant, the content is. The concept of a compiled library clearly show what we need.

Re: Introducing a second library? - Rolf Osterthun - 2015-02-23

Hey,

Steffen Wrote:this is a perfect example for how much more compact our current syntax is over JSON.
Yes, you are right. I could have also used xml (even worse) in my example or something like:
Code:
// comment
v 30 24 20 c 16 n 1 0 0 uv 1 0 // and so on...
v 26 24 16 c 16
v -26 24 16 c 16
v -30 24 20 c 16
v -26 24 -16 c 16
v -30 24 -20 c 16
v 26 24 -16 c 16
v 30 24 -20 c 16
v 30 0 20 c 24
v -30 0 20 c 24
v -30 0 -20 c 24
v 30 0 -20 c 24
v 30 24 20 c 24
v -30 24 20 c 24
v -30 24 -20 c 24
v 30 24 -20 c 24
v -30 0 -20 c 16
v 30 0 -20 c 16
v 30 0 20 c 16
v -30 0 20 c 16
r 10 4 0 1 0 0 0 -5 0 0 0 1 stud4.dat
r -10 4 0 1 0 0 0 -5 0 0 0 1 stud4.dat
r 0 24 0 26 0 0 0 -20 0 0 0 16 box5.dat invert
q 0 1 2 3
q 3 2 4 5
q 5 4 6 7
q 7 6 1 0
l 8 9
l 9 10
l 10 11
l 11 8
l 12 13
l 13 14
l 14 15
l 15 12
l 12 8
l 13 9
l 15 11
l 14 10
q 16 17 18 19
q 18 0 3 19
q 19 3 5 16
q 17 7 0 18
r 20 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r 0 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r -20 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r 20 0 -10 1 0 0 0 1 0 0 0 1 stud.dat
r 0 0 -10 1 0 0 0 1 0 0 0 1 stud.dat
r -20 0 -10 1 0 0 0 1 0 0 0 1 stud.dat
I choose JSON for my example because it is human readable and "Code for parsing and generating JSON data is readily available in many programming languages." (wikipedia)

But it was not my goal to suggest JSON. I would like to see the (a) library expandable to vertex colors, normals, uv-maps, etc.

Roland Melkert Wrote:I agree JSON is pretty useless for just replacing the current (offiicial) .dat's.

But when you start flattening things JSON will be the better format as it allows for grouping / indexing the vertices so normals and texture info starts to make more sense.
You are right. I should have added some normals and other stuff to the example (e.g.):
Code:
"Vertices": [
    {
      "Position": [30, 24, 20],
      "Normal": [0, 1, 0],
      "Color": 16
    },
    {
      "Position": [26, 24, 16],
      "Normal": [0, 1, 0],
      "Color": 16
    },
}
[...]
"Meshes": [
  {
    "Name": "Inside"
    "HideCondition": {
      "And": [
        "Ref": {
          "Position": [20, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [0, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [-20, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [20, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [0, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [-20, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        }      
      ]
    }
    "Ref": {
      "Position": [10, 4, 0],
      "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
      "File": "stud4.dat"
    },
    "Ref": {
      "Position": [-10, 4, 0],
      "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
      "File": "stud4.dat"
    },
    "Ref": {
      "Position": [0, 24, 0],
      "Orientation": [26, 0, 0, 0, -20, 0, 0, 0, 16],
      "File": "box5.dat"
      "BFC": "Invert"
    }
  },
  {
    "Name": "Outside",
    [...]
  }

Again, just some thoughts.

Rolf

Re: Introducing a second library? - Rolf Osterthun - 2015-02-24

Hey Roland,

Roland Melkert Wrote:Also you probably want to list all unique positions, texcords, normals, colors in single big arrays followed by triangle and quad listings using those arrays.

this is something where I'm really not so sure.

In .obj files (for example) you have to follow a fixed sequence (write 1//1 for a position and a normal). If I may exaggerate slightly, this could guide to something like 1//1//////4 ;-).

In .ply files you have to define what follows (at least I think so). Too complex in my opinion.

What I could imagine is someting like (here a quad):
Code:
4 0 1 2 3 n 0 c 16
or
Code:
4 0 1 2 3 n4 0 0 0 0 c4 16 16 4 16 // with Gradient Brush

But currently, I would prefer "uniqe combo's of those elements". This is something I have also seen in many computer games. It would be nice if the library would be so flexible that you could even add bones, bone weights and bone indices (in theory). But it's a wash.

Roland Melkert Wrote:And to get any real advantage from the format the data probably needs to flattened (e.g. whole recursive 3001.dat).

Hm. I really like the hierarchical structure of the LDraw file format. Here are the ghosts: do you want an expandable high perfomance library or an expandable hierarchical library with smaller files.

Rolf