Introducing a second HL library?

[Roland Melkert]

Also you probably want to list all unique positions, texcords, normals, colors in single big arrays followed by triangle and quad listings using those arrays. But instead like OpenGL wants (uniqe combo's of those elements), permit separate indexes per element.

And to get any real advantage from the format the data probably needs to flattened (e.g. whole recursive 3001.dat).

[Steffen]

this is a perfect example for how much more compact our current syntax is over JSON.

[Roland Melkert]

this is a perfect example for how much more compact our current syntax is over JSON.

I agree JSON is pretty useless for just replacing the current (offiicial) .dat's.

But when you start flattening things JSON will be the better format as it allows for grouping / indexing the vertices so normals and texture info starts to make more sense.

[Ben Supnik]

this is a perfect example for how much more compact our current syntax is over JSON.

I think that if the -only- thing on the table here is JSON vs. .dat/.ldr syntax, then it's sort of "who-cares"...I can write a program to convert from one to the other and then back losslessly - it is a re-encoding of the same information.

The real question of a second library is whether it is -semantically- different...for example, a "second" library could be:
- Flattened - every triangle in a part is in the part file and not in primitives.
- Have no conditional lines.
- Have quads reduced to pairs of triangles.
- Have the shared vertices pre-indexed.
- Have normals pre-computed for apparently-smooth surfaces.
- Have UV coordinates pre-computed for textured parts (from the spec'd projection).

These are changes such that conversions between the two formats would be "lossy". If a second library isn't semantically different, it doesn't provide substantial benefits to apps.

From what I've read in this thread, Roland's proposal was very much "second library is a derived product of the first" - e.g. some of the potentially expensive and/or annoying tasks of processing the library could be removed. UV mapping and normals could be pre-computed once in a tool and the library flattened, BFC fixed and the result would leave viewers with a much simpler rendering format.

Such a "compiled" library probably needs -no- approval from any kind of oversight committee; if I wanted this for BrickSmith I could code it private to BrickSmith and call it a day. There may be good leverage in several programs sharing a compiled format, but if everyone else says this is stupid, one developer can go do it.

But then others have jumped on "this would be a way to get around backward compatibility." But that's a very different notion; if the -new- format is the master and the old format is the derived product and they're not semantically the same, then this represents fnudamental changes in (1) what the library is and (2) what programs can actually edit the master data. That would need real oversight and approval, and it's a much taller order.

Cheers
Ben

[Gerald Lasser]

The only advantage I see is that a vertice is defined and used for multiple objects then (quad, triangles, lines). So (manually) changing a vertice is pretty easy. If you do this manually now, it can be quite a hassle to find all occurrences of the vertice you want to change.

[Nicola]

I think that if the -only- thing on the table here is JSON vs. .dat/.ldr syntax, then it's sort of "who-cares"...I can write a program to convert from one to the other and then back losslessly - it is a re-encoding of the same information.

The real question of a second library is whether it is -semantically- different...for example, a "second" library could be:
- Flattened - every triangle in a part is in the part file and not in primitives.
- Have no conditional lines.
- Have quads reduced to pairs of triangles.
- Have the shared vertices pre-indexed.
- Have normals pre-computed for apparently-smooth surfaces.
- Have UV coordinates pre-computed for textured parts (from the spec'd projection).

These are changes such that conversions between the two formats would be "lossy". If a second library isn't semantically different, it doesn't provide substantial benefits to apps.

Agreed! The shape of the format is irrelevant, the content is. The concept of a compiled library clearly show what we need.

[Rolf Osterthun]

Hey,

this is a perfect example for how much more compact our current syntax is over JSON.

Yes, you are right. I could have also used xml (even worse) in my example or something like:

Code:

// comment
v 30 24 20 c 16 n 1 0 0 uv 1 0 // and so on...
v 26 24 16 c 16
v -26 24 16 c 16
v -30 24 20 c 16
v -26 24 -16 c 16
v -30 24 -20 c 16
v 26 24 -16 c 16
v 30 24 -20 c 16
v 30 0 20 c 24
v -30 0 20 c 24
v -30 0 -20 c 24
v 30 0 -20 c 24
v 30 24 20 c 24
v -30 24 20 c 24
v -30 24 -20 c 24
v 30 24 -20 c 24
v -30 0 -20 c 16
v 30 0 -20 c 16
v 30 0 20 c 16
v -30 0 20 c 16
r 10 4 0 1 0 0 0 -5 0 0 0 1 stud4.dat
r -10 4 0 1 0 0 0 -5 0 0 0 1 stud4.dat
r 0 24 0 26 0 0 0 -20 0 0 0 16 box5.dat invert
q 0 1 2 3
q 3 2 4 5
q 5 4 6 7
q 7 6 1 0
l 8 9
l 9 10
l 10 11
l 11 8
l 12 13
l 13 14
l 14 15
l 15 12
l 12 8
l 13 9
l 15 11
l 14 10
q 16 17 18 19
q 18 0 3 19
q 19 3 5 16
q 17 7 0 18
r 20 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r 0 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r -20 0 10 1 0 0 0 1 0 0 0 1 stud.dat
r 20 0 -10 1 0 0 0 1 0 0 0 1 stud.dat
r 0 0 -10 1 0 0 0 1 0 0 0 1 stud.dat
r -20 0 -10 1 0 0 0 1 0 0 0 1 stud.dat

I choose JSON for my example because it is human readable and "Code for parsing and generating JSON data is readily available in many programming languages." (wikipedia)

But it was not my goal to suggest JSON. I would like to see the (a) library expandable to vertex colors, normals, uv-maps, etc.

I agree JSON is pretty useless for just replacing the current (offiicial) .dat's.

But when you start flattening things JSON will be the better format as it allows for grouping / indexing the vertices so normals and texture info starts to make more sense.

You are right. I should have added some normals and other stuff to the example (e.g.):

Code:

"Vertices": [
    {
      "Position": [30, 24, 20],
      "Normal": [0, 1, 0],
      "Color": 16
    },
    {
      "Position": [26, 24, 16],
      "Normal": [0, 1, 0],
      "Color": 16
    },
}
[...]
"Meshes": [
  {
    "Name": "Inside"
    "HideCondition": {
      "And": [
        "Ref": {
          "Position": [20, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [0, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [-20, 24, 10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [20, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [0, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        },
        "Ref": {
          "Position": [-20, 24, -10],
          "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
          "File": "nstud.dat"
        }      
      ]
    }
    "Ref": {
      "Position": [10, 4, 0],
      "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
      "File": "stud4.dat"
    },
    "Ref": {
      "Position": [-10, 4, 0],
      "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
      "File": "stud4.dat"
    },
    "Ref": {
      "Position": [0, 24, 0],
      "Orientation": [26, 0, 0, 0, -20, 0, 0, 0, 16],
      "File": "box5.dat"
      "BFC": "Invert"
    }
  },
  {
    "Name": "Outside",
    [...]
  }

Again, just some thoughts.

Rolf

[Rolf Osterthun]

Hey Roland,

Also you probably want to list all unique positions, texcords, normals, colors in single big arrays followed by triangle and quad listings using those arrays.

this is something where I'm really not so sure.

In .obj files (for example) you have to follow a fixed sequence (write 1//1 for a position and a normal). If I may exaggerate slightly, this could guide to something like 1//1//////4 ;-).

In .ply files you have to define what follows (at least I think so). Too complex in my opinion.

What I could imagine is someting like (here a quad):

Code:

4 0 1 2 3 n 0 c 16

or

Code:

4 0 1 2 3 n4 0 0 0 0 c4 16 16 4 16 // with Gradient Brush

But currently, I would prefer "uniqe combo's of those elements". This is something I have also seen in many computer games. It would be nice if the library would be so flexible that you could even add bones, bone weights and bone indices (in theory). But it's a wash.

And to get any real advantage from the format the data probably needs to flattened (e.g. whole recursive 3001.dat).

Hm. I really like the hierarchical structure of the LDraw file format. Here are the ghosts: do you want an expandable high perfomance library or an expandable hierarchical library with smaller files.

Rolf