Introducing a second HL library?

[Roland Melkert]

With all the recent commotion I'm thinking it might be useful to introduce a second 'official' LDraw library.

It would be identical parts wise as the master one, but all parts will be formatted in a single .dat using universal winding etc. We could also add normals to those single files without polluting the master LDraw library.

I would basically be a collection of pre calculated cache files. This library could even be made available in e.g. .obj format.

Anyone think this is worth a second look / go ?

[Willy Tschager]

Going for a second library would be better than trying to fix the existing one. We would have to have:

* A new file extension such as .prt to be distinguishable from the old library
* Normals and all the fancy things you want in
* Contain connection points
* An automated conversion process with one or a bunch of progs in place before the conversion starts on the whole. We surely DON'T want to recycle some 10000 files through the PT. Anyone remembering the workload for the new header?

My 2 cents,

w.

[Nicola]

I kind of like the idea. It permits to depart from the ldraw format without breaking compatibility. You could put all needed info inside (normals, edges, condlines, etc) and the developers could take full advantage of everything.
Were you thinking about "flattening" che hierarchy, or keeping it? maybe it's better if a single .prt file contains all the data for a part without dependency (ie, all subparts are "inlined"), to mantain the concept of "one file, one part".

The only problem i see is that ldr and mpd file are themselves in the ldraw format. So opening and ldr file would mean parse both the "old" (current) and new format, also, nothing stops an ldr/mpd file from having tri/quads/lines directly inside, which would be a problem. Designing a new format for "models" too would mean that programs based on the new library will be unable to open old files out in the wild (effectively creating some kind of fork). Indeed one of the characteristic of the current format is that there's no precise boundaries between model, submodel, part, subpart and primitive.

Also i get the feeling that this would be some kind of "side project" that will fail to bring any real innovation on the ldraw world. But maybe not, maybe some developer will start using the new library and we can make a smooth transition without breaking anything..

[Merlijn Wissink]

I've been thinking about this for the past few days, a second new and updated LDraw library. I can see no disadvantages for the library iteself.

However, everything else based on the first library will become, well, useless. Unless the software is being updated of course.
For example, the golden combination of MLcad + LPub (both aren't being updated anymore) for making instructions will become "useless". So, someone will need to make new software for making instructions and the question is if there will be. If not, people keep using the old library (for making instructions in this case) and then the part authors don't see any reason why they should make parts in the new format and also stick with making new parts for the old library.
And, this is just 1 example.

I really like the idea of a new library, but if we make a complete new library, we also need to make a complete new toolset (new software) for all the things that can be done with LDraw models (e.g. making instructions, which is one of the most important ones).

[Nicola]

If i understoor correctly the idea, the current library will still be the one that get updated and receive manteinance, while the new library will simply be calculated from the first one (i imagine it will be automatically periodically aligned), so the current one would not be useless at all. Indeed all the work will still happen on the current one until new softwares come around taking advantages of the new one. (this if i get roland proposal right).

I agree that LPub is currently one of the primary reason for using LDraw, being the only option available for building hi quality instructions. It surely is for me at least

[Roland Melkert]

I was thinking to ether keep these flattened files in normal LDraw format but using new meta/line types for adding normals etc. Or using a completely new (or existing 3d) format.

End user tools still need to understand basic LDraw (type 1, mpd format) to use the library but instead of having to recursively and process the raw library files they use the pre processed ones from the new library.

Any older software just keeps using the original files directly. Also new software can decide them self's if they (also) support the raw format (needed for unofficial files for example) or take the 'easy' way and only use the extended files.

If in the long run people (part authors) like creating the single file parts better or prefer a reduced recursive nature we could reassess things concerning the master library.

[Roland Melkert]

Yes, I was thinking when ever a new library is released some program basiclly does what a modern viewer does (calculating normals, fixing errors, etc) and generate the flattened files from the 'raw' LDraw files into e.g. complete-ext.zip.

[Willy Tschager]

I really like the idea of a new library, but if we make a complete new library, we also need to make a complete new toolset (new software) for all the things that can be done with LDraw models (e.g. making instructions, which is one of the most important ones).

As for LPub someboby will have to fix it and some of the current bugs right away:

http://sourceforge.net/projects/lpub4/

Then there is also LIC:

http://forums.ldraw.org/showthread.php?t...1#pid13871

w.

[Willy Tschager]

Also i get the feeling that this would be some kind of "side project" that will fail to bring any real innovation on the ldraw world. But maybe not, maybe some developer will start using the new library and we can make a smooth transition without breaking anything..

Not necessarily. Get some geeks, make the tools and see what happens.

w.

[Chris Dee]

I like the idea of the new library being programmatically derived each time the current library is updated, if indeed that is feasible.

So it sounds to me like the existing library could be considered to be conventional "source code" and the new, enhanced version a kind of "compiled" version. Since objects in the new library would never be created manually there's no reason for it to be text format, so it could be binary, and thus possibly smaller (4-byte float numbers support 6 significant decimal digits), although if the complilation inlines all subpart and primitive references that may not be the case. In keeping with the LDraw ethic, I think it should be Open Source or Creative Commons. It also wouldn't need to be 8000+ separate files in one folder, which I'm aware has been mentioned as a scaleability concern.

However, it seems pointless (and ambitious) for us to try and define a new binary graphics file format, if there is already one out there that provides all we need. I don't know enough to evaluate that possibilty myself. I think connectibility might be a missing feature of existing formats, but we haven't really incorporated that into the LDraw file format either, apart from a notional acceptence that studs form a connection point.

[Rolf Osterthun]

Hey,

yes, I also think this is worth a second look. This is not the first time I think about such a thing.

But, I would love to see a new file format: Something like a mixture between standard LDraw and an extension of the .obj file format:

• At first you have to define all needed vertices (optional with normals, uv-map, color, etc). I can imagine that it would also be possible to add new properties later (e.g. bone-weight, render information, etc).
  
• With these vertices you can now define (via indices) lines, triangles, (quads, polygons,) optional lines, etc. Here it might be possible to add new properties later (render information, textures, etc), too.
  

But (without thinking long about it), I would try to keep the reference lines and primitives. I think this architecture is something that is perfect for a 3D LEGO file format. But you have to be consequent: when there is a cone primitive, you are not allowed to use it in a spherical area ;-). If you need something special you can still ‘inline’ the primitive.

The tools should only interpret that lines they are interested in. All other lines have to be ignored (not only the comments). By this it would be much easier to extent the format in the future.

With something like this it would be possible to create gradient brushes

and texturing would also be covered.

Just some of my thoughts…

Rolf

[Nicola]

I like the idea of the new library being programmatically derived each time the current library is updated, if indeed that is feasible.

So it sounds to me like the existing library could be considered to be conventional "source code" and the new, enhanced version a kind of "compiled" version. Since objects in the new library would never be created manually there's no reason for it to be text format, so it could be binary, and thus possibly smaller (4-byte float numbers support 6 significant decimal digits), although if the complilation inlines all subpart and primitive references that may not be the case. In keeping with the LDraw ethic, I think it should be Open Source or Creative Commons. It also wouldn't need to be 8000+ separate files in one folder, which I'm aware has been mentioned as a scaleability concern.

However, it seems pointless (and ambitious) for us to try and define a new binary graphics file format, if there is already one out there that provides all we need. I don't know enough to evaluate that possibilty myself. I think connectibility might be a missing feature of existing formats, but we haven't really incorporated that into the LDraw file format either, apart from a notional acceptence that studs form a connection point.

If we go for an existing format, we should choose one that permits attaching custom data, so that we can put ldraw-related informations in it, such as connection points (as you said) or condlines, etc.
I think we don't have that complicated of a format, we don't have textures, UV, material related stuff, etc. So a custom format could still be a good (and simple enought) solution.

Actually i think the brevity of ldraw format is not as great as it is pictured. Sure, the recursive nature saves space, but the vertices are stored in a very inefficient way: they are repeated in each face instead of being listed beforehand and referenced in faces. This makes "complex" parts very verbose and possibly waste all the gain from primitives.

[Orion Pobursky]

Maybe an XML wrapper file? We could have a tag the references the raw DAT code and other tags for connections, textures, etc...

[Philippe Hurbain]

we don't have textures

We do, through texmap extensions.

Actually i think the brevity of ldraw format is not as great as it is pictured. Sure, the recursive nature saves space, but the vertices are stored in a very inefficient way: they are repeated in each face instead of being listed beforehand and referenced in faces. This makes "complex" parts very verbose and possibly waste all the gain from primitives.

Agreed (though conversion into other 3D formats show that LDraw format is overall very efficient). And it has the big benefit of beeing "local" (a single line carries all information needed to be meaningful).

[Roland Melkert]

I too was thinking .obj, I just don't know if it allows for extension without confusing other software capable of importing 'clean' obj. Because it would be very nice to use standard .obj for the main mesh etc but still have the option to (at least) add conditional lines without breaking the format for other tools to use it as is.

Generating the basic .obj would actually be pretty basic as it's almost the same info/data you need to generate in order to draw the part efficiently in OpenGL.

[Willy Tschager]

So what?! I love when someone pops in from time to time making a lot of fuss how and what should be change - better WE should change - and then vanishes into thin air right afterwards:

http://forums.ldraw.org/showthread.php?t...6#pid13116

w.

[Nicola]

So what?! I love when someone pops in from time to time making a lot of fuss how and what should be change - better WE should change - and then vanishes into thin air right afterwards:

http://forums.ldraw.org/showthread.php?t...6#pid13116

w.

well, if you're talking about me, i'm not vanished, i'm here and i expressed my points of view in these posts and replies. Some people replied but if discussion stops there's no point in talking alone Maybe there's no real interest in changing things, which i'm ok with.

also, i didn't even "popped in". Much to my surprise, my project brigl got picked up by most major lego sites, lately on brickset where i expressed my opinions on ldraw. My comments on brickset were discussed in here. So i just felt like i had to reply.

I already stopped working on brigl years ago, when i found that i was unable to craft the delicate algorithms required to make ldraw work on modern tools (such as Three.js) and lost interest.

So you can think i'm a poor programmer who couldn't get his algorithms together and pretends that everybody else changes theirs, and that's a legit point of view, but i still think that all the extra work needed to deal with the ldraw format is completey superfluous and only stems from some bad/outdated foundations of the architecture. When you find yourself wasting days to fix a problem that you know shouldn't even be there, you lose motivation.

As for helping, as i said i'm not a C/C++ programmer, and onestly i don't have the time to learn a new language. I can work with java and of course javascript, if there's anything that can be done with them i'd love to.

Tomorrow i'll write some more details about my ideas

[Roland Melkert]

I think the first step needed before any other is to get rid of all detectable and auto correctable defects in the existing official library.

I'm willing to try and do that using a dev version of my LDCad, which will then generate corrected files for any file currently generating warnings in it's log as they are already corrected in memory. So I just need to make some (small) adjustments in order to write those files to disk.

But afterwards someone else needs to take those (probably many) files and merge them with the pt etc, anyone interested?

[Steffen]

Of course our library should be error-free.
It simply isn't yet, because humans make errors and because we lacked time to find all.
However, over time these errors will be gone.

I like it that your post has triggered us to cure these problems.

When 2 years ago I wrote a quick LDRAW parser myself, I also ran it on all official and unofficial files for testing,
and found many BFC syntax errors in official files.
I took the effort to cure all of them and uploaded them to the PT.

However, what I do not like is that you're attacking the file format itself so fundamentally.
Yes, the files have errors like bowtie quads etc., but it still is up to your program to be robust against such.
You don't even have to correct them there.
You can simply skip broken things.
This responsibility cannot be taken from your shoulders. Otherwise your program would require that all its input
is error-free, which does not happen in real life. If you feed a broken JPG image to your painting program,
it should report an error or ignore certain parts of the file, but it certainly shouldn't crash.
Your program needs to be robust against broken input.

Regarding using BFC CCW everywhere, of course, this is trivial to do for all our parts and subparts,
and I like doing that - simply for simplicity.
However, we still need the full BFC syntax like 0 BFC INVERTNEXT so we do not have to duplicate
all our primitives. So changing all parts and subparts to 0 BFC CCW is a nice-to-have, but I would regard
that a low-prio task. We should more focus on fixing the remaining [few!] errors in official files
like bow-tie quads and unknown colors.

The discussion about normals is more complex and already in a different thread.
I'd like to add to that discussion maybe in another post.

I like the simplicity of the LDRAW file format a LOT over other formats, especially binary ones.
The file format has made us very productive, the library has grown HUGE over the past years,
and although there are quirks and problems at some places, many, many, many parts have been created.
There's 2 sides of our library: authors are working a lot with the files and producing parts,
and consuming applications parse them. Bringing both worlds together sometimes needs balancing.
For example, yes, it might be difficult for brigl to fix a bowtie quad - but, hey, then simply ignore it.
It anyway is broken in the input file. On the other hand, curing that quad is easy for a part author,
and it is not the fault of the file format that the bowtie quad is present. Would we choose another file
format, then there would be other errors in it, and your application again would need to be robust against them.

[Nicola]

I think the first step needed before any other is to get rid of all detectable and auto correctable defects in the existing official library.

I agree 100%, this is so cheap there's no reason not to procede right away.

I'm willing to try and do that using a dev version of my LDCad, which will then generate corrected files for any file currently generating warnings in it's log as they are already corrected in memory. So I just need to make some (small) adjustments in order to write those files to disk.

But afterwards someone else needs to take those (probably many) files and merge them with the pt etc, anyone interested?

I hope this is not a problem

[Nicola]

Of course our library should be error-free.
It simply isn't yet, because humans make errors and because we lacked time to find all.
However, over time these errors will be gone.

I like it that your post has triggered us to cure these problems.

When 2 years ago I wrote a quick LDRAW parser myself, I also ran it on all official and unofficial files for testing,
and found many BFC syntax errors in official files.
I took the effort to cure all of them and uploaded them to the PT.

However, what I do not like is that you're attacking the file format itself so fundamentally.
Yes, the files have errors like bowtie quads etc., but it still is up to your program to be robust against such.
You don't even have to correct them there.
You can simply skip broken things.
This responsibility cannot be taken from your shoulders. Otherwise your program would require that all its input
is error-free, which does not happen in real life. If you feed a broken JPG image to your painting program,
it should report an error or ignore certain parts of the file, but it certainly shouldn't crash.
Your program needs to be robust against broken input.

Regarding using BFC CCW everywhere, of course, this is trivial to do for all our parts and subparts,
and I like doing that - simply for simplicity.
However, we still need the full BFC syntax like 0 BFC INVERTNEXT so we do not have to duplicate
all our primitives. So changing all parts and subparts to 0 BFC CCW is a nice-to-have, but I would regard
that a low-prio task. We should more focus on fixing the remaining [few!] errors in official files
like bow-tie quads and unknown colors.

The discussion about normals is more complex and already in a different thread.
I'd like to add to that discussion maybe in another post.

I like the simplicity of the LDRAW file format a LOT over other formats, especially binary ones.
The file format has made us very productive, the library has grown HUGE over the past years,
and although there are quirks and problems at some places, many, many, many parts have been created.
There's 2 sides of our library: authors are working a lot with the files and producing parts,
and consuming applications parse them. Bringing both worlds together sometimes needs balancing.
For example, yes, it might be difficult for brigl to fix a bowtie quad - but, hey, then simply ignore it.
It anyway is broken in the input file. On the other hand, curing that quad is easy for a part author,
and it is not the fault of the file format that the bowtie quad is present. Would we choose another file
format, then there would be other errors in it, and your application again would need to be robust against them.

Well first off let me say that i appreciate all the effor that have been put in ldraw, you guys did an incredible job in putting all of this together. I hope this is clear, i'm grateful and i use ldraw programs regulary to do my stuff (mostly instructions). I don't want to appear ungrateful.

About input problems, i think a program should be robust enought not to crash and report any error, but should never try to "autofix" things. HTML parsers of the previous decade showed this very clearly: the first browsers tried to autocorrect erratic html, and what they caused was more erratic html being created, no consistency in how different browsers rendered things, and an hell for anybody trying to implement a renderer.
It looks like ldraw had the same problem, with lenient softwares that caused erratic parts to go unnoticed and even slip uncorrected into the official library.

Btw, as you said, the existance and the fixing of broken parts have nothing to do with the file format itself, they're separated issues, we could have broken part in any other format

About the format itself, i already expressed my opinions. I think it has a lot of unnecessary fuss in it. It's hardly "simple", a simple 3d format usually only have vertices, triangles and normals. Nothing else. It doesn't need to invent new concepts. I think the library has grown huge thanks to your endeavour, not thanks to the format. I admit i don't know much about how part designers work, but i got that they use a suite of custom tools and often correct things manually directly in the text. It would be very, very hard to convince me that this process is more effective than working with any modern cad software. Maybe they're accustomed to it, and became proficient, but that doesn't mean it's ideal.

Being text or binary based makes no difference, it's the content that is important and the same content can be stored indifferently in binary or text.

[Mario Pascucci]

Hi all.

My apologies in advance for this long message.

As you know, I'm working on a LDraw model editor. After your really useful replies to my concerns about yet another editor, I re-think my program to do a "multiple-personality" tool: model, part and connection editor. At first view it isn't a great problem, program is modular and uses a lot of high level libraries I coded to parse and efficiently render LDraw parts. So I started re-working program in this new "flavor".

So, in the purpose to contribute to LDraw community, here follows a list of things/features on I'm working (please take as a mere proposal):

Connections
I try to extract connection points from part itself, using primitives as marker to detect a connection point. I.e. if I read a part with "stud.dat" primitive, I add to part connection list a "stud-type" connection, oriented and placed using primitive matrix and placement. This is done for various stud and bottom tube type, technic axle, axlehole, pin, peghole, and program is able not only to detect it automatically, but it uses connection points to detect and "snap" new placing part to nearest "opposite" connection point.
Connection points are oriented, so you cannot connect a part in wrong way.
A drawback is that lots of parts does not use primitives that can be uniquely associated with a connection point. An example is a 1x1 brick (or plate or tile): it uses stud.dat for top stud (and program correctly detect "STUD" connection), but "box5.dat" primitive cannot be certainly detected as a connection site, because it is used in so many parts as a "main body".
So I started thinking a strategy for describe connections for parts where cannot be detected "safely". At the moment my strategy is:
- I look in a "connection database": if part isn't in DB, I call a procedure that "auto-detect" connections.
- if part is in DB, I read all connection from it.
Benefits are:
- if parts in library use official primitives, it is highly probable that new parts does not need a special connection description, because primitives "marks" connection points;
- does not needs any modification in part library
Drawbacks are:
- we need to manually edit connection points for parts where can't be auto-detected
- we need a new format for connection description.

At the moment, I chose XML for connection description (it is reasonably easy to read and write in many programming languages and has a structure that allows a quick check for "correctness", you can use something like "lint", an utility that verifies syntax). Strategy is: connections database is a folder where is one file for part, first time I load a part from library I search for file in this folder, if a file exists, I read it and put in a cache, else I call the "auto-detect connection" procedure and cache results.

An example of XML file for connection description is attached.
It is a minifig hand, that has three connection type: a "STUD" connection, for placing bricks or plate on top; a "CLIP" connection, to "grab" bar, utensil, handles; finally a "BAR" connection in "wrist", to attach hand on an arm.
File extension can be (to say...) CXML (for "connection markup language") or CML to comply with 3 char extension limit, I renamed in "txt" to allow attachment, but in fact it is a txt file.

Program
It is in Java, and, despite the "urban legends" that follows this language, it is "damn fast"™
It is (and will be) open source and free.

At the moment program is in "early alpha":
- it can read a model (DAT, LDR, MPD, L3B and so on) using a library I coded for read and parse LDraw files;
- it detect connection points from primitives
- it allows placing of new parts using a "snap" very similar to LEGO Digital Designer, with some minor differences
- does not checks for part collision (I have no idea how to do without a lot of heavy and complex geometric computation)
- it saves a flat LDR of edited model (no submodel, no MPD, but it is planned)

I'm working to merge my connection editor in model editor (it merely needs to add toolbars with specific functions and add some logic to program to handle/display/edit connections), and to add tools for editing part (place lines, triangle, quads, primitives, ...).

It is a lot of work, but I am confident that I can manage it, without hurry.

If you wish, I can post some screenshot to show program aspect and progress.

Meanwhile, I can help to detect and correct some library errors, you have only to ask

HTH.

Mario

[Willy Tschager]

Some people replied but if discussion stops there's no point in talking alone

It's good practice here that - if you really seek something changed or new - to take the lead and spearhead the project. You cannot delegate the thing to someone else 'cos nobody gets paid here and in case the discussion stops it's up to you carrying it on.

w.

[Max Martin Richter]

Have you had a look into the system of SR3D Builder? There is a folder with connectivity information for each part. I really like, how it is done there. Unfortunately the system isn't in developed anymore because Sergio passed away. :-(

/Max

[Mario Pascucci]

Hi Max.

No, I never used SR3D.
For connection data in SR3D I think there is a license problem: SR3D isn't open source, and require a specific license for commercial use, so I think I cannot use SR3D data without breaking license and intellectual property.
No information is available on SR3D web about data and source license, but in home page there is a copyright notice that say "all right reserved", so I cannot assume anything about use of SR3D data.

Mario

[Roland Melkert]

LDCad has connection information too, it uses LDraw meta's in a shadow library.

The nice thing (imho ) about the format is it has very little hard coded connection limitations, all is purely based on shapes / text parameter matching.

Maybe we could share the format, or base a more global one upon it.

[Orion Pobursky]

I think if we're going to go through all the trouble to create a parallel library, why not make something like a wrapper file format that's just as easy to read but easier to extend. Someone a long time ago suggested an XML version of the library. I envision something like this:

Code:

<ldrawfile>
<header>
(Header tags)
</header>
<connections>
(Connection tags)
</connections>
<datcode>
(Link to raw dat file)
</datcode>
<binaryfile>
(Link to a binary file)
</binaryfile>
</ldrawfile>

The advantage of this approach is that XML parsers are plentiful and we can easily extend the format without breaking existing programs (since the old dat file with still exist albeit without the new connection, normal, etc info).

[Mario Pascucci]

IMHO we can do a further step ahead...

It is quite simple to do a converter DAT->XML and XML->DAT if we have a well-defined XML document definition, an XSD file that define tags, structure, types and limits/bounds for parameters (for non-developers).
Tags can be optional, have attributes, etc.
Example:

Code:

<dat type="official" ldrawid="3001.dat">
<description>
Brick 2x4
</description>
<author name="john smith" email="[email protected]" />
<keywords>
<kw name="brick" />
<kw name="original parts" />
</keywords>
...
<connections>
<conn type="stud" base="x,y,z" vector="xv,yv,zv" offset="xo,yo,zo" />
...
</connections>
<primitives>
<triangle v1="x,y,z" v2="x1,y1,z1" v3="x2,y2,z2" />
<ref color="4" part="stud.dat" place="x,y,z" matrix="a,b,c,d,e,f,g,h,i" />
<line type="edge" v1="x,y,z" v2="x1,y1,z1" />
<line v1="x,y,z" v2="x1,y1,z1" />
<quad argbcolor="a0ff00c0" normal="xn,yn,zn" v1="x,y,z" v2="x1,y1,z1" v3="x2,y2,z2"  v4="x3,y3,z3" />
<auxline .....
</primitives>

We can maintain a pure DAT library for backward compatibility and use new XML format for new/improved library. Parser will be really easy to do, and a converter XML->DAT will automatically add/update parts for old format.

As usual, this is brainstorming ;-)

Mario

[Merlijn Wissink]

Sounds nice and doable to me. Since you keep the original .dat files as the source, it won't stop anyone from making new parts in the "old" library.

[Orion Pobursky]

I'm OK with it as long as there's never a time when the xml file exists without a corresponding dat file.

[Chris Dee]

+1

[Nicola]

what about json instead of xml? same complexity to use, much smaller and much easier to read for human

[Nicholas Zurn]

It probably wouldn't hurt to have another version of all of the parts in the library in a format that is easily used in other programs and can be converted to other files successfully.


The mechanical side of me wishes there were CSG files for all of the parts.

[Steffen]

XML to me always appeared too overheading. Example: Pure XML for a quad would be:

Code:

<quad>
<color>16</color>
<vertex><x>1</x><y>2</y><z>3</z></vertex>
<vertex><x>4</x><y>5</y><z>6</z></vertex>
<vertex><x>7</x><y>8</y><z>9</z></vertex>
<vertex><x>10</x><y>11</y><z>12</z></vertex>
</quad>

If you want to make this more compact, you could go for something like this:

Code:

<quad>
<color>16</color>
<vertex>1 2 3</vertex>
<vertex>4 5 6</vertex>
<vertex>7 8 9</vertex>
<vertex>10 11 12</vertex>
</quad>

But to be honest: this is no longer pure XML. It is custom strings embedded into XML.
You still need to parse the strings. If you accept that, then you also must accept this,
which takes the principle 1 step further:

Code:

<quad>
<color>16</color>
<coordinates>
1 2 3
4 5 6
7 8 9
10 11 12
</coordinates>
</quad>

And even 1 step further:

Code:

<quad>16 1 2 3 4 5 6 7 8 9 10 11 12</quad>

Then you finally land at the current LDRAW format, which simply encodes <quad>...</quad> by the leading "4":

Code:

4 16 1 2 3 4 5 6 7 8 9 10 11 12

I write this to argument against XML. I have to deal with it a lot elsewhere, and despite the fact that parsers
for it exist a lot etc. pp., its syntax overhead always is ineffective and annoying.
JSON would be something much better IMHO. Parsers for it exist as well a lot.

[Steffen]

About input problems, i think a program should be robust enought not to crash and report any error, but should never try to "autofix" things. HTML parsers of the previous decade showed this very clearly: the first browsers tried to autocorrect erratic html, and what they caused was more erratic html being created

I fully agree with you in this aspect. Programs need to be robust against rubbish, but should ignore it.
Otherwise people create more and more rubbish and are not aware of it. HTML is the exact example how things went mad.
Thanks a lot for pointing this out.

Regarding the LDRAW file format simplicity, I would like to answer, YES, I still regard LDRAW file format "simple":
It exactly IS what you claimed, i.e, just 3D vertices and color. It just has added tiny, tiny additions to it:
(a) BFC
(b) conditional lines
© inclusion of other files
Later,
(d) textures
were added. But (a)+© are trivial. (b) is a bit tricky and un-conventional but can programmatically done.
However, yes, I agree that some things like normals, connectivity info etc. is missing, but I would
still name the current format "simple". I would prefer to extend it to support normals and connectivity.
Not to replace it. It should be simple and easy to add those 2 syntax elements downwards-compatibly.

[Nicola]

XML to me always appeared too overheading.

..


I write this to argument against XML. I have to deal with it a lot elsewhere, and despite the fact that parsers
for it exist a lot etc. pp., its syntax overhead always is ineffective and annoying.
JSON would be something much better IMHO. Parsers for it exist as well a lot.

I too have to deal with it daily at work Here's an example of how a 3d geometry can be expressed in json (just an example, we can put whatever format of course):

Code:

{
    "metadata": {
        "version": 3.1,
        "type": "geometry",
        "generator": "GeometryExporter"
    },
    "vertices": [50,50,50,...],
    "normals": [1,0,0,...],
    "uvs": [[0,1,...]],
    "faces": [56,0,2,1,0,1,2,0,0,0,0,...]
}

(taken from here)

Boh machine and human readable. JSON can take advantage of array definitions which XML has not.

[Santeri Piippo]

+1 to JSON. I was going to suggest it myself. Though the syntax is awfully unforgiving...

[Steffen]

yes, I sometimes found it much easier to model parts directly in POVRay using CSG than in LDRAW, sigh......., that's true......

[Steffen]

Oh, I hear this terrible news here the first time - what a loss!
This makes me so sad!

Just found a post about it at Eurobricks...
http://www.eurobricks.com/forum/index.ph...pic=100887

[Mario Pascucci]

XML can be more compact if you use "attributes". To follow your example for a quad:

Code:

<quad color="4" normal="xn,yn,zn" v1="x1,y1,z1" v2="x2,y2,z2" v3="x3,y3,z3" v4="x4,y4,z4" />

You can omit attributes, or use combination, if document definition permits it:

Code:

<!-- a quad without normal and custom color with alpha -->
<quad argbcolor="c080ff80" v1="x1,y1,z1" v2="x2,y2,z2" v3="x3,y3,z3" v4="x4,y4,z4" />

It is valid XML and it isn't too "verbose".
Consider that you can automate transformation using stylesheet (XSL) and syntax checking using tools like XMLlint (verify an XML file against its XSD)

BTW, any format can be accettable, if permits easy syntax checking and well-supported parsing in many programming languages.

Mario

[Michael Horvath]

Are there CSG file formats? I wouldn't consider POV-Ray a "file format" since it can contain so many other things besides CSG.

[Nicholas Zurn]

OpenSCAD, OpenJSCAD, some special languages, and some CAD/solid modeling software.

[Magnus Forsberg]

We should more focus on fixing the remaining [few!] errors in official files

What do you mean by a 'few' files?
This page presents over 700 official files to fix.

I had a look into some of the broken parts that was pointed out in this thread, and I doubt that a quick fix, only flipping bad quads and making them bfc is going to give us good parts that render well. I found gaps, missing lines and surfaces, missplaced lines and surfaces, missing cond-lines, non-primitives. Older designs, made in the early days, that don't live up to todays standards.
(These are now corrected and uploaded to PT (Minifig Robot Leg, Flippers, Minifig Lifevest and more.)

They all seem to need far more work than a program could fix. They also need the hand of a human part author with a good eye. Would a 'simple' tool correct those errors? Can LDCad do that?

I see this as yet another argument for never releasing a new part that 'needs work'. We will forever be caugth in a never ending spiral of correcting designs that should have been 'correct' at the first release.

[Philippe Hurbain]

But we can (and imho that's the way to go) correct the blatant errors that can be easily auto-corrected, and then, if someone feels it, manually fix structural errors... After all, as imperfect as they are, these parts have done the job for years!

[Magnus Forsberg]

True,
let's use the tools we have today, and hope for better tools in the making.
But how do you add normals to a missing surface, or connectivity information to a missing primitive?

[Philippe Hurbain]

But how do you add normals to a missing surface,

As Nicola (I think it's him) explained, it's easy to derive normals from a smooth shading algrorithm. Only fine tweaking for special needs would have to rely on human intervention...

or connectivity information to a missing primitive?

Anyway you can't define connectifity only from primitives (eg. tile in minifig hand). It's a big helper, but not sufficient.

[Roland Melkert]

Would a 'simple' tool correct those errors? Can LDCad do that?

LDCad currently corrects (from memory have to check for exact listing): non flat quads (2 triangles), hourglass quads, ref captions, duplicate points (quad to triangle, con line to nor line) and all zero matrix col/rows.

These corrections are made (at the ldraw line level) in memory after parsing they are just never written (although code for writing 2..5 lines is present). So with a couple of small adjustments I could make a special version which would rewrite those corrected files when e.g. save all is used.

After which a tool like winmerge could visualize the changes in regards to the original 2014-01 lib in order to do a quick check.

Or I could just limit the saved corrections to hourglass and non flat quads if you think the other ones need intensive human intervention/checkups.

[Roland Melkert]

I read about this sad news a month ago, very sudden indeed still not much details available though.

steerco/webadmin: Doesn't this news deserve a frontpage / Announcements forum notice now it more confirmed?

[Rolf Osterthun]

Hey Roland,

at first: sorry for the delay.

I too was thinking .obj, I just don't know if it allows for extension without confusing other software capable of importing 'clean' obj. Because it would be very nice to use standard .obj for the main mesh etc but still have the option to (at least) add conditional lines without breaking the format for other tools to use it as is.

As far as I found out, there is no possibility to extend the normal .obj-format. Indeed, that would have been a great solution.

Generating the basic .obj would actually be pretty basic as it's almost the same info/data you need to generate in order to draw the part efficiently in OpenGL.

Agreed.

I made a little experiment and created a JSON compatibe file that could be a basis for a further discussion. Here you can see the conversion of the file "s\3002s01.dat":

Code:

{
  "Header": {
    "Description": "~Brick  2 x  3 without Front Face",
    "Name": "s\3002s01.dat",
    "Author": "James Jessiman",
    "!LDRAW_ORG": "Subpart UPDATE 2003-03",
    "!LICENSE": "Redistributable under CCAL version 2.0 : see CAreadme.txt"
  },    
  "Metadata": {
    "BFC": "CCW"
  },
  "History": {
    "2002-05-07": "[unknown] BFC Certification"
    "2003-07-03": "[Steffen] Subfiled for patterns"
    "2003-12-19": "[PTadmin] Official Update 2003-03"
    "2007-08-29": "[PTadmin] Header formatted for Contributor Agreement"
    "2008-07-01": "[PTadmin] Official Update 2008-01"
  },
  "Vertices": [
    {
      "Position": [30, 24, 20],
      "Color": 16
    },
    {
      "Position": [26, 24, 16],
      "Color": 16
    },
    {
      "Position": [-26, 24, 16],
      "Color": 16
    },
    {
      "Position": [-30, 24, 20],
      "Color": 16
    },
    {
      "Position": [-26, 24, -16],
      "Color": 16
    },
    {
      "Position": [-30, 24, -20],
      "Color": 16
    },
    {
      "Position": [26, 24, -16],
      "Color": 16
    },
    {
      "Position": [30, 24, -20],
      "Color": 16
    },
    {
      "Position": [30, 0, 20],
      "Color": 24
    },
    {
      "Position": [-30, 0, 20],
      "Color": 24
    },
    {
      "Position": [-30, 0, -20],
      "Color": 24
    },
    {
      "Position": [30, 0, -20],
      "Color": 24
    },
    {
      "Position": [30, 24, 20],
      "Color": 24
    },
    {
      "Position": [-30, 24, 20],
      "Color": 24
    },
    {
      "Position": [-30, 24, -20],
      "Color": 24
    },
    {
      "Position": [30, 24, -20],
      "Color": 24
    },
    {
      "Position": [-30, 0, -20],
      "Color": 16
    },
    {
      "Position": [30, 0, -20],
      "Color": 16
    },
    {
      "Position": [30, 0, 20],
      "Color": 16
    },
    {
      "Position": [-30, 0, 20],
      "Color": 16
    }
  ],
  "Meshes": [
    {
      "Ref": {
        "Position": [10, 4, 0],
        "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
        "File": "stud4.dat"
      },
      "Ref": {
        "Position": [-10, 4, 0],
        "Orientation": [1, 0, 0, 0, -5, 0, 0, 0, 1],
        "File": "stud4.dat"
      },
      "Ref": {
        "Position": [0, 24, 0],
        "Orientation": [26, 0, 0, 0, -20, 0, 0, 0, 16],
        "File": "box5.dat"
        "BFC": "Invert"
      },
      "Quad": {
        "Vertices": [0, 1, 2, 3]
      },
      "Quad": {
        "Vertices": [3, 2, 4, 5]
      },
      "Quad": {
        "Vertices": [5, 4, 6, 7]
      },
      "Quad": {
        "Vertices": [7, 6, 1, 0]
      },
      "Line": {
        "Vertices": [8, 9]
      },
      "Line": {
        "Vertices": [9, 10]
      },
      "Line": {
        "Vertices": [10, 11]
      },
      "Line": {
        "Vertices": [11, 8]
      },
      "Line": {
        "Vertices": [12, 13]
      },
      "Line": {
        "Vertices": [13, 14]
      },
      "Line": {
        "Vertices": [14, 15]
      },
      "Line": {
        "Vertices": [15, 12]
      },
      "Line": {
        "Vertices": [12, 8]
      },
      "Line": {
        "Vertices": [13, 9]
      },
      "Line": {
        "Vertices": [15, 11]
      },
      "Line": {
        "Vertices": [14, 10]
      },
      "Quad": {
        "Vertices": [16, 17, 18, 19]
      },
      "Quad": {
        "Vertices": [18, 0, 3, 19]
      },
      "Quad": {
        "Vertices": [19, 3, 5, 16]
      },
      "Quad": {
        "Vertices": [17, 7, 0, 18]
      },
      "Ref": {
        "Position": [20, 0, 10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      },
      "Ref": {
        "Position": [0, 0, 10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      },
      "Ref": {
        "Position": [-20, 0, 10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      },
      "Ref": {
        "Position": [20, 0, -10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      },
      "Ref": {
        "Position": [0, 0, -10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      },
      "Ref": {
        "Position": [-20, 0, -10],
        "Orientation": [1, 0, 0, 0, 1, 0, 0, 0, 1],
        "File": "stud.dat"
      }      
    }
  ]
}

I do not know the JSON format very well, so please be lenient when there are errors. This should only inspire the discussion.

With something like this it would be easy to add normals or uv-maps. It would be possible to create gradient brushes and for the rubber parts we could also add bones and bone weights ;-). It would be possible to add render-information (e.g. for the slopes) and we could add hide-conditions or something like that.

But: this would be a new format...

Rolf

[Magnus Forsberg]

I don't think that file is a good example, or maybe it is...

If it was made from todays primitives it would be simplified to something like this:
.

Code:

1 16 0 4 0 1 0 0 0 -5 0 0 0 1 stug4-1x2.dat
0 BFC INVERTNEXT
1 16 0 24 0 26 0 0 0 -20 0 0 0 16 box5.dat
4 16 30 24 20 26 24 16 -26 24 16 -30 24 20
4 16 -30 24 20 -26 24 16 -26 24 -16 -30 24 -20
4 16 -30 24 -20 -26 24 -16 26 24 -16 30 24 -20
4 16 30 24 -20 26 24 -16 26 24 16 30 24 20
1 16 0 24 0 30 0 0 0 -24 0 0 0 20 box4t.dat
1 16 0 0 10 1 0 0 0 1 0 0 0 1 stug-1x3.dat
1 16 0 0 -10 1 0 0 0 1 0 0 0 1 stug-1x3.dat