LDraw.org Discussion Forums
Introducing a second HL library? - Printable Version

+- LDraw.org Discussion Forums (https://forums.ldraw.org)
+-- Forum: General (https://forums.ldraw.org/forum-12.html)
+--- Forum: Official File Specifications/Standards (https://forums.ldraw.org/forum-32.html)
+--- Thread: Introducing a second HL library? (/thread-14432.html)

Pages: 1 2 3 4 5 6


Re: Introducing a second HL library? - Nicola - 2014-11-13

Steffen Wrote:Of course our library should be error-free.
It simply isn't yet, because humans make errors and because we lacked time to find all.
However, over time these errors will be gone.

I like it that your post has triggered us to cure these problems.

When 2 years ago I wrote a quick LDRAW parser myself, I also ran it on all official and unofficial files for testing,
and found many BFC syntax errors in official files.
I took the effort to cure all of them and uploaded them to the PT.

However, what I do not like is that you're attacking the file format itself so fundamentally.
Yes, the files have errors like bowtie quads etc., but it still is up to your program to be robust against such.
You don't even have to correct them there.
You can simply skip broken things.
This responsibility cannot be taken from your shoulders. Otherwise your program would require that all its input
is error-free, which does not happen in real life. If you feed a broken JPG image to your painting program,
it should report an error or ignore certain parts of the file, but it certainly shouldn't crash.
Your program needs to be robust against broken input.

Regarding using BFC CCW everywhere, of course, this is trivial to do for all our parts and subparts,
and I like doing that - simply for simplicity.
However, we still need the full BFC syntax like 0 BFC INVERTNEXT so we do not have to duplicate
all our primitives. So changing all parts and subparts to 0 BFC CCW is a nice-to-have, but I would regard
that a low-prio task. We should more focus on fixing the remaining [few!] errors in official files
like bow-tie quads and unknown colors.

The discussion about normals is more complex and already in a different thread.
I'd like to add to that discussion maybe in another post.

I like the simplicity of the LDRAW file format a LOT over other formats, especially binary ones.
The file format has made us very productive, the library has grown HUGE over the past years,
and although there are quirks and problems at some places, many, many, many parts have been created.
There's 2 sides of our library: authors are working a lot with the files and producing parts,
and consuming applications parse them. Bringing both worlds together sometimes needs balancing.
For example, yes, it might be difficult for brigl to fix a bowtie quad - but, hey, then simply ignore it.
It anyway is broken in the input file. On the other hand, curing that quad is easy for a part author,
and it is not the fault of the file format that the bowtie quad is present. Would we choose another file
format, then there would be other errors in it, and your application again would need to be robust against them.

Well first off let me say that i appreciate all the effor that have been put in ldraw, you guys did an incredible job in putting all of this together. I hope this is clear, i'm grateful and i use ldraw programs regulary to do my stuff (mostly instructions). I don't want to appear ungrateful.

About input problems, i think a program should be robust enought not to crash and report any error, but should never try to "autofix" things. HTML parsers of the previous decade showed this very clearly: the first browsers tried to autocorrect erratic html, and what they caused was more erratic html being created, no consistency in how different browsers rendered things, and an hell for anybody trying to implement a renderer.
It looks like ldraw had the same problem, with lenient softwares that caused erratic parts to go unnoticed and even slip uncorrected into the official library.

Btw, as you said, the existance and the fixing of broken parts have nothing to do with the file format itself, they're separated issues, we could have broken part in any other format Smile

About the format itself, i already expressed my opinions. I think it has a lot of unnecessary fuss in it. It's hardly "simple", a simple 3d format usually only have vertices, triangles and normals. Nothing else. It doesn't need to invent new concepts. I think the library has grown huge thanks to your endeavour, not thanks to the format. I admit i don't know much about how part designers work, but i got that they use a suite of custom tools and often correct things manually directly in the text. It would be very, very hard to convince me that this process is more effective than working with any modern cad software. Maybe they're accustomed to it, and became proficient, but that doesn't mean it's ideal.

Being text or binary based makes no difference, it's the content that is important and the same content can be stored indifferently in binary or text.


Re: Introducing a second HL library? - Mario Pascucci - 2014-11-13

Hi all.

My apologies in advance for this long message.

As you know, I'm working on a LDraw model editor. After your really useful replies to my concerns about yet another editor, I re-think my program to do a "multiple-personality" tool: model, part and connection editor. At first view it isn't a great problem, program is modular and uses a lot of high level libraries I coded to parse and efficiently render LDraw parts. So I started re-working program in this new "flavor".

So, in the purpose to contribute to LDraw community, here follows a list of things/features on I'm working (please take as a mere proposal):

Connections
I try to extract connection points from part itself, using primitives as marker to detect a connection point. I.e. if I read a part with "stud.dat" primitive, I add to part connection list a "stud-type" connection, oriented and placed using primitive matrix and placement. This is done for various stud and bottom tube type, technic axle, axlehole, pin, peghole, and program is able not only to detect it automatically, but it uses connection points to detect and "snap" new placing part to nearest "opposite" connection point.
Connection points are oriented, so you cannot connect a part in wrong way.
A drawback is that lots of parts does not use primitives that can be uniquely associated with a connection point. An example is a 1x1 brick (or plate or tile): it uses stud.dat for top stud (and program correctly detect "STUD" connection), but "box5.dat" primitive cannot be certainly detected as a connection site, because it is used in so many parts as a "main body".
So I started thinking a strategy for describe connections for parts where cannot be detected "safely". At the moment my strategy is:
- I look in a "connection database": if part isn't in DB, I call a procedure that "auto-detect" connections.
- if part is in DB, I read all connection from it.
Benefits are:
- if parts in library use official primitives, it is highly probable that new parts does not need a special connection description, because primitives "marks" connection points;
- does not needs any modification in part library
Drawbacks are:
- we need to manually edit connection points for parts where can't be auto-detected
- we need a new format for connection description.

At the moment, I chose XML for connection description (it is reasonably easy to read and write in many programming languages and has a structure that allows a quick check for "correctness", you can use something like "lint", an utility that verifies syntax). Strategy is: connections database is a folder where is one file for part, first time I load a part from library I search for file in this folder, if a file exists, I read it and put in a cache, else I call the "auto-detect connection" procedure and cache results.

An example of XML file for connection description is attached.
It is a minifig hand, that has three connection type: a "STUD" connection, for placing bricks or plate on top; a "CLIP" connection, to "grab" bar, utensil, handles; finally a "BAR" connection in "wrist", to attach hand on an arm.
File extension can be (to say...) CXML (for "connection markup language") or CML to comply with 3 char extension limit, I renamed in "txt" to allow attachment, but in fact it is a txt file.

Program
It is in Java, and, despite the "urban legends" that follows this language, it is "damn fast"™ Big Grin
It is (and will be) open source and free.

At the moment program is in "early alpha":
- it can read a model (DAT, LDR, MPD, L3B and so on) using a library I coded for read and parse LDraw files;
- it detect connection points from primitives
- it allows placing of new parts using a "snap" very similar to LEGO Digital Designer, with some minor differences
- does not checks for part collision (I have no idea how to do without a lot of heavy and complex geometric computation)
- it saves a flat LDR of edited model (no submodel, no MPD, but it is planned)

I'm working to merge my connection editor in model editor (it merely needs to add toolbars with specific functions and add some logic to program to handle/display/edit connections), and to add tools for editing part (place lines, triangle, quads, primitives, ...).

It is a lot of work, but I am confident that I can manage it, without hurry.

If you wish, I can post some screenshot to show program aspect and progress.

Meanwhile, I can help to detect and correct some library errors, you have only to ask Smile

HTH.

Mario


Re: Introducing a second HL library? - Willy Tschager - 2014-11-13

Nicola Wrote:Some people replied but if discussion stops there's no point in talking alone Smile

It's good practice here that - if you really seek something changed or new - to take the lead and spearhead the project. You cannot delegate the thing to someone else 'cos nobody gets paid here and in case the discussion stops it's up to you carrying it on.

w.


Re: Introducing a second HL library? - Max Martin Richter - 2014-11-13

Have you had a look into the system of SR3D Builder? There is a folder with connectivity information for each part. I really like, how it is done there. Unfortunately the system isn't in developed anymore because Sergio passed away. :-(

/Max


Re: Introducing a second HL library? - Mario Pascucci - 2014-11-13

Hi Max.

No, I never used SR3D.
For connection data in SR3D I think there is a license problem: SR3D isn't open source, and require a specific license for commercial use, so I think I cannot use SR3D data without breaking license and intellectual property.
No information is available on SR3D web about data and source license, but in home page there is a copyright notice that say "all right reserved", so I cannot assume anything about use of SR3D data.

Mario


Re: Introducing a second HL library? - Roland Melkert - 2014-11-13

LDCad has connection information too, it uses LDraw meta's in a shadow library.

The nice thing (imho Smile ) about the format is it has very little hard coded connection limitations, all is purely based on shapes / text parameter matching.

Maybe we could share the format, or base a more global one upon it.


Re: Introducing a second HL library? - Orion Pobursky - 2014-11-13

I think if we're going to go through all the trouble to create a parallel library, why not make something like a wrapper file format that's just as easy to read but easier to extend. Someone a long time ago suggested an XML version of the library. I envision something like this:
Code:
<ldrawfile>
<header>
(Header tags)
</header>
<connections>
(Connection tags)
</connections>
<datcode>
(Link to raw dat file)
</datcode>
<binaryfile>
(Link to a binary file)
</binaryfile>
</ldrawfile>

The advantage of this approach is that XML parsers are plentiful and we can easily extend the format without breaking existing programs (since the old dat file with still exist albeit without the new connection, normal, etc info).


Re: Introducing a second HL library? - Mario Pascucci - 2014-11-14

IMHO we can do a further step ahead...

It is quite simple to do a converter DAT->XML and XML->DAT if we have a well-defined XML document definition, an XSD file that define tags, structure, types and limits/bounds for parameters (for non-developers).
Tags can be optional, have attributes, etc.
Example:
Code:
<dat type="official" ldrawid="3001.dat">
<description>
Brick 2x4
</description>
<author name="john smith" email="john@smith.somewhere.local" />
<keywords>
<kw name="brick" />
<kw name="original parts" />
</keywords>
...
<connections>
<conn type="stud" base="x,y,z" vector="xv,yv,zv" offset="xo,yo,zo" />
...
</connections>
<primitives>
<triangle v1="x,y,z" v2="x1,y1,z1" v3="x2,y2,z2" />
<ref color="4" part="stud.dat" place="x,y,z" matrix="a,b,c,d,e,f,g,h,i" />
<line type="edge" v1="x,y,z" v2="x1,y1,z1" />
<line v1="x,y,z" v2="x1,y1,z1" />
<quad argbcolor="a0ff00c0" normal="xn,yn,zn" v1="x,y,z" v2="x1,y1,z1" v3="x2,y2,z2"  v4="x3,y3,z3" />
<auxline .....
</primitives>

We can maintain a pure DAT library for backward compatibility and use new XML format for new/improved library. Parser will be really easy to do, and a converter XML->DAT will automatically add/update parts for old format.

As usual, this is brainstorming ;-)

Mario


Re: Introducing a second HL library? - Merlijn Wissink - 2014-11-14

Sounds nice and doable to me. Since you keep the original .dat files as the source, it won't stop anyone from making new parts in the "old" library.


Re: Introducing a second HL library? - Orion Pobursky - 2014-11-14

I'm OK with it as long as there's never a time when the xml file exists without a corresponding dat file.