Modernizing The Spec [Poll]

[Orion Pobursky]

To continue our earlier discussion about modernizing the spec, there seems to be 2 camps. This poll is designed to see where developer sentiments lie.

[Steffen]

I cannot answer this poll.
It narrows down my brain to 2 channels, and it refuses )
For me, the answer depends on WHAT is going to be added.
And, if it is added to the current file format, will it be added DOWNWARDS compatibly or NOT.
So I cannot vote here
May I ask to re-word and re-start the vote like this:
(a) add new commands directly to existing files downwards compatibly, i.e., old software will not break
(b) add a wrapper style file like XML or JSON to individual parts
© add 1 big wrapper style file like XML or JSON which contains additive information for not 1 but MULTIPLE parts

[Orion Pobursky]

Since nobody has voted I just changed the poll to be more specific and added your third suggestion

[Steffen]

thank you, Orion.
however, there seem to have been 2 votes already, 1 for (a), and 1 for (b).

I've just voted for (a).

My reasoning was:
The syntax may be special and different from nowadays formats JSON, XML, etc, but it is so simple and compact
that it can be immediately understood.
It is human-readable, compact, can be easily diffed using text comparison tools.
It compresses well.
It can be easily extended using our 0 !xyz syntax.
Writing a parser for it is quite simple.

I do not like the idea of putting second-level wrapper file(s) around the existing *.dat's.

In the long run, however, we might need a more fundamental change:
I think it would be better to store parts in JSON format instead of the proprietary *.dat syntax.
They could reference each other as usual.
It would be much easier to add new syntax to them.

However, that solution is only feasible if we have a converter tool from *.dat to the new JSON standard
AND (!) back.

That'll be the only way to still be able to use the existing tooling.

One after the other, the tooling could learn to use the new JSON syntax.

HOWEVER, at the moment
- no such converter tooling exists
- no JSON syntax proposal exists
, so in THAT case my vote is (a)
until we have these available.

Creating a wrapper layer to me just appears as an intermediate solution or - to use the hard word - workaround.

My suggestion is to create a JSON syntax suggestion and converter tools,
then setup a new vote.

[Orion Pobursky]

My intention with this question was to take the "long view" and not necessarily consider what we have right now but where we want to head towards in the future. Converters are easy to write.

I like the idea of keeping the DAT syntax as it is right now and adding all the non-geometry stuff (i.e. normals, textures, etc) into a wrapper file. This will not break backwards compatibility as the existing software doesn't know about that stuff anyway. As long as a DAT file exists for every part and we don't define any part geometry stuff in the wrapper file then we don't need to worry about old software.

[Ben Supnik]

Hi Y'all,

It seems like this poll is an attempt to give ourselves permission to add to the .ldr file format more aggressively - in the past, some combination of cultural inertia and the need for backward compatibility with end-of-lifed apps has made .ldr extensions "off limits".

So I'd like to argue for -direct- extension of the LDraw format. I think in the long run it will produce the clearest, most understandable .ldr files; if everything is wrapped, interaction between extensions and scope is going to be more complicated.

There are two kinds of possible extensions:
1. Extensions where backward compatibility is easy, because the extension can be ignored (e.g. the proposed META to re-specify the origin.
2. Extensions where backward compatibility is hard because the extension -replaces- existing functionality (e.g. you can have one META sphere command instead of 300 triangle primitives).

The current texture extension tries to be in category 1 by providing backup geometry - you can ignore all of the new stuff and just draw the backup geometry. But to be really useful the TEXMAP extension needs to be in category 2, e.g. there certainly shouldn't be a backup triangle-based pattern to the part, because the whole point of texturing extensions is to save work for authors by letting them use a PNG file for patterns.

With that in mind, we can (relatively) easily provide backward compatibility against category 1 extensions (that are DIRECTLY in the .ldr and .dat files) by making a processor that strips them out, and making a second downloadable edition of the library that is stripped of new stuff.

I like the idea of a post-processed distribution more than wrappers because a second distro is a one-time tax on development - we have to develop and deploy the tools, where-as having wrapper files is a tax on development of every single program that ever parsers a new .ldr file for as long as we go forward.

For category 2 extensions, there's no way to win - if the newly authored part is truly not backward compatible with older clients for some reason, we have to either drop the clients, author 2 versions of the part, or not use the new extension. But that's something that needs to be decided on an extension-by-extension basis. For example, texturing was authored to intentionally make the backward compatibility case "ugly but not fatal" - you can easily have a solid triangle replacing your textured triangle - the part looks unpatterned but doesn't look like a bug. As we add new extensions, we'll have to consider this.

With that in mind, at least some of the things that have been tossed around that are somewhat 'major':
- Better normal vector support
- LOD
- Connectivity
- Meta data to "fix" authoring issues

Are all pretty clearly in category 1.

Two more comments on this:
- I do have a plan^H^H^H^Hpipe dream of doing some kind of connectivity in Bricksmith in the next year or so...and I strongly believe connectivity is best served with an in-file-format extension. Because LDraw parts are constructed from primitives, we can (in some cases) annotate a small number of primitives with connectivity meta-data and get a pretty big win in terms of part coverage. Doing that with a wrapper is going to make such an extension significantly less intuitive than it would otherwise be. (I think you can demonstrate that a wrapper system is functionally equivalent to extending the format - but it still puts a tax on all programs dealing with the wrappers.)

- When it comes to LOD, there was some support for LOD via a parallel file system, instead of in-file-format extensions. It's already how LOd is partly being dealt with, so there's precedent, and it's very unobtrusive. I don't think this parallel-library idea is wrong, and it wouldn't be crazy to do LOD that way even if most other extensions are going to be in-file.

Cheers
Ben

[Orion Pobursky]

It seems like this poll is an attempt to give ourselves permission to add to the .ldr file format more aggressively - in the past, some combination of cultural inertia and the need for backward compatibility with end-of-lifed apps has made .ldr extensions "off limits".

The "end-of-lifed apps" you mention also happen to be the most popular for creating models with the library. I'd like to at least try to not break compatibility with them unless we have a really good reason. See as that it's not really that hard to keep backward compatibility as long as we don't add different geometry commands (and even those could be approximated with primitives) this really isn't an issue

[Ben Supnik]

The "end-of-lifed apps" you mention also happen to be the most popular for creating models with the library. I'd like to at least try to not break compatibility with them unless we have a really good reason.

Right - I figured it was the strong popularity of older apps that aren't being actively developed that was the strong motivation to try to keep parts completely backward compatible.

My view is that -if- we have a breaking directive in the format, a 'filtered' library is a nice way to have both old-program compatibility and new options...clearly we wouldn't go to the extra work or add that level of complexity if a backward compatible directive is possible.

Cheers
Ben

[Jude Parrill]

I'm abstaining from voting because a) I'm not application/program developer of LDraw tools, and have no plan to do so in the immediate future, and b) I'm barely a part author/reviewer, although that mainly has to do with my work lately which has been crazy.

However, this is a topic I feel is very important for this community. Moving forward, I feel like LDraw needs to modernize and keep up with technology, or risk becoming out-dated, out-moded, and otherwise fading into obscurity.

I also feel this isn't matter that shouldn't be jumped into lightly or quickly, but one that carefully considered by all. The LDraw community needs to reach out to developers, authors, and reviewers, past and present, and look for ideas and opinions from them. Then, as much as is possible, form a plan that suits as many as possible while still pushing the standard forward into the future.

I very often feel that the LDraw community is too bogged down and mired in the past, clinging desperately to the dream of perfect backwards compatibility. And while this is a very noble cause, I feel it has caused this community to stagnate and decay. Instead of welcoming new ideas and new technologies, they are often viewed fearfully with the phrase "but that could break backwards compatibility" being uttered, while others nod along solumnly, "we must never break the backwards compatibility."

Now, I apologise if this sounds harsh or mean, as that is not my intention. Rather, it is my wish to galvanize this community into action. Don't fear the Spector of the Past, but have a Vision of the Future (bonus nerd points if you get this reference). Think forward, not backwards.

Please don't misunderstand my intent here, either. I'm not suggesting you throw out everything and start fresh. I'm not suggesting you wantonly throw out backwards compatibility altogether. I'm suggesting you find a way to work with everyone... developers, authors, reviewers, etc... and find a way to move into the future together. Because yes, it's likely going to result in a lot of work... FOR EVERYONE! And I know that's not easy or fun. But if done correctly, if done as a team... as a community... as a group of people working to make something bigger and better than the sum of it's parts, well, then I think it could be something pretty special.

[Merlijn Wissink]

I completely agree to you.

Everyone keeps thinking about backwards compatibilty, but sometimes you just have to go forward. All major programs and filetypes have different versions that are incompatible with each other. For example Microsoft Word: there's .doc and .docx. They also had to decide at some point: "Screw this, we update everyhing." And, although there's no direct compatibilty between the two files, they can be converted into one another.

And, that's what I'm sure about: if we end up with something that's not directly backwards compatible, I'm sure there's someone who will write a converter.

But, I also understand why people fear to break backwards combatibily. It's a circle that difficult to get out: the most popular programs use the old standard, if you break the standard you can't use the old software anymore. But, new software won't be created as quick as it could be, because the standard is quite old.

It's a difficult decision.

[Max Martin Richter]

As I wrote in at least one review in the past, sometimes I wish to had a LDraw 2.0, where all the origin problems, moved-to references, obsolete files, wrong part numbers, etc. could get compensated.
Of course I'm ok with the style of the 1.0 format, but we could add even more information in meta commands and/or correct some other issues (like technic peghole position) there.
So I would prefer something like dat2, basically made as dat, but with more metas and corrected files...
If needed a xml-file for a raw conversion of older models should work as well...

/Max

[Mario Pascucci]

Hi all.
Using LDraw library from 2012, mainly for developing new utilities, I think backward compatibility is a must.

From other point of view, IMHO, library needs some little improvements:
- more "strict" identification for part files (official and unofficial) against model file. It is somewhat difficult to identify if I'm reading a model (that contains only lines type 1) or a part, that can contains all geometric primitives, graphic primitives and references.
- use of "category" for all parts in library, to ease search for parts by category or by use.

HTH

Mario

[Steffen]

more "strict" identification for part files (official and unofficial) against model file. It is somewhat difficult to identify if I'm reading a model (that contains only lines type 1) or a part, that can contains all geometric primitives, graphic primitives and references.

That is already fine today: Parts use file extension *.dat, models (should) use *.ldr or *.mpd.

use of "category" for all parts in library, to ease search for parts by category or by use.

We are already striving for that goal.
Exception is that if the first word of the title already IS its category, it is not necessary to add a redundant category statement.
For example, you don't need to put a part named "Wheel X Y Z" into category "!CATEGORY Wheel".
Software supporting the !CATEGORY meta statement needs to be aware of that.
I find this mechanism simple and convincing.

[Chris Dee]

Although we could recommend an extension to the 0 !LDRAW_ORG statement to identify model files (e.g. '0 !LDRAW_ORG Model'), enforcing that in (text) files not managed through a 'system' would be impossible. All files in the official LDraw library do have a valid !LDRAW_ORG statement.

All parts do have a 'category'. If no !CATEGORY statement is present, the first word of the description must be in the category list. This is enforced by the Parts Update process.

[Mario Pascucci]

That is already fine today: Parts use file extension *.dat, models (should) use *.ldr or *.mpd.

I've found lot of files (not officials) that doesn't follow this convention, but probably were old files.
There are another problem with files with *.ldr extension that are in MPD format, so I do not rely on file extension to read model files.

Thank you for !CATEGORY explaining, I missed that, sorry.

Mario

[Roland Melkert]

more "strict" identification for part files (official and unofficial) against model file. It is somewhat difficult to identify if I'm reading a model (that contains only lines type 1) or a part, that can contains all geometric primitives, graphic primitives and references.

This has been discussed before:

http://forums.ldraw.org/showthread.php?t...543#pid543

It's been some time but I do remember the problem with using the LDRAW_ORG meta for it seemed wrong as it suggests the file is maintained by ldraw.org.

[Nicholas Zurn]

Interesting that everything still involves the legacy spec.

Why isn't there an option for new spec that easily converts to old spec, minus the things that didn't exist in the old spec?

[Steffen]

I do not rely on file extension to read model files

You should. Everything which does not comply with that standard is an error.
We cannot change our spec based on such errors.
People who create such errors will also violate any new spec.

Simply ignore the wrong/broken files.

It makes no sense to change our spec for them.

[Steffen]

a 'filtered' library is a nice way to have both old-program compatibility and new options

No, I do not agree, because these tools are being USED to EDIT and CREATE the library ITSELF.

Would you create that filter you mentioned, then the reverse direction is not possible,
and easy editing of part files is no longer possible.

[Roland Melkert]

dat/ldr/mpd usage are guide lines, not mandatory (except for the library part files).

Even if you put your type 1 lines in a .txt it should still be usable by standard compliant parsers.

A realword example of this is the l3b extension SR3DBuilder uses.

[Steffen]

but even with that, the spec holds:

"Everything that is not *.dat is not a part."

[Travis Cobbs]

Yes, but the reverse is not true, and it would be a really bad idea to assume that all .dat files are parts. Even in the library, lots of the files are non-parts, because they are primitives or sub-parts. So while it might be acceptable to decide that a file is not a part because it doesn't have a .dat extension, you can't learn anything at all about a file with a .dat extension.

[Chris Dee]

So while it might be acceptable to decide that a file is not a part because it doesn't have a .dat extension, you can't learn anything at all about a file with a .dat extension.

But the role of every .dat file in the official library can be identified from its !LDRAW_ORG statement.

[Michael Heidemann]

To be honest, I do not clearly understand what you are talking about. Do you have an example for me?

[Travis Cobbs]

Certainly. As far as I can tell, Steffen's argument is to pay attention to the file extension when trying to decide if something is a part. I'm just trying to say that doing so doesn't help much. LDView ignores the file extension when determining if something is a part. It's my opinion that this is correct behavior.

As it happens the, "Is it a part?" question can give both yes and no answers for the exact same file, and LDView's seams option requires it to get the right answer in both cases. Mirrored parts tend to have a wrapper part with a mirror matrix pointing to the other one. If the non-mirrored part is referenced from the model, it is a part. If the mirrored part is also referenced from the model, it is a part, but its child (the same file that was a part before) is not a part.

This means that the logic to determine if something is a part has to both check to see if the current file itself is a part, and also verify that none of the current file's parent files is a part.

If I were writing LDView today, I would probably rely purely on the !LDRAW_ORG statement, in addition to the tree traversal logic mentioned above (no decendents of parts are parts). However, since LDView was initially created before that statement existed, I also have it so any file loaded from a parts directory is also a part, as long as it doesn't have any part parents, grandparents, etc. I also support a number of other meta commands that over the years have been used by people to denote that a file is a part.

[Nicola]

I had troubles with deciding if a file is a part or not too, i never figured out an algorithm for doing it. Your example of mirrored parts clearly shows the problems.
One thing i think we can agree is that the file extension is a poor way to decide, as no tool does any distinction and they all work interchangeably.
Btw, i don't think there's a rule that forbids ldr file from having tiangle/quads commands directly, inbetween subparts commands.

[Nicola]

To be honest, I do not clearly understand what you are talking about. Do you have an example for me?

He probably meant that there should be another option that reads something like "create a new format from scratch, and make tools to convert from the new to the old format".

[Nicholas Zurn]

Yeah, that's pretty much what I meant.




Also, seems like the PLY format has some similarities.

[Michael Heidemann]

This means that the logic to determine if something is a part has to both check to see if the current file itself is a part, and also verify that none of the current file's parent files is a part.

Very interesting attempt to solve the "i am a part" question.

But I do not fully understand.
example:
104.dat
part type: part
parent: 71185.dat
part type: part

So is the 104.dat not a part because one parent 71185.dat is also a part?

I like the idea to have a definition of what is a part independent from the file header, because only the official files needs to carry one.

If you have a general rule in deciding what is a part build in ldview, please try to give a description of how you did it. I like to integrate this into my applications as well.

[Travis Cobbs]

104.dat is a part if loaded directly. However, the 104.dat referenced inside 71185.dat is not a part. This can only be determined by examination of the model tree at run-time. In LDView, that means that one instance of 104.dat might be treated as a part, while another instance in the same main model might be treated as not a part.

Frankly, if someone has a file in their model that they feel should be a part, and either they or the author of the file couldn't be bothered to include a single line in the file with an unofficial part meta-statement, then I don't have much sympathy for them.

Having said this, LDView has been around for so long that it has quite a number of meta-statements that it recognizes as signifying a part, and furthermore, any file loaded from a directory it considers to be a parts directory will also be considered as a part (unless one of its parents in the tree is a part, as stated above).

[Chris Dee]

I like the idea to have a definition of what is a part independent from the file header, because only the official files needs to carry one.

It would be possible to enforce a valid header for unofficial parts at submission time. I was initially reluctant to implement the full header check on submission for risk of losing valuable contributions. After seven years, is that less of an issue?

[Michael Heidemann]

Sorry for the confusion.

In my understanding all files that are not official files are unoffial files. The difference is that official files has to follow our rules and unofficial do not.
Please do not change the PT. It is good the way it is today!

I am searching for a guideline how to determine if a file is a part just from the file content without analysing any comment lines.

[Chris Dee]

Please do not change the PT. It is good the way it is today!

It may be good for you, but reviewers still need to check and I usually fix badly-formed headers (with admin edits). I'd like other peoples views on the current state of the trade-off between this extra reviw overhead and the risk of newbie authors getting frustrated and dropping out because they can't meet the header standards.

[Philippe Hurbain]

We more or less rely on Datheader for the job... Would it possible to more or less include DH functionnality in submission script?

[Chris Dee]

Yes. There is a Perl script with more or less an equivalent set of checks that gets to check headers during the Parts Update preparation process, so that could be called during submission. Sadly, because the PT runs on Linux, ir's not possible to re-use DATheader as is.