LDraw.org Discussion Forums
Modernizing The Spec [Poll] - Printable Version

+- LDraw.org Discussion Forums (https://forums.ldraw.org)
+-- Forum: General (https://forums.ldraw.org/forum-12.html)
+--- Forum: Official File Specifications/Standards (https://forums.ldraw.org/forum-32.html)
+--- Thread: Modernizing The Spec [Poll] (/thread-15017.html)

Pages: 1 2 3 4


Modernizing The Spec [Poll] - Orion Pobursky - 2014-12-27

To continue our earlier discussion about modernizing the spec, there seems to be 2 camps. This poll is designed to see where developer sentiments lie.


Re: Modernizing The Spec [Poll] - Steffen - 2014-12-27

I cannot answer this poll.
It narrows down my brain to 2 channels, and it refuses Smile)
For me, the answer depends on WHAT is going to be added.
And, if it is added to the current file format, will it be added DOWNWARDS compatibly or NOT.
So I cannot vote here Sad
May I ask to re-word and re-start the vote like this:
(a) add new commands directly to existing files downwards compatibly, i.e., old software will not break
(b) add a wrapper style file like XML or JSON to individual parts
© add 1 big wrapper style file like XML or JSON which contains additive information for not 1 but MULTIPLE parts


Re: Modernizing The Spec [Poll] - Orion Pobursky - 2014-12-27

Since nobody has voted I just changed the poll to be more specific and added your third suggestion


Re: Modernizing The Spec [Poll] - Steffen - 2014-12-27

thank you, Orion.
however, there seem to have been 2 votes already, 1 for (a), and 1 for (b).

I've just voted for (a).

My reasoning was:
The syntax may be special and different from nowadays formats JSON, XML, etc, but it is so simple and compact
that it can be immediately understood.
It is human-readable, compact, can be easily diffed using text comparison tools.
It compresses well.
It can be easily extended using our 0 !xyz syntax.
Writing a parser for it is quite simple.

I do not like the idea of putting second-level wrapper file(s) around the existing *.dat's.

In the long run, however, we might need a more fundamental change:
I think it would be better to store parts in JSON format instead of the proprietary *.dat syntax.
They could reference each other as usual.
It would be much easier to add new syntax to them.

However, that solution is only feasible if we have a converter tool from *.dat to the new JSON standard
AND (!) back.

That'll be the only way to still be able to use the existing tooling.

One after the other, the tooling could learn to use the new JSON syntax.

HOWEVER, at the moment
- no such converter tooling exists
- no JSON syntax proposal exists
, so in THAT case my vote is (a)
until we have these available.

Creating a wrapper layer to me just appears as an intermediate solution or - to use the hard word - workaround.

My suggestion is to create a JSON syntax suggestion and converter tools,
then setup a new vote.


Re: Modernizing The Spec [Poll] - Orion Pobursky - 2014-12-27

My intention with this question was to take the "long view" and not necessarily consider what we have right now but where we want to head towards in the future. Converters are easy to write.

I like the idea of keeping the DAT syntax as it is right now and adding all the non-geometry stuff (i.e. normals, textures, etc) into a wrapper file. This will not break backwards compatibility as the existing software doesn't know about that stuff anyway. As long as a DAT file exists for every part and we don't define any part geometry stuff in the wrapper file then we don't need to worry about old software.


Re: Modernizing The Spec [Poll] - Ben Supnik - 2014-12-28

Hi Y'all,

It seems like this poll is an attempt to give ourselves permission to add to the .ldr file format more aggressively - in the past, some combination of cultural inertia and the need for backward compatibility with end-of-lifed apps has made .ldr extensions "off limits".

So I'd like to argue for -direct- extension of the LDraw format. I think in the long run it will produce the clearest, most understandable .ldr files; if everything is wrapped, interaction between extensions and scope is going to be more complicated.

There are two kinds of possible extensions:
1. Extensions where backward compatibility is easy, because the extension can be ignored (e.g. the proposed META to re-specify the origin.
2. Extensions where backward compatibility is hard because the extension -replaces- existing functionality (e.g. you can have one META sphere command instead of 300 triangle primitives).

The current texture extension tries to be in category 1 by providing backup geometry - you can ignore all of the new stuff and just draw the backup geometry. But to be really useful the TEXMAP extension needs to be in category 2, e.g. there certainly shouldn't be a backup triangle-based pattern to the part, because the whole point of texturing extensions is to save work for authors by letting them use a PNG file for patterns.

With that in mind, we can (relatively) easily provide backward compatibility against category 1 extensions (that are DIRECTLY in the .ldr and .dat files) by making a processor that strips them out, and making a second downloadable edition of the library that is stripped of new stuff.

I like the idea of a post-processed distribution more than wrappers because a second distro is a one-time tax on development - we have to develop and deploy the tools, where-as having wrapper files is a tax on development of every single program that ever parsers a new .ldr file for as long as we go forward.

For category 2 extensions, there's no way to win - if the newly authored part is truly not backward compatible with older clients for some reason, we have to either drop the clients, author 2 versions of the part, or not use the new extension. But that's something that needs to be decided on an extension-by-extension basis. For example, texturing was authored to intentionally make the backward compatibility case "ugly but not fatal" - you can easily have a solid triangle replacing your textured triangle - the part looks unpatterned but doesn't look like a bug. As we add new extensions, we'll have to consider this.

With that in mind, at least some of the things that have been tossed around that are somewhat 'major':
- Better normal vector support
- LOD
- Connectivity
- Meta data to "fix" authoring issues

Are all pretty clearly in category 1.

Two more comments on this:
- I do have a plan^H^H^H^Hpipe dream of doing some kind of connectivity in Bricksmith in the next year or so...and I strongly believe connectivity is best served with an in-file-format extension. Because LDraw parts are constructed from primitives, we can (in some cases) annotate a small number of primitives with connectivity meta-data and get a pretty big win in terms of part coverage. Doing that with a wrapper is going to make such an extension significantly less intuitive than it would otherwise be. (I think you can demonstrate that a wrapper system is functionally equivalent to extending the format - but it still puts a tax on all programs dealing with the wrappers.)

- When it comes to LOD, there was some support for LOD via a parallel file system, instead of in-file-format extensions. It's already how LOd is partly being dealt with, so there's precedent, and it's very unobtrusive. I don't think this parallel-library idea is wrong, and it wouldn't be crazy to do LOD that way even if most other extensions are going to be in-file.

Cheers
Ben


Re: Modernizing The Spec [Poll] - Orion Pobursky - 2014-12-31

Ben Supnik Wrote:It seems like this poll is an attempt to give ourselves permission to add to the .ldr file format more aggressively - in the past, some combination of cultural inertia and the need for backward compatibility with end-of-lifed apps has made .ldr extensions "off limits".

The "end-of-lifed apps" you mention also happen to be the most popular for creating models with the library. I'd like to at least try to not break compatibility with them unless we have a really good reason. See as that it's not really that hard to keep backward compatibility as long as we don't add different geometry commands (and even those could be approximated with primitives) this really isn't an issue


Re: Modernizing The Spec [Poll] - Ben Supnik - 2015-01-01

Orion Pobursky Wrote:The "end-of-lifed apps" you mention also happen to be the most popular for creating models with the library. I'd like to at least try to not break compatibility with them unless we have a really good reason.

Right - I figured it was the strong popularity of older apps that aren't being actively developed that was the strong motivation to try to keep parts completely backward compatible.

My view is that -if- we have a breaking directive in the format, a 'filtered' library is a nice way to have both old-program compatibility and new options...clearly we wouldn't go to the extra work or add that level of complexity if a backward compatible directive is possible.

Cheers
Ben


Re: Modernizing The Spec [Poll] - Jude Parrill - 2015-01-02

I'm abstaining from voting because a) I'm not application/program developer of LDraw tools, and have no plan to do so in the immediate future, and b) I'm barely a part author/reviewer, although that mainly has to do with my work lately which has been crazy.

However, this is a topic I feel is very important for this community. Moving forward, I feel like LDraw needs to modernize and keep up with technology, or risk becoming out-dated, out-moded, and otherwise fading into obscurity.

I also feel this isn't matter that shouldn't be jumped into lightly or quickly, but one that carefully considered by all. The LDraw community needs to reach out to developers, authors, and reviewers, past and present, and look for ideas and opinions from them. Then, as much as is possible, form a plan that suits as many as possible while still pushing the standard forward into the future.

I very often feel that the LDraw community is too bogged down and mired in the past, clinging desperately to the dream of perfect backwards compatibility. And while this is a very noble cause, I feel it has caused this community to stagnate and decay. Instead of welcoming new ideas and new technologies, they are often viewed fearfully with the phrase "but that could break backwards compatibility" being uttered, while others nod along solumnly, "we must never break the backwards compatibility."

Now, I apologise if this sounds harsh or mean, as that is not my intention. Rather, it is my wish to galvanize this community into action. Don't fear the Spector of the Past, but have a Vision of the Future (bonus nerd points if you get this reference). Think forward, not backwards.

Please don't misunderstand my intent here, either. I'm not suggesting you throw out everything and start fresh. I'm not suggesting you wantonly throw out backwards compatibility altogether. I'm suggesting you find a way to work with everyone... developers, authors, reviewers, etc... and find a way to move into the future together. Because yes, it's likely going to result in a lot of work... FOR EVERYONE! And I know that's not easy or fun. But if done correctly, if done as a team... as a community... as a group of people working to make something bigger and better than the sum of it's parts, well, then I think it could be something pretty special.


Re: Modernizing The Spec [Poll] - Merlijn Wissink - 2015-01-02

I completely agree to you.

Everyone keeps thinking about backwards compatibilty, but sometimes you just have to go forward. All major programs and filetypes have different versions that are incompatible with each other. For example Microsoft Word: there's .doc and .docx. They also had to decide at some point: "Screw this, we update everyhing." And, although there's no direct compatibilty between the two files, they can be converted into one another.

And, that's what I'm sure about: if we end up with something that's not directly backwards compatible, I'm sure there's someone who will write a converter.

But, I also understand why people fear to break backwards combatibily. It's a circle that difficult to get out: the most popular programs use the old standard, if you break the standard you can't use the old software anymore. But, new software won't be created as quick as it could be, because the standard is quite old.

It's a difficult decision.