[LSC Request] End of header meta command

[Tim Gould]

I think it would be really nice to have a meta command to indicate the end of the official part header information (especially) but also headers in general. Just something simple like:

Code:

0 !ENDOFHEADER

That way codes like LDMakeList of LDFind can know which comments are useful notes for the header (eg. in a needs work file) and which are just separators of the different part components (eg. 0 // Studs).

Of all the meta commands ever this one (hopefully) should be the easiest

eg.

The ENDOFHEADER meta-command indicates that the header has ended, and that any software that processes the header can stop looking. It is not required.

Code:

0 Brick  1 x  4
0 Name: 3010.dat
0 Author: James Jessiman
0 !LDRAW_ORG Part UPDATE 2003-02
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt

0 BFC CERTIFY CW

0 !HISTORY 2001-10-26 [PTadmin] Official Update 2001-01
0 !HISTORY 2002-05-07 [unknown] BFC Certification
0 !HISTORY 2002-06-11 [PTadmin] Official Update 2002-03
0 !HISTORY 2002-12-12 [hafhead] New subparted version
0 !HISTORY 2003-08-01 [PTadmin] Official Update 2003-02
0 !HISTORY 2007-06-06 [PTadmin] Header formatted for Contributor Agreement
0 !HISTORY 2008-07-01 [PTadmin] Official Update 2008-01

0 // This file has been updated often
0 !ENDOFHEADER
0 // Include the sub file
1 16 0 0 0 1 0 0 0 1 0 0 0 1 s\3010s01.dat
0 // Add a quad on top
4 16 -40 0 -10 40 0 -10 40 24 -10 -40 24 -10
0

[Michael Heidemann]

Yes, such a marker would help a lot to sort the comments in the file. But the only comments that are useful are at "Needs work" parts. As there has to be a comment telling us what is not correct with the part. All other official parts can stay like they are.

LDFind doesn't care about where in the file the data are stored. The whole file is scanned. As LDFind also should be usable with unofficial files i can not rely in any case on our official standards regarding the header

[Allen Smith]

The lack of such a meta command is a tremendous headache. It would make my life easier if it were mandatory and the whole part library had it.

Allen

[Tim Gould]

That would be even better.

I've been calling the header done at the end of file, appearance of this meta (which never appears, but the code is there), or the first occurence of a non-type 0 line.

PS. You might have missed it, but could you send me any or all XML files you generate.

[Orion Pobursky]

I've been calling the header done at the end of file, appearance of this meta (which never appears, but the code is there), or the first occurence of a non-type 0 line.

So why to we need a new meta? The end of the header is already defined as per above (minus the new meta).

[Philippe Hurbain]

One problem is in comments before start of body, that are mistakenly integrated to header.

[Allen Smith]

Because it's ambiguous. Not all type 0 lines are header info.

[Tim Gould]

What Philo and Allen said. See eg.

Code:

0 Animal Rat
0 Name: 40234.dat
0 Author: Philippe Hurbain [Philo]
0 !LDRAW_ORG Part UPDATE 2009-02
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt

0 BFC CERTIFY CCW

0 !HISTORY 2009-09-03 [PTadmin] Official Update 2009-02

0 // Shape acquired with NXT-based 3D scanner
0 // Programmed with pbLua - Thanks, Ralph!
0 // Scans assembled and processed with Meshlab,
0 // a tool developed with the support of the Epoch NOE
[0 !ENDOFHEADER]
0 // Body

[Michael Heidemann]

Exactly. Those informations are good but not header relevant IMHO.

Whereas:

Code:

0 Tile  2 x  2 with Map Red/Blue/Green Border Pattern (Needs Work)
0 Name: 3068bpa0.dat
0 Author: Stan Isachenko [angmarec]
0 !LDRAW_ORG Unofficial_Part
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt

0 BFC CERTIFY CCW
0 !KEYWORDS Adventurers

0 !HISTORY 2011-10-15 [MagFors] Corrected errors

0 // Needs work, many colinear vertices

this comment should go into the header data because it belongs to the "Needs Work" tag in the part description.

[Chris Dee]

It sounds to me like we need a "0 !NEEDSWORK" meta-statement, then any "0 // ..." comment can be treated as not part of the header.

Some already consider the header to be bloated, so adding a mandatory "0 !ENDOFHEADER" would generate more bad vibes.

[Roland Melkert]

I feel an extra meta is somewhat over kill.

Why not use a double empty line or something or some other pattern.

Or move a mandatory header meta to the bottom, so you know the header ends after that/those line(s).

[Tim Gould]

That would work too. Then all 0 // can be ignored.

Although I'd prefer 0 !NOTES (or similar) so we can include other header info. For example if a part is made by a script the settings used to generate it could be included.

And reading of the header can be terminated at the first 0 // or line type 1-5.

Tim

[Chris Dee]

This would also mean that I/we only need to update the 139 official parts the "need work" rather than all official parts.

[Tim Gould]

As far as I'm concerned I'd see it as voluntary. But I guess it would be nice to change the "Needs work" files to the new meta. But they 'need work' anyway so I don't think it's necessary. Your call.

Tim

[Roland Melkert]

If we go this way might I suggest using a more global meta with an eye on forwards compatibility.

Something like

Code:

0 !PTTAGS needsWork, placeHolder, someOtherFutureThing, etcEtcEtc

And also state when this meta is found it also indicates the end of the header. Just my 2cts.

[Travis Cobbs]

I believe that the purpose of the 0 !NEEDSWORK was intended to be as a container for a description of what work is needed. So it would (presumably) be followed by a text description of the work. This wouldn't work with a single meta-command holding multiple flags. Also, "needs work" isn't related to the Part Tracker. It's an acknowledgement that the file has issues, but is still ok for inclusion in the official library once certified.

[Chris Dee]

The meta statement sequence defined in the header spec is enforced for official parts (except maybe that !HELP lines can appear anywhere), and I'd still prefer !HISTORY to be the last header info as that is the part that grows.

My preference would be to add zero or more "0 !NEEDSWORK " meta statements immediately before the !HISTORY.

Every official part has at least one !HISTORY line, because the Parts Update mechanism generates one, so header parsers can then ignore everything after the last !HISTORY line.

[Steffen]

I find the introduction of a dedicated "end of header" meta complete overkill:

The header ends when the first line of type 1,2,3,4 or 5 is encountered.

To achieve this, I think that the introduction of a
0 !NEEDSWORK
meta would be a good idea.

All comments starting with
0 //
are intended for human-reading and should not carry semantic information for tools IMHO.

[Allen Smith]

0 header
0 stuff
0 with no
0 defined ending

0 !TEXMAP etc etc.
1 a real part
0 !TEXMAP END

QED

I happen to agree that the header is too onerous. I have difficulty envisioning usecases for some of its contents. The !HISTORY lines in particular belong in CMS metadata; they do not belong in the source file itself. I don't see much value in adding a !NEEDSWORK because I don't see why that data needs to be machine-readable either.

But you cannot make the case that the header is currently machine-parseable. Nor can you, without an end-of-header designator, possibly write a future-proof header delineator.

I'm not going to get worked up about this; there are far more important issues. I'm just trying to set the record straight.

Allen

[Roland Melkert]

I believe that the purpose of the 0 !NEEDSWORK was intended to be as a container for a description of what work is needed. So it would (presumably) be followed by a text description of the work.

You could still follow with normal comment lines, why introduce yet another comment like meta?

I still think you could indicate end of header with a pattern (double empty line for example)

[Chris Dee]

The !HISTORY lines in particular belong in CMS metadata; they do not belong in the source file itself.

The textual content of the !HISTORY lines would belong in a Change Management System if the Parts Tracker used one (which it currently doesn't). However, the

Code:

0 !HISTORY YYYY-MM-DD [PTadmin] Official Update YYYY-RR

lines serve as a surrogate for a version number, so do have value to identify which version of an official part you have installed, and since they are programmatically generated are parse-able.

[Tim Gould]

I'd like to revive this discussion with a very simple suggestion:

What if we terminate the header either at the first instance of a non-type-0 or at the first appearance of

Code:

0 ////

It's simple, fits existing standards, is easy to read algorithmically, and won't cause problems if it appears elsewhere.

Allen> A "needs work" tag is useful as machine-readable as it allows tools e.g. LDFind, to easily point out that a part could do with some touching up. Thus part authors can get a list of parts that they might improve.

Tim

[Chris Dee]

The Parts Tracker seems to cope well with identifying the end of the header section as a non-type-0 line OR a 0 BFC INVERTNEXT.

[Tim Gould]

Except it often gobbles up comments from within the part data. There are reasons to keep those internal comments away from end-users.

I'm certainly not proposing it as a compulsory solution, merely an optional one (and if DATHeader implements it then it will become a de facto standard).

Tim

[Tim Gould]

Agree. But not double lines as they are too easy to add by mistake. I've suggested "0 ////" below.

Tim

[Travis Cobbs]

I don't have a particular problem with an optional end-of-header token, but I don't feel that your proposed one is acceptable. We've already defined 0 // to be a comment which should never, under any circumstances, be treated as meta-information about the file. Your proposal goes against that.

[Santeri Piippo]

Allen> A "needs work" tag is useful as machine-readable as it allows tools e.g. LDFind, to easily point out that a part could do with some touching up. Thus part authors can get a list of parts that they might improve.

I could implement support for this in LDForge if it is ratified. I also would find the end-of-header meta command particularly useful. I think "////" sounds a little ambigous and unclear though. I'd prefer "0 !END_OF_HEADER" or something like that.

[Roland Melkert]

I agree with Travis '0 //' should be protected.

Having reread the thread I'm ok with the original !ENDOFHEADER suggestion.

It should be optional though, when it's not present the header stops when the first 'mesh' related line is read (1..5, bfc, tex, etc).

[Steffen]

no, no, no, I still strongly dislike the introduction of a new end-of-header token.

the existing defined semantics are enough to write some simple code to detect the end of the header: it is simply there where the first non-comment line comes, ie that is a line which is of type 1, 2, 3, 4 or 5, or a 0 BFC or 0 !TEXMAP or 0 !COLOUR

let me also question for what _purpose_ such a token would be useful:
1. what is so bad about LDFind if it indexes the whole file instead of just the first bunch of lines? don't you want to find _all_ occurrences of a word in a file?
2. if we're talking about using a revision control system for our parts like git, hg or Perforce, which I strongly suggest, then the _whole_ file will go into it, not just the header. That way our files would really get a lifecycle history of their full contents. We're talking about simple
fulltext files here still, which can easily be stored in tools like that, allowing easy diffing etc like for programming language source code files.

And yes, agreeing to the said things above: using anything that starts with 0 // must not be done under all circumstances,
because it breaks the rule that everything after 0 // is a totally free formatted,
no machine-readable-semantics-carrying human-readable comment.
the only degree of freedom we have here is choosing something that starts with 0 !...

[Tim Gould]

Which of these things could you not do with an optional end of header token?

[Steffen]

the question is the other way around: why introduce it when you can do without.

[Michael Horvath]

That could also be solved by creating a new line type just for headers.

[Tim Gould]

Well there are plenty of reasons given in this thread as to why it would be useful. So the onus is on you, I'm afraid.

So again I ask, why do you think we should not have it?

[Tim Gould]

That's a good solution. But you've seen how hard it is even to get a simple optional meta added. Do you really think we're going to add a new linetype?

[Michael Horvath]

A couple of reasons in favor of the new meta command:

1. Without it software authors will need to write extra code which takes time and effort.
2. Also, software will become more complex and relatively more likely to have bugs.
3. Extra code will slow the software down, which can matter maybe when batch operating on a long list of files.
4. Having a command like this is less ambiguous for new part or model authors, meaning they will not get as confused.

[Steffen]

no, I've read all the posts. none of them carry an argument why you need a new token and cannot do without. they just talk about what could be done with some header data,
but they do not state why the existing stuff isn't enough.
I understand that you really want that new token, but each time you want to add something to such a standard, you need to prove why it is needed and existing stuff isn't enough.
it is not up to me to prove the opposite.
we need to minimize the syntax, avoid clutter and save time.
and the existing syntax suffices to detect the end of the header.
so that's 4 arguments you need to counter.
there's a 5th: if you make the tag optional, the tools supporting it anyway need a fallback implementation for files not carrying that token. and that fallback implementation would exactly read as I wrote above. Thus, this code anyway needs to be written. Thus, you do not need the tag. The code already suffices.

[Roland Melkert]

I assumed it's an presentation issue, you don't want to see the first part author comment block directly after the true header in e.g. a part description while using the part bin.

The meta would allow for nicer presentation of part info hints etc.

[Tim Gould]

> no, I've read all the posts. none of them carry an argument why you need a new token and cannot do without. they just talk about what could be done with some header data,

Not to you. To others they do.

> but they do not state why the existing stuff isn't enough.

Yes they do.

> I understand that you really want that new token, but each time you want to add something to such a standard, you need to prove why it is needed and existing stuff isn't enough.
it is not up to me to prove the opposite.

Not true. That's merely an argument that the conservative position is the right position. I take a progressive position that an improvement is worth doing unless it has negative consequences that outweight its benefits. This doesn't.

> we need to minimize the syntax, avoid clutter and save time.

I do agree with this to some degree. But the new meta would avoid a different sort of clutter (see point below).

> and the existing syntax suffices to detect the end of the header.

As has been pointed out several times, no they don't. They suffice to define an end of the header + lines of comment. Go look at the number of parts on the parts tracker with "0 // Some comment that has no place in the header" at the end of the header.

> there's a 5th: if you make the tag optional, the all tools anyway need a fallback implementation for files not carrying that token. and that fallback implementatioh would exactly read as I wrote above. Thus, this code anyway needs to be written. Thus, you do not need the tag. The code already suffices.

Ideally I'd like to see it made compulsory to ease coding (see Michael and Allen's posts) but I know that would never happen due to LDraw conservatism. Conversion of the library could be done using the existing rules.

This argument is basically "because you offered a compromise your proposal obviously has no merit".

[Tim Gould]

Let's drop "0 ////". I agree that it's a bad choice.

If we can settle on a meta I'll add it to LDMakeList too. It will make for a cleaner parts.xml file.

Tim

[Tim Gould]

The meta would allow for nicer presentation of part info hints etc.

Exactly. It also makes it slightly easier to read by a person who can see that "oh! the header info ends here, the rest is part author chit chat".

[Ben Supnik]

I agree that the burden of proof for format extensions needs to be for why it is necessary.

But I think Roland has found one: the implicit algorithm can't tell the 'header' from in-part comments - they are different semantically but not syntactically; a new meta would make this visible in syntax for automated processing.

(If you are right and the header can be found strictly by existing algorithm, then formalizing that algorithm is valuable.)

[Tim Gould]

But I think Roland has found one: the implicit algorithm can't tell the 'header' from in-part comments - they are different semantically but not syntactically; a new meta would make this visible in syntax for automated processing.

I hope people realise that example was given in my original proposal and was the main reason for the proposal...

That way codes like LDMakeList of LDFind can know which comments are useful notes for the header (eg. in a needs work file) and which are just separators of the different part components (eg. 0 // Studs).

Tim

[Philippe Hurbain]

Except it often gobbles up comments from within the part data.

...and even BFC invertnext if it's the first line in file

[Steffen]

But I think Roland has found one: the implicit algorithm can't tell the 'header' from in-part comments - they are different semantically but not syntactically; a new meta would make this visible in syntax for automated processing.

Yes, that's true. That's the only thing that cannot be achieved by the current syntax.
However: let me counter this with the argument that this special case, where in-file comments
start right after the header, is very seldom. It only happens on exhaustively documented
parts, and my guess is that of all our currently official files this affects at most 10 or 20 or so files.
This doesn't convince me that the new token really is required. It does not really hurt if
these some extra comments are parsed as well, in these few special cases.
Especially since LDFind anyway indexes the whole file, which makes it more powerful.
We're talking here about a small beautification for a vanishing amount of files affected.
I cannot see the benefit. The small gain is outweighed by all the hassle of introducing the new language element.
This is my personal opinion.
In file format specification, minimalism is the way to go. It keeps the complexity of your format small.
If you too liberally add syntax elements, you end up with something as syntax-complex like perl.
You will not want to write an error-free parser for that...

[Steffen]

well, this can be easily repaired _without_ a new token.
it's a simple special case missing in the PT's logic.

[Steffen]

This argument is basically "because you offered a compromise your proposal obviously has no merit".

Yes, exactly. You have to admit that this is a valid counterargument against the new syntax.
If the new syntax is optional, and you anyway have to implement fallback code, then you could use the fallback code from the beginning. That proves the redundancy of the new syntax.

[Philippe Hurbain]

If the new syntax is optional, and you anyway have to implement fallback code, then you could use the fallback code from the beginning. That proves the redundancy of the new syntax.

That's a point. But it doesn't solve the startup comments issue...

[Tim Gould]

This argument is basically "because you offered a compromise your proposal obviously has no merit".

Yes, exactly. You have to admit that this is a valid counterargument against the new syntax.

On the contrary. I think it is close to the worst counterargument possible. Especially since I never mentioned simplification of coding so it was a side show anyway from my perspective.

[Tim Gould]

The issue affects close to every part I ever submit because, like a well-behaved programmer, I comment as much as possible in my files. And after editing a moderately complex but uncommented part recently, I'm very glad I do comment.

So let me summarise the pros.

1) There is minimal hassle in introducing a simple meta like 0 !END_OF_HEADER.
2) Such a simple meta will be parsed correctly by old software, and thus its implementation would not be compulsory.
3) Part authors who use it could add more verbose comments in their files, making re-editing easier.
4) We thus avoid obsfucated LDraw and improve the quality of parts.

On the other hand, for cons we have

1) It is new.

Tim

[Santeri Piippo]

I kind of believe we should try to keep things simple. I'd suggest make END_OF_HEADER mandatory if the body starts with a type-0-line, even if it were a BFC or TEXMAP statement. Thus the end of header is defined as either the first non-type-0 or END_OF_HEADER. No need for complex rules.

This is coming from someone who thinks allowing clock-wise winding for BFC and the ability to change a part's winding mid-file are stupid things, though. But I think we should keep things well-defined so I'm all for this meta-command as well as any NEEDS_WORK meta command.