Re: line endings in LDRAW files (CRLF vs. LF vs. CR)

[Steffen]

I just wanted to reply to http://forums.ldraw.org/showthread.php?tid=4401,4401
, but couldn't, so I am forced to post here.

I think that that voting is on a good way, but I think that the Mac users have been forgotten.
The currently suggested new text says
"it is recommended that applications understand CRLF and LF as line endings".
I think that that should be instead
"it is recommended that applications understand CRLF and LF and CR as line endings".

[Travis Cobbs]

To the best of my knowledge, Macs haven't use CR as a line ending since OS X was released 11 years ago.

[Orion Pobursky]

Additionally, there are no non-OSX compatible LDraw spec compliant programs.
That said, I may have missed the relevant discussion but why are we moving this into the spec at all? Isn't the restriction on official parts good enough? When I wrote the original draft for 1.0.0 I tried to leave the actual text file portion as open as possible. Not to mention that the end user isn't going to care about line endings if they create or edit an LDraw file in a non-LDraw text editor. If anything we should specify that any line ending is compliant and a LDraw compatible editor must be able to handle them.

[Steffen]

> we should specify that any line ending is compliant and a LDraw compatible editor must be able to handle them

fully agreeing here

[Roland Melkert]

I was thinking the same thing, and because it's only a hint to new software authors I don't think it's fair to burden them with supporting something that's never been used in practice. They still can if they want to of course but as a base 'advise' I don't think it's necessary.

[Tim Gould]

What about LFCR? I don't want my Acorn being ignored

Tim

[Allen Smith]

I think that that voting is on a good way, but I think that the Mac users have been forgotten.

If I were at all concerned about this, I would have raised the issue. But I am not concerned in the least.

Mac OS 9-style line endings are an endangered species on the way to functional extinction. They are seldom encountered in any file nowadays; even less so in newly-created files. The only LDraw program which ever used CR line endings was BrickDraw3D, and that project never got far enough along in development to create a model with it. (I couldn't even get it to run recently when I was trying to provide a screenshot of it for a recent comment thread.)

Bricksmith will read CR files, but only to be polite just in case. Any other software author is free to do likewise. But creating a CR-ended LDraw file would be highly inadvisable.

Allen

[Allen Smith]

That said, I may have missed the relevant discussion but why are we moving this into the spec at all? Isn't the restriction on official parts good enough? When I wrote the original draft for 1.0.0 I tried to leave the actual text file portion as open as possible. Not to mention that the end user isn't going to care about line endings if they create or edit an LDraw file in a non-LDraw text editor. If anything we should specify that any line ending is compliant and a LDraw compatible editor must be able to handle them.

In an ideal world, I would agree with you. However, I don't think that's how the ecosystem developed.

I view the spec as a means of assisting prospective developers to read and create well-behaved LDraw files. Unfortunately, a discussion of line endings is necessary to that end. For example, an LDraw program which did not read CRLF files would be useless. And an LDraw program which did not write CRLF files might produce output incompatible with other LDraw programs. I think it's fair to say there's a community expectation of using CRLF, and thus I think it's reasonable to specify using it.

Allen

[Orion Pobursky]

Point taken. Why not then specify (i.e. require) that a LDraw compliant program be able to read all line endings but write CRLF? That makes more sense to me and in the end is less restrictive on the end user.

[Roland Melkert]

That's pretty much what the current vote is about (minus the cr as result of being 'old')

[Orion Pobursky]

No. The current vote only suggests the ability to read nom-CRLF line endings. I think it should be mandatory (minus legacy programs of course)

[Roland Melkert]

If you make support for all formats mandatory, you indirectly make writing of those formats valid. The current vote makes crlf the standard but suggests software to (as a fallback) also look out for linux format endings for legacy support.

[Orion Pobursky]

You misunderstand me. I'm saying the all LDraw compatible programs should write CRLF but be required to be able to read any line ending. This will mean that any file with something other than CRLF will become CRLF when edited and saved in a LDraw program.

[Allen Smith]

You misunderstand me. I'm saying the all LDraw compatible programs should write CRLF but be required to be able to read any line ending. This will mean that any file with something other than CRLF will become CRLF when edited and saved in a LDraw program.

I think what Roland is saying is that if you define all line endings as valid LDraw code, it is illogical to then define only one of those line endings as a valid output format.

Of course, the reason nobody wants files created with something other than CRLF is that we're all afraid there are programs floating around out there which will only accept CRLF (and presumably those programs are getting minimal if any maintenance). It's for that reason I didn't want to make all formats mandatory—it sets a potentially unrealistic expectation.

Allen

[Roland Melkert]

I did understand, but imho dictating software (reading) behavior is not part of a standard. A standard should only describe the file format. Suggestions and 'tips' should only be present in a footnote like manner. It's up to the authors to decide how clever and fool proof they make the reading code in dealing with 'non standard' files (aka Quirks mode).

I don't know how the others think about this.

[Orion Pobursky]

You make a good point. I withdraw my objection.

[Steffen]

I fully agree with Roland.

Normally, I'd suggest to define a standard
"a LDRAW file may contain any line ending, be it CRLF or LF" or even
"a LDRAW file may contain any line ending, be it CRLF or LF or CR or LFCR"

But as our existing software, getting no maintenance anymore,
only tolerates CRLF, we should specify
"LDRAW files shall have CRLF as line endings".
This will allow the existing software to read them.

At the same time we should, independently from the file syntax specification,
recommend to tool authors to also tolerate other line endings when reading.
When writing, they always should output CRLF.

[Greg Teft]

As a coder, I thought I'd throw my two cents in.
It is reasonable to expect character- & byte- based parsers and tokenizers to use CRLF as a definitive record break per the long standing specification.
As I've been working on a Java API that will parse and write to the LDraw specs, I used core Stream readers that read a line at a time. I then 'trim' so that the line type is the first character, and there's no trailing white space.
When this runs in a Unix-style system, using just CR as a line-break, trimming also makes the LF's magically go away as they are considered white space.
When I write a file back out I explicitly use the CRLF by emitting the ASCII equivalents of those two characters (0x0D, 0x0A) instead of the various ways insert a line-break using the core API.

Modern applications should be able to parse either way, but in support of long-established standards, they should write the CRLF.

[Don Heyse]

Speaking of stream readers, could it possibly be a problem with piping ldraw files between processes if the CRLF is omitted from the last line? The new wording seems to allow this. I'm pretty sure that if you have an ldraw processing program reading from stdin on a terminal (in canonical mode) and you paste the text of an ldraw file without the final CRLF, it won't read the last line until you press ENTER.

Same thing if you're reading ldraw data sent via TCP or UDP over the internet. The packets can fragment as they travel from source to destination. So you really have no idea if there are a few more characters needed to finish up your ldraw line stuck out there somewhere, until you get that CRLF.

Or maybe I'm just confused about what the new wording means.

[Greg Teft]

First, I forgot to comment on the last newline discussion. Generally, I wouldn't recommend that files or output written by an application omit that last newline. especially as it's been in the file spec for so long. I was also surprised to see that the last lonely '0' has become discouraged. That was a great way to know the file was complete.

'readLine' operations expect that last newline, and STDIN is a different animal from a File Stream when it comes to really being done. I think shell and perl scripts will consider piped input complete when the first application exits regardless of receiving a final newline, but it's been a while since I coded for it. That also may be a feature of the shell, and not the language. A terminal or console usually won't write a line to the stream until the enter key is pressed. Character and byte operations are generally more flexible and can consume from a stream until it runs out. Although, I've also seen byte and char consumption coded to wait for the newline too before it considers a record or token complete, as it is typically a record delimiter in data file.

When I wrote a piped program in C long, long ago, I expected an EOF (Ctrl-D) or to indicate DONE! but the applications were coded to talk to each other, and networks weren't nearly as fast as they are now. With the advent of Java's Remote Method Invocation (RMI) and HTTP put/post, I've relied on those mechanisms to handle the communication between processes for quite some time now.

Can I ask what you're coding in? I can try a few different things and see how they behave. I can handle Java, most shells, perl, and I think I have an ANSI C/C++ compiler on Linux. I don't currently have .net on my PC.

[Don Heyse]

I try to stick to coding with gcc since that's available for just about every sort of computer. But I haven't done any sort of ldraw coding in almost a year now. I might try to finish implementing the texture mapping extensions this summer though. That seems to be heating up.

I was just sorta wondering out loud about pipes and the possible impact of dropping the final CRLF because once upon a time I had a pipe capable version of ldglite working with immediate mode graphics. And there was also a piped version of lsynth.

[Roland Melkert]

No one is stopping you from adding a last new line or '0', it's just removed from the core spec. Also the official library guidelines could still demand it (don't know if it already does).

As for piping I unless you're piping from std in I don't think it (should) matter(s) if you send a final linebreak, like suggested above. And when using std in your still don't know if it's done or just a empty line so an additional 'eof' like character is still needed far as I know.