Official Library File Format Restictions

[Orion Pobursky]

As part of our meeting last year in Billund, we expressed a desired to make the learning curve for authoring parts a little less steep. As part of that, I'm working on clarifying the rules for the Official Library. Recently we've had a discussion about complete assemblies and disallowing sticker shortcuts for stickers applied to single, flat parts. As a result of those discussion, I made a draft revision to the File Format Restrictions for the Official Library. Proposed changes in red:
https://www.ldraw.org/hidden-content/dra...brary.html

I also posted about my concerns about documentation fragmentation. I personally feel that the File Format Restrictions for the Official Library document should one single source for all the rules for parts in the official library. With that reasoning in mind, I think "Official Library Policy On Embedding POV-Ray Code" should be cancelled since those METAs are already disallowed. Also, the "Parts numbering scheme for parts with unknown numbers" and "Official Library Specification for Sticker Parts" should be folded into the main document. There are also some pages on the PT that can be added as well.

Basically, I'm opening the entire "File Format Restrictions for the Official Library" document up for revision. How should this document be laid out? What else should be included? I have the ability to separate sections in to tabs and other formatting tricks. I also think a downloadable PDF for quick, offline reference might be a good idea. Maybe even a Quick Reference One Sheet for commonly used info.

I'd like to have a full and frank discussion about this. Remember, if you have a problem, propose a solution. Also, we're not going to break backwards compatibility. Aside from those 2 rules almost anything goes.

[Willy Tschager]

Also, the "Parts numbering scheme for parts with unknown numbers" and "Official Library Specification for Sticker Parts" should be folded into the main document. There are also some pages on the PT that can be added as well.

Fine with this.

Basically, I'm opening the entire "File Format Restrictions for the Official Library" document up for revision. How should this document be laid out? What else should be included? I have the ability to separate sections in to tabs and other formatting tricks. I also think a downloadable PDF for quick, offline reference might be a good idea. Maybe even a Quick Reference One Sheet for commonly used info.

No tabs. Would just be the same thing in blue. I'd say one long document with a content table at the top or side as it is now.

w.

[Evert-Jan Boer]

Nice work on the sticker format rules, see I need to do some re-work on formed stickers

What about sticker sheets with more than 26 stickers? Continue with aa, ab, ac etc?

[Gerald Lasser]

I agree with Willy, just wanted to add to the numbering scheme, that we should add also the patterned part references.

Edit:

Section Duplicates:


Identical lines of type 1, 2, 3, 4 and 5 are not permitted.

The order of end points is not considered when checking for identical lines of line types 2 and 5.

The ordering of vertex points for triangles and quads is not considered when checking for identical polygons (line types 3 and 4).

Control points are not considered when checking for identical conditional lines, line type 5

Section Overlapping:
Overlapping Cond-Lines are not permitted, however this is sometimes unavoidable when e.g. two cylinder sections are joined as the condlines are present in the sub-files.
IMHO this also speak against complementary cond-lines that we need.

Section Colors:
This part along with the two sub-items shall be removed as it is outdated:
"There are two cases where use of colour 16 for lines is acceptable:"
Using Lines (type 2) to simulate patterns or grip lines shall not be allowed.

Assemblies:
I think we shall add two samples

May be also add samples here for dual-molded parts or marbled injections?

Unknown Parts:
Shall this still refer to Peeron.com?

[Willy Tschager]

Blanks in description
One thing that is a constant issue for newbies at the PT is the blank in the front of numbers in the description and I couldn't find the specs for it. (This would obviously go into header).

T-Junk
https://wiki.ldraw.org/wiki/T-Junction

Category
https://www.ldraw.org/article/340.html could be integrated into https://www.ldraw.org/article/398.html

Parts that are assemblies
The description on the PT is much clearer and should be copied over from: https://www.ldraw.org/library/tracker/ref/numberfaq/

Sticker numbers
Same applies for sticker numbers: https://www.ldraw.org/library/tracker/ref/numberfaq/ which should be integrated with Sticker Parts Standard

[Chris Dee]

What about sticker sheets with more than 26 stickers? Continue with aa, ab, ac etc?

If there are stickers with mirror image versions, then I suggest using a letter-number suffix for the mirror image pairs for example, a1, a2. This may help to keep within the 26 alpha characters - see here for a specific example. Then aa, ab, et seq.

[Chris Dee]

As part of our meeting last year in Billund, we expressed a desired to make the learning curve for authoring parts a little less steep. As part of that, I'm working on clarifying the rules for the Official Library. Recently we've had a discussion about complete assemblies and disallowing sticker shortcuts for stickers applied to single, flat parts. As a result of those discussion, I made a draft revision to the File Format Restrictions for the Official Library. Proposed changes in red:
https://www.ldraw.org/hidden-content/dra...brary.html

I also posted about my concerns about documentation fragmentation. I personally feel that the File Format Restrictions for the Official Library document should one single source for all the rules for parts in the official library. With that reasoning in mind, I think "Official Library Policy On Embedding POV-Ray Code" should be cancelled since those METAs are already disallowed. Also, the "Parts numbering scheme for parts with unknown numbers" and "Official Library Specification for Sticker Parts" should be folded into the main document. There are also some pages on the PT that can be added as well.

Basically, I'm opening the entire "File Format Restrictions for the Official Library" document up for revision. How should this document be laid out? What else should be included? I have the ability to separate sections in to tabs and other formatting tricks. I also think a downloadable PDF for quick, offline reference might be a good idea. Maybe even a Quick Reference One Sheet for commonly used info.

I'd like to have a full and frank discussion about this. Remember, if you have a problem, propose a solution. Also, we're not going to break backwards compatibility. Aside from those 2 rules almost anything goes.

I would prefer to separate the rules about the content of the files from those about naming the files.

[Orion Pobursky]

I would prefer to separate the rules about the content of the files from those about naming the files.

That's fair a request. I'm in the middle of reformatting the document. It'll be fairly easy to peel off that part into a new document once the text is finalized and ratified.

[Orion Pobursky]

We need to work on this:
https://www.ldraw.org/hidden-content/dra...unknownnum

Now that Peeron is in legacy mode, we should remove references to it.

[Orion Pobursky]

T-Junk
https://wiki.ldraw.org/wiki/T-Junction

T-Junctions are already disallowed (in most cases) in the base spec.

[Orion Pobursky]

I'm making significant document structural changes so I'm going to discontinue highlighting the changes in red

[Orion Pobursky]

That's fair a request. I'm in the middle of reformatting the document. It'll be fairly easy to peel off that part into a new document once the text is finalized and ratified.

Hmm, The Part number FAQ is very comprehensive. Maybe I'll just make a cleaned up and indexed version of that into a new document.

[Chris Dee]

We need to work on this:
https://www.ldraw.org/hidden-content/dra...unknownnum

Now that Peeron is in legacy mode, we should remove references to it.

Agreed.

[Max Martin Richter]

Just some comments from my side

Colour restrictions I thought we stopped the use of the mentioned exception for colour 16 (and other colours) for type 2 lines?

Sticker description May I point to this discussion: https://forums.ldraw.org/thread-23971.html. I would like another line like "the background colour of the sticker must be noted at the end of the description followed by the word background"

/Max

[Orion Pobursky]

I'm not quite done (and there's some weird going on with the TOC) but I am mostly finished with reformatting. I thought I'd share my progress before I turn in for the night.

[Orion Pobursky]

So I'm kinda at a loss as to where to put the last 2 subjects (stud groups and ldconfig). Maybe stugs to the Part Number spec and LDConfig on its own?

[Orion Pobursky]

Stugs have been moved to the Part Number spec. I moved the LDConfig paragraph under the Colour Restictions heading.

Id like more comment. Particularly I'd like comments on the general rules for shortcuts.

[Willy Tschager]

Stugs have been moved to the Part Number spec. I moved the LDConfig paragraph under the Colour Restictions heading.

Id like more comment. Particularly I'd like comments on the general rules for shortcuts.

I'm not keen on shortcuts - though some are surely usefull. In particular I don't like shortcuts with flat stickers on bricks as well as all minifig assemblies that go beond the very basic torso-arms, hip-leg.

You should consider incorporate Colour Definitions https://www.ldraw.org/article/547.html into LDraw.org Standards: Colour Definition Language Extension as a practical example.

There is nothing on patterned parts and the requirment of their plane counterpart in the library.

Nothing on canvas/fabric/cloth - guess it should go into the part numbering and we should address a convention when Canvas or Cloth is used.

w.

[Philippe Hurbain]

Nothing on canvas/fabric/cloth - guess it should go into the part numbering and we should address a convention when Canvas or Cloth is used.

Speaking of canvas... I think it would be a good thing to rename formed cloth from ...c0x to ...-fx ?

[Orion Pobursky]

I don't like shortcuts with flat stickers on bricks

I know we hashed this out already in the closed discussion. I'd like to hear from the larger community

as well as all minifig assemblies that go beond the very basic torso-arms, hip-leg.

I agree and that's in there.

You should consider incorporate Colour Definitions https://www.ldraw.org/article/547.html into LDraw.org Standards: Colour Definition Language Extension as a practical example.

Hmm. That document is really meant to be a quick reference for the current LDConfig.ldr

There is nothing on patterned parts and the requirment of their plane counterpart in the library.

This can be added

Nothing on canvas/fabric/cloth - guess it should go into the part numbering and we should address a convention when Canvas or Cloth is used.

Do you mean have a flat and formed part?

[Orion Pobursky]

Speaking of canvas... I think it would be a good thing to rename formed cloth from ...c0x to ...-fx ?

Yes, I think so.

[Orion Pobursky]

I think we need to rename this document. How about:

Official Library Part Guidelines

or

Official Library Policies and Procedures

[Orion Pobursky]

There is nothing on patterned parts and the requirment of their plane counterpart in the library.

There is now text to this effect.

[N. W. Perry]

I think we need to rename this document. How about:

Official Library Part Guidelines

or

Official Library Policies and Procedures

Perhaps something like "Additional Specifications for Official Parts"? As a new user browsing through the documentation, I found it unclear at first that there is a distinction between the LDraw file format itself, and the stricter standards for the parts library. I think those two core functions of LDraw are somewhat conflated and interchangeable in the mind of the casual user.

[N. W. Perry]

I know we hashed this out already in the closed discussion. I'd like to hear from the larger community

I would agree that shortcuts for stickered bricks are not necessary. They make sense for resale sites like Bricklink, where stickered bricks might exist as physical units in a seller's inventory, but for digital modeling purposes there's no such requirement since stickers can be assembled or disassembled just like any other part.

To me the purpose of shortcuts is so that parts that ship pre-assembled can appear that way in parts lists and set inventories.

[Orion Pobursky]

Those two core functions of LDraw are somewhat conflated and interchangeable in the mind of the casual user.

We're not necessarily catering to the "casual user". In fact, the casual user probably never has need to read either document. Or maybe you mean something different and I'm just misunderstanding.

[N. W. Perry]

We're not necessarily catering to the "casual user". In fact, the casual user probably never has need to read either document. Or maybe you mean something different and I'm just misunderstanding.

The not-already-familiar-with-LDraw-but-looking-to-become-more-so user, perhaps. The person whose interest in LEGO exceeds their familiarity with computers and programming languages, but find the need to move beyond the capabilities of LDD and Studio.

[Willy Tschager]

At the very end you should add this:

https://forums.ldraw.org/thread-22605-po...l#pid32133

w.

[Willy Tschager]

Nothing on canvas/fabric/cloth - guess it should go into the part numbering and we should address a convention when Canvas or Cloth is used.

What I'm talking about is this:

https://forums.ldraw.org/thread-23171.html

and:

https://forums.ldraw.org/thread-23089.html

I think as for sticker, we should somehow formalize them. We have

• Sheet Cardboard
  
• Sheet Fabric
  
• Sheet Plastic
  

As category but nothing about proper naming and we still have:

https://www.ldraw.org/cgi-bin/ptscan.cgi...escription
https://www.ldraw.org/cgi-bin/ptscan.cgi...escription
https://www.ldraw.org/cgi-bin/ptscan.cgi...escription

all over the library along with:

https://www.ldraw.org/cgi-bin/ptscan.cgi...escription

w.

[Gerald Lasser]

Stugs have been moved to the Part Number spec. I moved the LDConfig paragraph under the Colour Restictions heading.

Id like more comment. Particularly I'd like comments on the general rules for shortcuts.

For the sake of completeness, could you add to the STUD numbering the need for the "fast-draw" and the "MLCAD Fallback" Stud? i.e. the reduce resolution for p/8 and the "stu8xx"?

I could not find this in the doc.

[Orion Pobursky]

I want to remove this paragraph completely. We don't follow it consistently and it really is no value added:

  The word (Complete), parenthesis included, should placed at the end of the part
  description name for all shortcuts.
  Use 'Shortcut' when the file is provided for convenience of use
  (example: Minifig files like <979.dat).
  Use 'Complete' when the file represents a composite part, as purchased
  (example: Shock Absorber, 75348.dat).
  [Notice that neither of the example files follow this shortcut/complete standard,
  and the minifig file doesn't follow the file-naming standard,
  either.]

[Orion Pobursky]

I think we need to rename this document. How about:

Official Library Part Guidelines

or

Official Library Policies and Procedures

Any other thoughts on this?

[Orion Pobursky]

What I'm talking about is this:

https://forums.ldraw.org/thread-23171.html

and:

https://forums.ldraw.org/thread-23089.html

I think as for sticker, we should somehow formalize them. We have

• Sheet Cardboard
  
• Sheet Fabric
  
• Sheet Plastic
  

As category but nothing about proper naming and we still have:

https://www.ldraw.org/cgi-bin/ptscan.cgi...escription
https://www.ldraw.org/cgi-bin/ptscan.cgi...escription
https://www.ldraw.org/cgi-bin/ptscan.cgi...escription

all over the library along with:

https://www.ldraw.org/cgi-bin/ptscan.cgi...escription

w.

I've added some text about flexible parts under the appropriately named "Flexible Parts" section.

[Orion Pobursky]

More edits:

I renamed the document to LDraw.org Official Library Specifications
I read through the entire document and reformatted/rewrite for clarity
I added the sticker shortcut stuff that was previously discussed in private forum
I added more of flexible parts

I'm sure I forgot some suggestions. Please read and comment. 

I think we're in the end game. Let's knock this out.

[Magnus Forsberg]

I have some comments on this.

While both upper and lower case letters are permitted in filenames, filenames are case-insensitive. Currently, all official parts are issued with upper-case only names.

I dont understand this:  "all official parts are issued with upper-case only names."

 AFAIK there are no files in the library with a upper case file ending.
Some dat-files may contain references to primitives using .DAT as a file ending, but it is considered an error and are auto-corrected by LDPE, if you tell it to do that, changing the ending to lower case .dat.
DATHeader ignore this, and accept a file using .DAT in the code.
In some files there is a comment about the use of .DAT on stud primitves to please LEdit. I have ignored that and change any old file I edit and replace .DAT with .dat.

I think it needs a clarification.

All files in the LDraw Parts Library are required to carry the .DAT extension.

Same here. What is this wording trying to explain? Why .DAT? Why upper case?

[*]The shortcut may not include more than one part, although that part may be a composite part (one with a "c" in the filename).
[*]The shortcut may not be a flat sticker on the flat surface of the part unless unless that surface is not parallel with one of the 3 cardinal ...

We have lots of composite parts without the "c" in the filename.
"Unless" is duplicated in the second sentence.

0 !KEYWORDS Set 10442-1, Bricklink 973pb0042

The suffix "-1" on set numbers are not needed, unless there are more than one known sets with the same set number.

[Travis Cobbs]

While both upper and lower case letters are permitted in filenames, filenames are case-insensitive. Currently, all official parts are issued with upper-case only names.

I dont understand this:  "all official parts are issued with upper-case only names."

 AFAIK there are no files in the library with a upper case file ending.
Some dat-files may contain references to primitives using .DAT as a file ending, but it is considered an error and are auto-corrected by LDPE, if you tell it to do that, changing the ending to lower case .dat.
DATHeader ignore this, and accept a file using .DAT in the code.
In some files there is a comment about the use of .DAT on stud primitves to please LEdit. I have ignored that and change any old file I edit and replace .DAT with .dat.

I think it needs a clarification.

I agree. Unless I am misunderstanding, complete.zip has for a long time used all lower case letters for 8.3 files. So, the files are actually all lower case, not all upper case. The parts with longer filenames also use all lower case letters, as far as I know. I personally feel that this should be required for parts.

All files in the LDraw Parts Library are required to carry the .DAT extension.

Same here. What is this wording trying to explain? Why .DAT? Why upper case?

I have two things to say here. First, now that textures are allowed, I think it should specify "All part files", not "All files". Secondly, whether this is .DAT or .dat will depend on whether or not the "all upper case" text is change. I think that goes without saying, though.

[Orion Pobursky]

So is the consensus all lower case? Ive confirmed that is the de-facto way we a doing things. 

I'll make the change on the dat extension.

[Travis Cobbs]

So is the consensus all lower case? Ive confirmed that is the de-facto way we a doing things. 

I'll make the change on the dat extension.

Given that the actual files all have lower case names, and in all cases but 2 they reference each other with lower case names, I definitely feel that the document should indicate this.

FYI, the 2 DAT instances are as follows:

Code:

ldraw travis$ find . -name \*.dat -exec grep -H DAT$ {} \;
./parts/6567c01.dat:1 47 0 0 0 1 0 0 0 1 0 0 0 1 s\6567s01.DAT
./parts/6567c02.dat:1 40 0 0 0 1 0 0 0 1 0 0 0 1 s\6567s01.DAT

[Chris Dee]

Given that the actual files all have lower case names, and in all cases but 2 they reference each other with lower case names, I definitely feel that the document should indicate this.

FYI, the 2 DAT instances are as follows:

Code:

ldraw travis$ find . -name \*.dat -exec grep -H DAT$ {} \;
./parts/6567c01.dat:1 47 0 0 0 1 0 0 0 1 0 0 0 1 s\6567s01.DAT
./parts/6567c02.dat:1 40 0 0 0 1 0 0 0 1 0 0 0 1 s\6567s01.DAT

Just to confirm the parts library is published with all lower-case file names. There are way more than 2 examples of upper-case usage on type 1 lines in the official library. Some even have mixed case (e.g. 6104).

I have a vague recollection that in the days of ldraw.exe upper-case STUD.DAT (or maybe all primitive) references on type 1 lines had a special meaning, but cannot remember what. It might have been to do with over-riding lo-res substitution in fast render mode.

[Orion Pobursky]

I made the change to required all lower case. This involve removing the option for A-Z from the allowed characters and changing the note at the end of the paragraph. 

I also changed the extension portion and added a section about texture files.

[Orion Pobursky]

I also changed the extension portion and added a section about texture files.

I just noticed while cleaning up the code that the TEXMAP spec requires PNG. I've changed the texture text to reflect this.

[Magnus Forsberg]

I just noticed while cleaning up the code that the TEXMAP spec requires PNG. I've changed the texture text to reflect this.

Did you miss my other objections/corrections?

[Orion Pobursky]

Did you miss my other objections/corrections?

I forgot, sorry. They have been made.

[Christoph Kubisch]

when programming a ldraw loader and adding bevel edges automatically, I noticed that several parts don't follow those rules (T-junctions existed, overlapping surfaces etc.). Meaning the rules are not followed and I implemented some mesh operations to fix them on the fly. 

Looking at the rules, statements like "must be avoided, but still legal", don't help imo.

Is the plan to robustly identify them, do people go through them manually one by one to have them rebuilt etc.?
Are there tools that do this. Could the tools fix the parts directly? Like one sentence there can have a lot of consequences for the existing content, is that accounted for?

[Orion Pobursky]

when programming a ldraw loader and adding bevel edges automatically, I noticed that several parts don't follow those rules (T-junctions existed, overlapping surfaces etc.). Meaning the rules are not followed and I implemented some mesh operations to fix them on the fly. 

Looking at the rules, statements like "must be avoided, but still legal", don't help imo.

Is the plan to robustly identify them, do people go through them manually one by one to have them rebuilt etc.?
Are there tools that do this. Could the tools fix the parts directly? Like one sentence there can have a lot of consequences for the existing content, is that accounted for?

This is addressed in the intro. These rules have evolved over time. There will be some parts in the Official Library that don't conform. This document applies to all parts awaiting approval and any future official parts that are resubmitted to be fixed.

[Willy Tschager]

Blanks in description
One thing that is a constant issue for newbies at the PT is the blank in the front of numbers in the description and I couldn't find the specs for it. (This would obviously go into header).

Thanks for adding this. However what IMHO is still missing in the example at https://www.ldraw.org/article/398.html:


***********************************************


General

This the descriptive name of the part. If the description contains dimensions, a leading space should be added to single digit numbers to aid in sorting.
Examples:
0 Brick  2 x  4
0 Brick  1 x 10
 (Note the lack of a leading space on the 2 digit number)

***********************************************

is how to deal with blanks in combinations with decimals as in sticker dimensions.

Sticker 3.1 x 3.1 Round

Furthermore this information should be mirrored to https://www.ldraw.org/article/339.html

***********************************************

The description of the part should begin with 'Sticker ...' followed by the dimensions ...

[color=rgba(0, 0, 0, 0.87)][size=small][font=Tahoma, Verdana, Arial, sans-serif]***********************************************[/font][/size][/color]

[color=rgba(0, 0, 0, 0.87)][size=small][font=Tahoma, Verdana, Arial, sans-serif]w.[/font][/size][/color]

[Willy Tschager]

I made the change to required all lower case. This involve removing the option for A-Z from the allowed characters and changing the note at the end of the paragraph. 

I also changed the extension portion and added a section about texture files.

https://www.ldraw.org/article/512.html

Code:

Extension

All files in the LDraw Parts Library are required to carry the .DAT extension.

.DAT has to be substituted by .dat. Otherwise the whole discussion will start over again at the next revision.

w.

[Willy Tschager]

https://www.ldraw.org/article/512.html#colours

Code:

where a line is being used for fine detail within a pattern and that pattern is re-used within many parts (and hence sub-parted) in different colours (for example, sub-part s/973p11a.dat)

We do no longer use emphasis lines. Should be deleted.

w.

[Willy Tschager]

A constant hassle for newbies is the usage of Metallic vs. Chrome in patterns. May be https://www.ldraw.org/article/512.html#ldconfig is the right place to address that in patterns only Metallic is used to represent shiny colors.

w.

[Willy Tschager]

At the very end you should add this:

https://forums.ldraw.org/thread-22605-po...l#pid32133

w.

I still believe that the list what triggers a HOLD should be added and this:

https://www.ldraw.org/library/tracker/ref/l3pmsg/

should be deleted or updated.

w.