Online tool for OMR compliance

[Lasse Deleuran]

When uploading sets to BrickHub.org, it now checks for OMR compliance, and in some cases, can fix errors directly on the site. here is a video showing how to use it. The checks and fixes are currently implemented in this Javascript file: LDROMR.js and are:

• Check and fix file headers so they read "0 FILE <set number> - <name>.ldr" then "0 <description>", followed by "0 Name:<set number> - <name>.ldr".
  
• Ensure the file name is "<set number> - <name>.mpd" with optional qualifiers starting from "-1" (See discussion below)
  
• Check and fix moved parts: If a part is named "~Moved <name>" then replace its usages with "<name>.dat"
  
• Check and fix placements: If a large number of parts are placed with more than three decimals in the position or rotation, then round the placement to three decimals. Example: "1 16 0.0001 0 0 ..." becomes "1 16 0 0 0 ...".
  
• Check and fix all author lines: Ensure all authors are the same and read "!Author Name [username]"
  
• Check and fix all license lines to read "Redistributable under CCAL version 2.0 : see CAreadme.txt"
  
• Set all "0 LDRAW_ORG" lines to read either "Model" or "Unofficial_Model".
  
• Set the "0 LDRAW_ORG Unofficial_Part" in all unofficial parts of the file.
  
• Ensure all unofficial parts are copied into the MPD file and named with the set number prefix in the "0 FILE" and "0 Name:" lines.
  
• If the set is from 2007 or after and contains old colors (6, 7, or 8), then replace them with the newer colored parts.
  
• If the set is from before 2007 and contains new colors (70, 71, or 72), then replace them with the old colored parts.
  
• Inline textured parts according to the non-finalized spec.
  
• Substitute parts that have been replaced by LEGO. As an example, 2470 replaced 4459 in 1990, so the parts will be checked against the year.
  
• Verify that "Theme" lines correctly identify a theme.
  
• Verify that "History" lines are well formed.
  
• Verify that "History" lines are in the correct order.
  
• Check that tyres are colored 256 instead of 0.
  

The purpose of this work is to make it even easier to submit OMR-compliant model file - especially for people who are not using Windows, so they can't easily use MPDCenter.

I do not expect that I have found all OMR checks, and some of my checks might even be erroneous. This is why I am creating this thread. Let's discuss OMR checks and fixes

[Orion Pobursky]

Is this .js file something we can use on LDraw.org or is it tightly coupled to all your other code?

[Philippe Hurbain]

Great tool and tutorial video! I'll check that in more details when I do my next model...

[Eugen]

The purpose of this work is to make it even easier to submit OMR-compliant model file - especially for people who are not using Windows, so they can't easily use MPDCenter.

Is it possible port it to C++? It would be cool merge such OMR checker in open-source cross platform software such as LeoCAD, (that work not only on Windows, but on Linux MacOS and other OS too)

• https://github.com/leozide/leocad/issues/273

[Lasse Deleuran]

Is this .js file something we can use on LDraw.org or is it tightly coupled to all your other code?

I have done my best to make it reusable for others, and focused on basic tree traversal. The Javascript file basically consists of pairs of functions, and converting it to any other object oriented or functional language should be a breeze. 

The potentially difficult part would be to match the data model. The LDraw (or MPD) file structure is parsed into a tree structure: Each 'file' in an MPD file is represented by one type of nodes. Each step within it as a second type of node, and finally each line is a third type of node.

Other programs might not model steps as separate nodes, but as long as you model MPD files as tree structures, then translation to another program should be fairly easy.


I have added documentation to the LDROMR.js file to help potential developers both to add their own checks and to translate everything to another language.

[Lasse Deleuran]

Great tool and tutorial video! I'll check that in more details when I do my next model...

That would be great. And please tell me how it goes.

[Orion Pobursky]

I have done my best to make it reusable for others, and focused on basic tree traversal. The Javascript file basically consists of pairs of functions, and converting it to any other object oriented or functional language should be a breeze. 

The potentially difficult part would be to match the data model. The LDraw (or MPD) file structure is parsed into a tree structure: Each 'file' in an MPD file is represented by one type of nodes. Each step within it as a second type of node, and finally each line is a third type of node.

Other programs might not model steps as separate nodes, but as long as you model MPD files as tree structures, then translation to another program should be fairly easy.


I have added documentation to the LDROMR.js file to help potential developers both to add their own checks and to translate everything to another language.

Awesome. I’ll look into it at some point. 

Santeri is working on a part validator but it’s written in python. I don’t want to step on his toes.

[Lasse Deleuran]

Is it possible port it to C++? It would be cool merge such OMR checker in open-source cross platform software such as LeoCAD, (that work not only on Windows, but on Linux MacOS and other OS too)

• https://github.com/leozide/leocad/issues/273
  

Porting to C++ is easy.
Fitting it into the internal structure might be tricky. I do not know enough of this to give an estimation.

If it fails, then my "sample_functions.htm" file can be used as a starting point for an OMR conversion service. Let's see what people will find useful.

[Willy Tschager]

Lasse,

I tried your tool today with the 926 - Command Centre as for some strange reasons it is in the AIOI but not the OMR. I deliberately added some errors just to see how the site copes with them. Here is the source and the result I downloaded from Brickhub.


  926 - Command Centre.mpd (Size: 15.65 KB / Downloads: 2)

  926 - 926.mpd (Size: 14.71 KB / Downloads: 2)

It picked the {} in the Usename, it fixed the MLCad metas:

0 ROTATION CENTER 0 0 0 1 "Custom"
0 ROTATION CONFIG 0 0

BUT

(please ignore my tone if you might find it harsh ;-)

• It didn't load the MPD name - just the set number and as you can see it returned a 926 - 926.mpd.
  
• I find the page overwhelming at first.
  
• I don't like the fact that the thumbnail generator/instructions/parts is in between the section where I actually edit the header and the suggestions what has to be changed in order to get it OMRized.
  
• I cannot actually see the code itself or at least the header
  
• All line breaks get deleted and you cannot longer view the file in notepad for a quick check.
  
• The site continuous to switch between Unofficial_Model and Model. When I set it to Model it still suggests to set to Unofficial model
  
• I would love to get an indication in the header section itself if for example:
  Author: Willy Tschager [Holly-Wood] ✎
  is right or wrong OMR wise (green tick/red cross)
  
• There is a Download button and a Delete button, where is the upload button when I'm done? I know there is the button which says Upload once you have picked the file from the disk but my understanding is that I LOAD, correct and upload.
  

w.

[Lasse Deleuran]

Lasse,

I tried your tool today with the 926 - Command Centre as for some strange reasons it is in the AIOI but not the OMR. I deliberately added some errors just to see how the site copes with them. Here is the source and the result I downloaded from Brickhub.




It picked the {} in the Usename, it fixed the MLCad metas:

0 ROTATION CENTER 0 0 0 1 "Custom"
0 ROTATION CONFIG 0 0

BUT

(please ignore my tone if you might find it harsh ;-)

• It didn't load the MPD name - just the set number and as you can see it returned a 926 - 926.mpd.
  
• I find the page overwhelming at first.
  
• I don't like the fact that the thumbnail generator/instructions/parts is in between the section where I actually edit the header and the suggestions what has to be changed in order to get it OMRized.
  
• I cannot actually see the code itself or at least the header
  
• All line breaks get deleted and you cannot longer view the file in notepad for a quick check.
  
• The site continuous to switch between Unofficial_Model and Model. When I set it to Model it still suggests to set to Unofficial model
  
• I would love to get an indication in the header section itself if for example:
  Author: Willy Tschager [Holly-Wood] ✎
  is right or wrong OMR wise (green tick/red cross)
  
• There is a Download button and a Delete button, where is the upload button when I'm done? I know there is the button which says Upload once you have picked the file from the disk but my understanding is that I LOAD, correct and upload.
  

w.

First of all. Thanks for the feedback and I'm sorry I could not come back to this earlier. I'm used to a much harsher tone in my everyday work, so it is no problem that you write it all up in brief form. I can also understand the frustration, so let's dive into the points:

• File name being "926 - 926.mpd": I found the bug in the uploader. This has been fixed now. Uploading your file results in the name being read correctly: "Command Centre - Main File". Thanks for reporting it!
  
• Page being overwhelming: I have reduced the amount of text and done some restructuring. I hope it is less overwhelming now, but I would still like to take suggestions.
  
• Suggestions below image: Good point. The suggestions have now been moved up above the image.
  
• View the file source code in the page: I am working on a solution for this. The parts list has a pop-up for Bricklink wanted list code, so this feature should be easy to add for the file content.
  
• Line breaks: The code was saving with Linux line ends. I have changed it to Windows style, which is also more correct according to the standard.
  
• Unofficial_Model vs Official Model: It was not easy to make a good decision here. I want people to be able to change to either of these values, but this also causes you to be able to change back and forth all the time. I have come up with a new solution. When you set it to 'Model' it is unlikely that you want to change it back to unofficial, so I have removed the option - unless you just pressed 'Model' - then you are still able to press 'Unofficial' in case you pressed the other button by mistake.
  
• Having red/green ticks next to the header fields: This makes sense. I will look into a way where this will not be too cluttered while still showing the OMR messages - this is currently a challenge for me to get right for users on touch-based devices, hence the more blunt OMR line blocks below.
  
• Download vs Upload: I am not sure I understand this correctly. 'Upload' always means computer->website, while 'Download' means website->computer.
  

I will return once I have a solution for the being able to view the file content.

And again. Please bring in more suggestions.

[Willy Tschager]

File name being "926 - 926.mpd": I found the bug in the uploader. This has been fixed now. Uploading your file results in the name being read correctly: "Command Centre - Main File". Thanks for reporting it!

Now it loads:

Surface Explorer - Main Model

and saves:

6880 - Surface Explorer - Main Model.mpd

while my file was:

6880 - Surface Explorer.mpd

So this is still messy. I expected the mpd name 6880 - Surface Explorer to show up and save the same name if untouched.

Download vs Upload: I am not sure I understand this correctly. 'Upload' always means computer->website, while 'Download' means website->computer.

What I expected was that the file doesn't end up immediately at my Brickhub page. I see the OMRizer more as a test/edit page before your model gets stored on Brickhub. So I log in, LOAD my file, do all the needed editing and then DOWNLOAD/UPLOAD/DELETE the model. I'm somewhat confused that More models uploaded by Holly Wood is on the same page. The OMRized should be more of a subset where I edit a particular file I pick from My page.

w.

[Lasse Deleuran]

As it is right now, OMR compliance checks is something that are triggered when an official model is detected. I would prefer if all official models on brickhub.org also became OMR compliant. You are right that it seems a bit strange to see your other models in the bottom of the page, so I have removed that feature. To "publish" a file, the toggle "Show on front page and in searches" is used. I feel this formulation is less technical than "SUBMIT"/"Publish" or similar.



I have updated the page to give a more intuitive representation of the connection between set number, set name and the file which is downloaded. When selecting "Official LEGO set", the title becomes the file name and by clicking on either the set number or the set name, these become editable (the dash and ".mpd" are fixed). This was the best I could come up with tonight. I am open to better suggestions.

The OMR function for updating header lines has also been improved. It now tries to insert the correct set number according to the one in the title (previously it simply appended, resulting in non-optimal file names).

Another fix has been in detection of suffixes for files. Assume the set number is "1234". Now when it detects:

1234 - Model.mpd
1234 - Model
Model.ldr
1234-Model.ldr
43849 - Model.ldr

it will correct to:

1234 - Model.ldr

A lot of the files in the AIOI have "1234 - Model.mpd" instead og "1234 - Model.ldr". Is this technically OMR compliant?

[Johann Eisner]

In my opinion, .ldr is wrong and .mpd is correct.

Please refer LDraw.org Standards: Official Model Repository (OMR) Specification Version 1.0.3

Has been discussed here before: https://forums.ldraw.org/thread-23070.html

Excerpt from the OMR Specification:

Base File Naming
Each model in the OMR will consist of several files that are packaged together into a single MPD file. For sets that include instructions for multiple models, each model will have its own MPD file. Each MPD for the set will be named in the following manner:

Code:

<Set Number>[-<Optional Qualifier>] - <Set Name>[ - <Sub Model Name>]

Where:

Code:

<Set Number>

: The number assigned on the container of the set.

Code:

<Optional Qualifier>

: Is a sequential number, starting with 2.

Code:

<Set Name>

: The name of the set printed on the container in English.

Code:

<Sub Model Name>

: This is Optional in most cases. This is required for alternate models that are detailed in instructions (e.g. the Creator theme). In this case the naming is left to the discretion of the author but should be descriptive of the model contained in the MPD.
The

Code:

<Optional Qualifier>

is not mandatory and gets added only if there is more than one set that could be assigned the same

Code:

<Set Number>

. The first set using a given number would be understood to never contain the qualifier however numbering should start with the oldest set and some investigation should be done in existing set databases.
Example:

Code:

6901 - Mobile Lab.mpd (Produced in 1980)
6901-2 - Space Plane.mpd (Produced in 1998)

For playsets or other sets where there are multiple models that are part of an integral whole, all of the submodels will be contained in one MPD.
Example:
The creator set 4896 - Roaring Roadsters has 3 models in the instructions:

Code:

4896 - Roaring Roadsters - Roadster.mpd
4896 - Roaring Roadsters - Dragster.mpd
4896 - Roaring Roadsters - SUV.mpd

[Lasse Deleuran]

The thing is. The example in the Specification has it the other way around with only ".ldr" sub files within a ".mpd" enclosing file:

Code:

0 FILE 6712 - Main.ldr
0 Main
0 Name: 6712 - Main.ldr
0 Author: Willy Tschager [Holly-Wood]
0 !LDRAW_ORG Model
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt

1 0 0 0 0 1 0 0 0 1 0 0 0 1 6712 - Fireplace.ldr
1 0 -140 -49 -32 0.5 0 -0.866 0 1 0 0.866 0 0.5 6712 - Horse.ldr
1 0 -140.866 -129 -31.5 0.5 0 -0.866 0 1 0 0.866 0 0.5 6712 - Sheriff.ldr
1 0 83 -99 64 1 0 0 0 1 0 0 0 1 6712 - Bandit.ldr
1 6 81 3 76 1 0 0 0 1 0 0 0 1 30139.dat
1 2 -51 8 86 1 0 0 0 1 0 0 0 1 6064.dat
1 7 80 0 -14 0.707 0 0.707 0 1 0 -0.707 0 0.707 3069bp03.dat

0 FILE 6712 - Horse.ldr
0 Horse
0 Name: 6712 - Horse.ldr
0 Author: Willy Tschager [Holly-Wood]

[Orion Pobursky]

Nowhere in the spec does it require a file to have a mpd extension. Since a MPD document is defined by the 0 FILE meta and not the file extension it is acceptable for the base file to have either a ldr or mpd extension.

[Johann Eisner]

Sorry, I misinterpreted the text. ?
I thought it was about the main file, not the submodel files.
Have a nice sunday.

[Lasse Deleuran]

Sorry, I misinterpreted the text. ?
I thought it was about the main file, not the submodel files.
Have a nice sunday.

No worries. I will try to let the page be as is and test it for helping with OMR compliance for some models that I have planned.

[Lasse Deleuran]

I am working on improving the OMR tool, but I have stumbled on an inconsistency of the optional qualifier in the spec.

For the file name it says:

Code:

<Optional Qualifier>

: Is a sequential number, starting with 2.

While for the MPD file structure we have:

Code:

<Optional Qualifier>

is a sequential number, starting with 1, added if there is more than one set that could be assigned <Set Number>.

Is it on purpose that these two differ?

[Orion Pobursky]

I am working on improving the OMR tool, but I have stumbled on an inconsistency of the optional qualifier in the spec.

For the file name it says:




While for the MPD file structure we have:




Is it on purpose that these two differ?

Nope, prolly a typo. Both should be 1.

[Lasse Deleuran]

Nope, prolly a typo. Both should be 1.

That would also be my first guess, given that many OMR files have the "-1" qualifier. But then the following text is incorrect in the spec:

The <Optional Qualifier> is not mandatory and gets added only if there is more than one set that could be assigned the same <Set Number>.

The first set using a given number would be understood to never contain the qualifier however numbering should start with the oldest set and some investigation should be done in existing set databases.

Example:

Code:

6901 - Mobile Lab.mpd (Produced in 1980)
6901-2 - Space Plane.mpd (Produced in 1998)

[N. W. Perry]

Nope, prolly a typo. Both should be 1.

This is the same discrepancy I inquired about here. I likewise assumed it must be a typo—but should they not both be 2? The spec says:

The first set using a given number would be understood to never contain the qualifier however numbering should start with the oldest set and some investigation should be done in existing set databases.

This suggests, if I interpret it correctly, that the "-1" qualifier is implied but never included in the filename, which I would guess is to avoid renaming in case a second set is released using what was previously a unique set number.

However, you've also pointed out that OMR compliance is a guideline rather than a firm regulation, so for a compliance-check tool it probably makes sense to allow a "-1" qualifier if present, but not to return an error if it's absent. (And since many existing models use the "-1", it probably also makes sense to align the spec to the status quo rather than the other way around.)

[Willy Tschager]

if I interpret it correctly, that the "-1" qualifier is implied but never included in the filename, which I would guess is to avoid renaming in case a second set is released using what was previously a unique set number.

If I remember correctly that was the intent.

w.

[Lasse Deleuran]

Alright. I have updated the OMR checks so that it allows both with "-1" and without.

Some checks have also been moved in an attempt to appear more logically positioned.

Finally, I have added an "OMR" option in the download menu, so that it is clear which file is OMR conform.

[Lasse Deleuran]

I keep adding checks to the list. The latest is coloring of tyres. They are often '0', but a more accurate color is '256' as it gives the proper appearance when rendering materials, such as:



Does anyone have a list of LEGO parts that were replaced, such as 2470 replacing 4459 in 1990? The more I can add of this type of checks, the better.

[Johann Eisner]

Alright. I have updated the OMR checks so that it allows both with "-1" and without.

Some checks have also been moved in an attempt to appear more logically positioned.

Finally, I have added an "OMR" option in the download menu, so that it is clear which file is OMR conform.

When I add a file to the OMR, I always remove the "-1" qualifier.

[Max Martin Richter]

The half bush (4265) was changed in from a-variant to b-variant in 1993 and from b-variant to the smooth reduced axle hole version in 1998 (312123a) and again to the semi-reduced axle hole version (32123b) in 2008.

BTW I have two set from 1990 still containing the 4459 pin. :-)

Some sets from 1992 I own, still have the 4273a-version, but the b-version was introduced in this year. (Change to smooth version (32126) was in 1998.)

Only sets produced in 1994 contained the 6542a clutch gear in black (I know at least on 8880 set and one 8858 set, where it is black). The 1993 set 8824 and all sets produced after 1994 had them in dark grey only. (6542b was introduced in 2011.)

In 1996 the axle joiner 6538a (introduced 1993) was changed to 6538b. In 2008 it was replaced by the smooth version.

In 1997 the axle 2L (3704) was replaced by the notched version (32062). Furthermore the 24T gear (3648a) was replaced by 3648b.

In 2001 the 3L Pin without friction (32556) was introduced.

In 2003 the pneumatic system changed the first time to studless design (new cylinder and new valve)

In 2008 the U-Joint 4L was replaced by the U-Joint 3L.

In 2009 the Pin Joiner (75535) was replaced by the slotted version (62462).

In 2011 the 16T Gear (4019) was replaced by the reinforced version (95925).

In 2012 the technic triangle (2905) was replaced by 99773.

In 2013 the 8T Gear (3647) was replaced by 10928.

In 2015 the pneumatic valve (and all continued pneumativ parts) where changed to stepped outlet.

I'm not sure, when the small wedge belt wheel was changed (4185a/b) - I assume late 80s or very early 90s. I'm actually not sure when the Pneumatic T-Piece was changed from straight to reinforced version - I assume it was in 1993? And I have no idea when the 24T crown gear was changed through its versions. The Reinforced first appeared in a set from 1990 in my collection, but there was a change from version a to b, which I don't know.)

[Lasse Deleuran]

The half bush (4265) was changed in from a-variant to b-variant in 1993 and from b-variant to the smooth reduced axle hole version in 1998 (312123a) and again to the semi-reduced axle hole version (32123b) in 2008.

BTW I have two set from 1990 still containing the 4459 pin. :-)

Some sets from 1992 I own, still have the 4273a-version, but the b-version was introduced in this year. (Change to smooth version (32126) was in 1998.)

Only sets produced in 1994 contained the 6542a clutch gear in black (I know at least on 8880 set and one 8858 set, where it is black). The 1993 set 8824 and all sets produced after 1994 had them in dark grey only. (6542b was introduced in 2011.)

In 1996 the axle joiner 6538a (introduced 1993) was changed to 6538b. In 2008 it was replaced by the smooth version.

In 1997 the axle 2L (3704) was replaced by the notched version (32062). Furthermore the 24T gear (3648a) was replaced by 3648b.

In 2001 the 3L Pin without friction (32556) was introduced.

In 2003 the pneumatic system changed the first time to studless design (new cylinder and new valve)

In 2008 the U-Joint 4L was replaced by the U-Joint 3L.

In 2009 the Pin Joiner (75535) was replaced by the slotted version (62462).

In 2011 the 16T Gear (4019) was replaced by the reinforced version (95925).

In 2012 the technic triangle (2905) was replaced by 99773.

In 2013 the 8T Gear (3647) was replaced by 10928.

In 2015 the pneumatic valve (and all continued pneumativ parts) where changed to stepped outlet.

I'm not sure, when the small wedge belt wheel was changed (4185a/b) - I assume late 80s or very early 90s. I'm actually not sure when the Pneumatic T-Piece was changed from straight to reinforced version - I assume it was in 1993? And I have no idea when the 24T crown gear was changed through its versions. The Reinforced first appeared in a set from 1990 in my collection, but there was a change from version a to b, which I don't know.)

Thanks. These are some great checks for OMR. I have introduced "phase in" as a concept in order to highlight the time where both versions of an element are in production and I have checked with Bricklink in order to ensure that the checks are not too strict. While I agree with your points (this is also how I remember sets), it seems that Bricklink has rather large periods of overlap. I have left these phase in periods lenient for now.

For anyone who would like to use the information, it is all hosted in this pli.js file and you can hotlink it without giving it a second thought

Bricklink doesn't distinguish some parts (such as crown gear type A and B) so I don't have proper timelines for those. I will add other parts, such as jumper plates, tiles, arches, etc. when I find time.