[LDCad] Rebrickable LDCad PBG export

[Orion Pobursky]

I love Rebrickable's PBG export. However, it has the drawback that minifig torsos and legs are listed as assemblies so unless they're generic enough to use the appropriate shortcut they'll always appear as unfound parts in the parts bin. I recently had a forum thread with the dev team about this and was basically told that this will never change since it's too much for them to track arms, hands, torsos, hips, and legs separately. I don't disagree with this decision from their point of view.

I proposed the following solution:
Associate the LDraw part number for the appropriate torso with the assembly. Add in the arms in the same color as the torso and the hands in yellow. This will at least get the correct torso pattern to show up in the parts bin and the if the arm or hand color is different, that's easily changeable by the user. They have yet to get back to me on this suggestion.

Then I got to thinking, we could use the Rebrickable API to provide our own PBG export. Either roll our own cross reference database or, since Bricklink does catalog the torsos, arms, and hands separately and Rebrickable provides the Bricklink number, pull this data from Bricklink via their API. We could also have an option to include the unpatterned version of a part if the patterned version doesn't exist and reference the appropriate flex part template for flexible parts.

Thoughts?

[N. W. Perry]

I certainly wouldn't mind a PBG export tool that includes all component parts of an assembly (perhaps as an option), whether that's minifig parts, hinge assemblies and turntables, etc.

While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc.

[Orion Pobursky]

While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc.

These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy.

[Roland Melkert]

Thoughts?

I'm not sure if I can (reliably) find parts based on the information in the generated pbg's. Some seem to have extra characters or different numbers all together.

But if I had some kind of grammar rules it could be done I guess. Could add it as a fallback option when a part can't be found.

Having some sort of reference library would be the preferred way I think.

Could optionally add it to the .dat's themselves trough keywords or a new meta e.g. "!ALIAS"

Or in LDCad's shadow if people don't want to pollute the library with third party stuff.

Then again one big (json?) file might be easier to generate/maintain.

[Orion Pobursky]

I'm not sure if I can (reliably) find parts based on the information in the generated pbg's.

You don't need to do anything. I'm already 30% done with the proof of concept and I might be finished this weekend depending on how much time I get to devote to it. It's really just
2 API calls and some data cross referencing.

[N. W. Perry]

These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy.

That, or by using a different database—you mentioned pulling some data from BL instead, for example. But BL has its own quirks and inaccuracies (like using "undetermined" part types, which don't correlate to a real-world element), so yeah, the accuracy of the export can't exceed that of the underlying inventory database without applying some further intelligence, whether human or artificial.

Incidentally, instructions can be misleading in some cases, and so can even the "official" renderings that go onto box artwork, etc. I've noticed cases where these sources will depict a yet-to-be-released mold variation of a part, whereas actual sets still ship with the older version. And in many cases, different variants will appear in different sets, sometimes even commingled, so there's no single "correct" inventory—which BL tries to account for with their Alternate Items section.

Also, mold variations often affect the undersides of parts, which are usually not visible in instructions or box art. The most conclusive sources I've found are pictorial or video reviews of sets (but you have to be sure the reviewer has an actual factory set and didn't just part it out from RB or BL), combined with the handy timeline graphs on RB's part detail pages.

Anyway, sorry for the long tangent.  Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories.

[Orion Pobursky]

Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories.

I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases.

[N. W. Perry]

I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases.

Yep, I'm right there with you. The only issues I would be thinking of for this are the little systematic quirks—basically, the automatic file clean-up I have to do now with each PBG I export. The cheese slopes thing is in that category: RB has them in their part listings correctly as 54200, but for some reason it always exports them with the alias number 63290 instead.

[Orion Pobursky]

Update on my progress:

This project turned out to be way easier that I envisioned in my head. The bulk of the work is already done by the Rebrickable data. I've have successfully pulled data from Rebrickable (via their API) and converted it to a PBG file. This replicates the functionality already at Rebrickable.

I want to do 3 things initally:
a. Include (by option) the appropriate templates for flex objects instead of the (unflexed) library versions.
b. Include (by option) the unpatterned version of a patterned part if the pattern isn't in the library.
c. Include the individual parts of patterned assemblies (i.e. torsos/arms/hards and hips/legs).
d. Expanding on c. Maybe break apart assemblies into component parts?
e. Fix MovedTo and Alias to point to the right parts.

b) Was easy and is implemented
a and c) Will required a cross reference file which will have to built manually (although some of it could be automated with a careful reading of the library)
d) We'd have to pick and choose since this isn't appropriate for some (if not most) assemblies. Again, something that can go into the cross reference file.
e) Should be easy but isn't preferred. I think the right answer is fix the issue but also throw a warning to submit a change request to the Rebrickable.

The bulk of the work now is to finalize the structure of and add the data to the cross reference file. I should have a beta test version out by the end of the week.

Note: I'm not going to correct individual patterned parts that are referenced incorrectly in Rebrickable. The correct answer is to submit a change request. Same for incorrect part numbers or part number mismatch.

Note to the note: Patterned parts that reference a Bricklink number can be automatically correlated to a Rebrickable number if Rebrickable has a Bricklink listing in it's data. In this case we can probably do the same as e) and correct the reference and also throw a warning to submit a change request.

[Orion Pobursky]

I just finished doing a full comparative scan of the library vs. RB's database. It took quite a bit of time since I had to wait 3 sec between requests to prevent triggering RB's automatic throttling. I broke the results down into 3 categories:

• Match, this is where  LDraw number was found in the Rebrickable database either directly (e.g. 3001 in both) or via RB's cross reference data (e.g. most patterned parts)
  
• Possible error, this is a match that refers to a part whose name begins with ~,=,_,| meaning the RB database is possibly pointing to a moved, obsolete, or alias part and a change request may need to be submitted
  
• Not found, the LDraw number does not exist in the RB database either directly or via internal cross reference
  

There's a 4th category that I'll add when I format the data and that's parts with a Bricklink number in the keywords. This 4th category will most likely overlap with the Found/Not Found categories above since I ran this as a separate scan.

The results can be found here:
https://docs.google.com/spreadsheets/d/1...sp=sharing

I've asked the RB team about the preferred method to submit a large amount change requests.

[Philippe Hurbain]

or alias part

What's supposed to be the problem with alias part? Shouldn't we create aliases to match design number provided by LEGO for a given set? (for what it is worth, as LEGO has a tendancy to "rewrite history" depending on part availability)

[Orion Pobursky]

What's supposed to be the problem with alias part? Shouldn't we create aliases to match design number provided by LEGO for a given set? (for what it is worth, as LEGO has a tendancy to "rewrite history" depending on part availability)

Most because they sort wrong in the parts bin but also some have complain about their usage (cough, Willy, cough, cough).

[Orion Pobursky]

Here's the alpha version. Still lots of work to do.

https://www.ldraw.org/files/pbg/

[Philippe Hurbain]

Here's the alpha version. Still lots of work to do.

https://www.ldraw.org/files/pbg/

Tested on 45678 Spike set, works rather well!
- desciption -> description (I guess that description will be updated to include set name?)
- Would be nice to append -1 to set number if no -x is entered.

[Orion Pobursky]

Tested on 45678 Spike set, works rather well!
- desciption -> description (I guess that description will be updated to include set name?)
- Would be nice to append -1 to set number if no -x is entered.

Thanks for testing. I'll implement those corrections/suggestions.

Now that it works as I intend it to, it's a matter of code polish and moving it to the main cms.

[N. W. Perry]

Here's the alpha version. Still lots of work to do.

https://www.ldraw.org/files/pbg/

Clicking "submit" appears to have no effect…    (Tried a few browsers, all the same result.)

[Orion Pobursky]

Clicking "submit" appears to have no effect…    (Tried a few browsers, all the same result.)

Do you have javascript turned off?

Also, there's a delay between submit and display for the code to download and process the inventory. If the set is large, there may be a substantial delay. Eventually I'll put in a visual indicator for this.

[N. W. Perry]

Do you have javascript turned off?

Also, there's a delay between submit and display for the code to download and process the inventory. If the set is large, there may be a substantial delay. Eventually I'll put in a visual indicator for this.

Javascript is on. Perhaps it's the delay—I'll leave the tab open for a while and see what happens.

[Magnus Forsberg]

Javascript is on. Perhaps it's the delay—I'll leave the tab open for a while and see what happens.

Could it be that you have to add the suffix -1 to the set number?

[N. W. Perry]

Could it be that you have to add the suffix -1 to the set number?

It could be…and it is. ;-)

[Orion Pobursky]

The new alpha location is here:
https://www.ldraw.org/hidden-content/pbg-test.html

Note: The download button doesn't do anything (yet)

[Roland Melkert]

The new alpha location is here:

I tried 5580 and it 'hangs' (console xhr inspection shows it fails to fetch the model from rebrickable)

It does work when using 5580-1 but it misses the hose templates (check is enabled) needed for the engine.

Minor note:
the pbg item sourceInv property is only really needed when using a template reference so you could filter those out to save bandwidth

[Orion Pobursky]

I tried 5580 and it 'hangs' (console xhr inspection shows it fails to fetch the model from rebrickable)

It does work when using 5580-1 but it misses the hose templates (check is enabled) needed for the engine.

Minor note:
the pbg item sourceInv property is only really needed when using a template reference so you could filter those out to save bandwidth

The fail code isn't very robust. Right now there are cases where the php exits but doesn't send any error message.

Also, as noted above, the "-1" is required for now.

Templates are part of the cross ref (unless I change my mind) and that file has exactly one test entry. So, odds are, if the template box is checked nothing will happen.

[Stefan Frenz]

The new alpha location is here:
https://www.ldraw.org/hidden-content/pbg-test.html

Note: The download button doesn't do anything (yet)

This is awesome!   Indeed the Rebrickable-search in LDInspector was meant in the very same way, but I discovered the same drawbacks listed in your early post and I couldn't generalize the template algorithm... So may I ask: is it open source?

[Orion Pobursky]

The new alpha location is here:
https://www.ldraw.org/hidden-content/pbg-test.html

Note: The download button doesn't do anything (yet)

- "-1" is automatically appended if not found and a message generated
- The download button works now
- If a part isn't found, the messages will provide you a link to the part on Rebrickable.

Still to be done:
 - Polish up the UI
 - Look up the set name
 - Implement template replacement
 - Populate the cross ref file

[Stefan Frenz]

- If a part isn't found, the messages will provide you a link to the part on Rebrickable.

Some time ago, Philippe helped me searching minifig parts and mentioned the description. This way I found many correct LDraw numbers as replacement for Rebrickable's PBG-parts and this is what is done in LDInspector when on the web pane "S" is pressed for a part (ranking the results is not always accurate, but sometimes I treat the results being not too bad). Maybe description-lookup could be useful to have a replacement guess?

I would love to have a web based Rebrickable and Bricklink part list that is comparable before the export (which I tried in LDInspector but what I am not yet happy with). I would like to offer help in coding, testing, whatever - so please let me know if I may support.

[Merlijn Wissink]

- "-1" is automatically appended if not found and a message generated
- The download button works now
- If a part isn't found, the messages will provide you a link to the part on Rebrickable.

Still to be done:
 - Polish up the UI
 - Look up the set name
 - Implement template replacement
 - Populate the cross ref file

I don't know if all of it uses the API, but if you're running into throttling issues, you could always use (for parts of the application) the offline download files: https://rebrickable.com/downloads/

Only drawback is you need to keep them updated.

[Stefan Frenz]

Today I tried with set 70618 (Ninjago Destiny's Bounty) and do like the result very much - I think, the corresponding functions at Rebrickable and LDInspector may be obsolete.

Two suggestions:

Code:

14395.dat is unofficial. Ensure your unofficial parts folder is up to date
15967
not found in the official library or on the Parts Tracker
...
34888
not found in the official library or on the Parts Tracker

Perhaps for "Polish up the UI": the 15967 has a link (which I like, but 14395.dat has not), and it has a new line (which 14395.dat has not, but I like).

The not found part listed as "34888" with link points to Rebrickable part 3068bpr0344 - there Rebrickable says "LDraw 34888" and "Printed version of 3068b". So maybe 3068b can be listed instead of not found part? Activating "Use unpatterned version of unfound patterned" does not change the behavior.

[Orion Pobursky]

Today I tried with set 70618 (Ninjago Destiny's Bounty) and do like the result very much - I think, the corresponding functions at Rebrickable and LDInspector may be obsolete.

That's a great compliment considering I'm not even half done.

Two suggestions:

Code:

14395.dat is unofficial. Ensure your unofficial parts folder is up to date
15967
not found in the official library or on the Parts Tracker
...
34888
not found in the official library or on the Parts Tracker

Perhaps for "Polish up the UI": the 15967 has a link (which I like, but 14395.dat has not), and it has a new line (which 14395.dat has not, but I like).

The not found part listed as "34888" with link points to Rebrickable part 3068bpr0344 - there Rebrickable says "LDraw 34888" and "Printed version of 3068b". So maybe 3068b can be listed instead of not found part?

The formatting (newline vs. none) this is definitely the "polish" I was referring to. As far as a link in the unoffical warning, should it point to Rebrickable or the PT (I think the PT).

For 34888, the script is "dumb" right now in that it looks for the part in the official and unofficial library and then stops there if the number isn't found and the part doesn't have a pattern code. I can have it look at the Rebrickable number to see if that has a pattern code. That's a fairly easy change.

Activating "Use unpatterned version of unfound patterned" does not change the behavior.

I actually just noticed this myself when testing last night. Additionally, it won't catch all of the patterns as the check I use assumes a part name format that isn't necessarily true (this more true for older sets you'll have to be pulling a fairly old set to encounter it).

[Orion Pobursky]

Some time ago, Philippe helped me searching minifig parts and mentioned the description. This way I found many correct LDraw numbers as replacement for Rebrickable's PBG-parts and this is what is done in LDInspector when on the web pane "S" is pressed for a part (ranking the results is not always accurate, but sometimes I treat the results being not too bad). Maybe description-lookup could be useful to have a replacement guess?

I would love to have a web based Rebrickable and Bricklink part list that is comparable before the export (which I tried in LDInspector but what I am not yet happy with). I would like to offer help in coding, testing, whatever - so please let me know if I may support.

Thanks for the offer.

I think this is a topic for a separate thread but here's what I want help on: the format for the cross ref file. I keep changing my mind on how I want to structure it. I think some input from other developer will help me finalize things. That said, I strongly prefer it be JSON (or XML if you twist my arm).

As far as the actual cross ref data, I've submitted over 400 pattern corrections to Rebrickable. I'm finished through p2<x> torso assembly cross ref via spread sheet. Minifig, minidoll, and Technic action figure cross ref remain my focus for assemblies (since these won't get LDraw equivalents). You can see my efforts here:
https://docs.google.com/spreadsheets/d/1...sp=sharing

[Orion Pobursky]

I found another bug.

Currently, if an LDraw number is listed for a part, the script takes that number unquestioningly. This results in the inclusion of a lot of alias parts which is the opposite of what I made this script to do. Instead it should check if the Rebrickable number is a valid LDraw part number and, if so, use that number.

[Orion Pobursky]

I found another bug.

Currently, if an LDraw number is listed for a part, the script takes that number unquestioningly. This results in the inclusion of a lot of alias parts which is the opposite of what I made this script to do. Instead it should check if the Rebrickable number is a valid LDraw part number and, if so, use that number.

This is now fixed.

[Orion Pobursky]

New feature that I implemented while trying to fix some part number inconsistencies:
The converter will now auto replace Moved To and Alias parts to the referred to part. Also, obsolete parts will generate a warning.

[Orion Pobursky]

- "-1" is automatically appended if not found and a message generated
- The download button works now
- If a part isn't found, the messages will provide you a link to the part on Rebrickable.

Still to be done:
 - Polish up the UI
 - Look up the set name
 - Implement template replacement
 - Populate the cross ref file

I did a bunch of updates:
- Lots of polish done
- Set name is now looked up and included
- Part number choice is now more intelligent.
- Automatically redirects Moved To and Alias parts

Known problems:
- Option checkboxes don't actually do anything
- Template replacement not implemented
- Cross ref file has zero entries so effectively doesn't do anything

[Jaco van der Molen]

Oh, very nice!

I tested with 75274, which gave very good results.
The only thing missing was 
[color=#4183c4]2454b not found in the official library or on the Parts Tracker
[/color]
Which is brick 1x2x5 that is rather common?

Links in this message don't point to a right page
65429, 65426, 30166 are unofficial. Ensure your unofficial parts folder is up to date

[size=small]The last 3 messages gave an alternative to patterned parts, which is great!

[/size]

[Jaco van der Molen]

The next test with 75276 gave 2 errors.
[color=#4183c4]28802, 2454b not found in the official library or on the Parts Tracker
[/color]
Of course the 1x2x5 brick again.

And the 28802 = Bracket 1 x 2 - 1 x 4 [Rounded Corners at Bottom, Square Corners at Top] which we do have.


And a bit nitpicking perhaps, but:

[color=#4183c4]30166 are unofficial. Ensure your unofficial parts folder is up to date.

If one unofficial brick is found, it should say "is unofficial".
I think this is an easy one to fix? ;-)[/color]

Maybe add a link to the latest unofficial zip?

[Orion Pobursky]

Oh, very nice!

I tested with 75274, which gave very good results.
The only thing missing was 
[color=#4183c4]2454b not found in the official library or on the Parts Tracker
[/color]
Which is brick 1x2x5 that is rather common?

2454b is a variant mold that we haven't modeled or made an alias
I actually posted about it here:
https://forums.ldraw.org/thread-23954.html

Links in this message don't point to a right page
65429, 65426, 30166 are unofficial. Ensure your unofficial parts folder is up to date

That's a bug. I'll have it fixed tonight.

[size=small]The last 3 messages gave an alternative to patterned parts, which is great!

[/size]

This and minifig assemblies is why I started this project. The non-pattern substitution isn't perfect (and probably never will be) but it's right the vast majority of the time.

[Orion Pobursky]

https://www.ldraw.org/hidden-content/pbg-test.html

More updates:
- I finally implemented the reason I wrote this generator: assembly replacement. Note that this is a manually created database. Currently it has about half of LDraw's minifg torsos (p codes 0 - a), control sticks (it was an easy test case), and unpattened hips and legs where the hips and leg colors are different (Rebrickable uses a pattern code scheme that allowed me to programmatically generate a list). This will expand out as I chug away at the list. The focus will be minifig/minidoll assemblies. If you have a category you want to see me implement sooner rather than later, let me know. The current, up-to-date list is located here:
https://www.ldraw.org/common/data/pbg_crossref.csv
- The de-pattern code is more robust
- The de-pattern checkbox actually works and is set to replace by default
- Fixed some other back end bugs.

[Vincent Messenet]

https://www.ldraw.org/hidden-content/pbg-test.html

More updates:
- I finally implemented the reason I wrote this generator: assembly replacement. Note that this is a manually created database. Currently it has about half of LDraw's minifg torsos (p codes 0 - a), control sticks (it was an easy test case), and unpattened hips and legs where the hips and leg colors are different (Rebrickable uses a pattern code scheme that allowed me to programmatically generate a list). This will expand out as I chug away at the list. The focus will be minifig/minidoll assemblies. If you have a category you want to see me implement sooner rather than later, let me know. The current, up-to-date list is located here:
https://www.ldraw.org/common/data/pbg_crossref.csv
- The de-pattern code is more robust
- The de-pattern checkbox actually works and is set to replace by default
- Fixed some other back end bugs.

Thanks Orion!

I just did a test with set 10272-1 and part 2339 is reported as not being official while it is: https://www.ldraw.org/parts/official-part-lookup.html?folder=parts&partid=2339

Otherwise, 3 patterns are replaced by default parts because their Ldraw references are wrong at Rebrickable:
10202p01: https://rebrickable.com/parts/10202pr000...ter-print/
10202p00: https://rebrickable.com/parts/10202pr000...ine-print/
90498p00: https://rebrickable.com/parts/90498pr000...rea-print/

Did you already submit a change request for these 3 parts?

[Orion Pobursky]

Thanks Orion!

I just did a test with set 10272-1 and part 2339 is reported as not being official while it is: https://www.ldraw.org/parts/official-part-lookup.html?folder=parts&partid=2339

You found a bug! It's a pretty rare bug. Basically this set uses 14395 which is on the PT as an alias. The part is flagged by the script as unofficial. Then, since the part is an alias, the script changes the part to the base reference but never rechecks or unflags the part as unofficial. I'll have to rearrange the order of checks to prevent this.

Otherwise, 3 patterns are replaced by default parts because their Ldraw references are wrong at Rebrickable:
10202p01: https://rebrickable.com/parts/10202pr000...ter-print/
10202p00: https://rebrickable.com/parts/10202pr000...ine-print/
90498p00: https://rebrickable.com/parts/90498pr000...rea-print/

Those are all unofficial and I've been focusing on the official library.

[Orion Pobursky]

You found a bug! It's a pretty rare bug. Basically this set uses 14395 which is on the PT as an alias. The part is flagged by the script as unofficial. Then, since the part is an alias, the script changes the part to the base reference but never rechecks or unflags the part as unofficial. I'll have to rearrange the order of checks to prevent this.

This bug is now fixed.

[Vincent Messenet]

This bug is now fixed.

Thanks!

[Orion Pobursky]

Now that I'm done with all 526(!) torso assembly entries (out of a possible 3700ish) in the cross reference file, I've moved the Set PBG generator out of experimental status. New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

and a new discussion thread started here:
https://forums.ldraw.org/thread-24060.html