LDCad Set PBG generator

[Orion Pobursky]

I finished entering all 526(!) torso assemblies (out of a possible 3700ish) into the cross reference file, I've moved the Set PBG generator out of experimental status.

New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

My next project is hips assemblies then Minidoll torso and leg assemblies followed by Technic action figures. I may sprinkle in other assemblies as I find them.

As always, If you find bugs or oddities, please post them here. If you find a misnumbered single file that's not part of an assembly, submit a change request at Rebrickable.

The current crossref file can always be found here:
https://www.ldraw.org/common/data/pbg_crossref.csv

[N. W. Perry]

I finished entering all 526(!) torso assemblies (out of a possible 3700ish) into the cross reference file, I've moved the Set PBG generator out of experimental status.

New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

My next project is hips assemblies then Minidoll torso and leg assemblies followed by Technic action figures. I may sprinkle in other assemblies as I find them.

As always, If you find bugs or oddities, please post them here. If you find a misnumbered single file that's not part of an assembly, submit a change request at Rebrickable.

The current crossref file can always be found here:
https://www.ldraw.org/common/data/pbg_crossref.csv

As it happens, a certain level of privileges is needed to submit change requests to RB, which I don't personally have. That's fine with me, but I just thought I'd make note of it.

But the tool looks good and seems to work well, so cheers there!

[Orion Pobursky]

A certain level of privileges is needed to submit change requests to RB

It really takes very little effort to get to that point. I started because I want to submit a set list and it took me no time adding some of my sets and following MOC builders.

That said, if this doesn't interest you, just post your changes here and I'll submit them.

[Magnus Forsberg]

Do you know if they are scanning our library?
Most of the set I have looked at have bad info on the patterned parts. It looks like Rebrickable assume a ldraw pattern number based on their own numbering system. Why should we push lots of change request to them when they should be pulling info from us?
All of the printed parts to the Minion set 75551 are made, but non of them have correct number at RB.

Is there a way to get stickers and/or stickered parts included in this generator?
A set list is not complete without the sticker sheet numbers.

[Orion Pobursky]

Do you know if they are scanning our library?
Most of the set I have looked at have bad info on the patterned parts. It looks like Rebrickable assume a ldraw pattern number based on their own numbering system. Why should we push lots of change request to them when they should be pulling info from us?
All of the printed parts to the Minion set 75551 are made, but non of them have correct number at RB.

Yes, they are in a limited sense. Patterns are hard since they have to be manually matched. I've submitted somewhere in the realm of 400 entries to their database for just patterns.

Also, for the set you've mentioned, all of those parts are unofficial. For patterns I've been focusing on just the one's in the official library before moving on to the PT.

Is there a way to get stickers and/or stickered parts included in this generator?
A set list is not complete without the sticker sheet numbers.

Funny you mention stickers since I just started with cross referencing stickers (while I think about how I'm going to approach hip assemblies which aren't as straight forward as torsos were). Sticker sheets would be considered "assemblies". This means they'll have to be added to the crossref file.

[N. W. Perry]

It really takes very little effort to get to that point. I started because I want to submit a set list and it took me no time adding some of my sets and following MOC builders.

That said, if this doesn't interest you, just post your changes here and I'll submit them.

Oh yeah, I'm sure it'll just happen at some point without my even realizing it. 

[N. W. Perry]

Yes, they are in a limited sense. Patterns are hard since they have to be manually matched. I've submitted somewhere in the realm of 400 entries to their database for just patterns.

I was going to add, most of the errors I run into do seem to be wrong pattern codes. That suggests to me it might be more an issue of how they track these parts in their database, rather than requesting changes case-by-case.

[Orion Pobursky]

I was going to add, most of the errors I run into do seem to be wrong pattern codes. That suggests to me it might be more an issue of how they track these parts in their database, rather than requesting changes case-by-case.

I'm not doing case by case any more. I've been in contact with the database admins and I send them batch edit in the form of CSV files.

[Orion Pobursky]

Funny you mention stickers since I just started with cross referencing stickers (while I think about how I'm going to approach hip assemblies which aren't as straight forward as torsos were). Sticker sheets would be considered "assemblies". This means they'll have to be added to the crossref file.

Stickers are in the crossref. Not all the entries are correct because not all sticker sheets are in the Rebrickable data nor are they numbered consistently. Also, since Rebrickable doesn't assign a color to the sheets, I've assumed the color white for everything. This can be changed as I manually research each sheet.

[Orion Pobursky]

Stickers are in the crossref. Not all the entries are correct because not all sticker sheets are in the Rebrickable data nor are they numbered consistently. Also, since Rebrickable doesn't assign a color to the sheets, I've assumed the color white for everything. This can be changed as I manually research each sheet.

All the stickers in our library that are also cataloged at RB are now properly cross-ref'd with the proper color (White, Trans-Sticker, or (in one instance) Chrome Silver). I did my best so if you find a sticker group with the wrong color, let me know and I'll change it.

[Stefan Frenz]

Thank you so much for this exporter.  I really appreciate the new UI and UX.  

Testing set 70618 again, I only wonder at some parts that are reported to be "not found in the official library or on the Parts Tracker" like 25375 (tassel, LDraw already has the base parts in the official library but composed base parts are on the parts tracker) and 25111 (sword, Rebrickable seems to have a LDraw-rendered image of unknown source: the referenced LDraw number does not exist and I could not find any sword with my limited search abilities). Is this something at Rebrickable?

Thanks for the offer.

I think this is a topic for a separate thread but here's what I want help on: the format for the cross ref file. I keep changing my mind on how I want to structure it. I think some input from other developer will help me finalize things. That said, I strongly prefer it be JSON (or XML if you twist my arm).

I don't know if this is the separate thread.   But I like to sustain my offer to support in any way (I dislike XML, too).

[Orion Pobursky]

25375 (tassel, LDraw already has the base parts in the official library but composed base parts are on the parts tracker) and 25111 (sword, Rebrickable seems to have a LDraw-rendered image of unknown source: the referenced LDraw number does not exist and I could not find any sword with my limited search abilities). Is this something at Rebrickable?

I just submitted a change request for 25375 to point it to 25375-f1

The render for 25111 is probably the raw LDD data.

I don't know if this is the separate thread.   But I like to sustain my offer to support in any way (I dislike XML, too).

I decided to go with something much simpler: a CSV file. It's much easier to work with as I can do work in a spreadsheet and just export as CSV. The location to the file the generator uses is given above.

[Orion Pobursky]

Update on the crossref.

The hips and legs are currently undergoing revision on the PT. I'm going to hold off including them until they are official.

As for Technic action figures, The only parts that change are the heads and torsos. I was thinking of including the Complete shortcut (that can be inlined) and the appropriate head/torso as loose parts.  Thoughts?

[Philippe Hurbain]

As for Technic action figures, The only parts that change are the heads and torsos. I was thinking of including the Complete shortcut (that can be inlined) and the appropriate head/torso as loose parts.

Sorry, not sure to understand what you intend to do exactly?

[Stefan Frenz]

I was thinking of including the Complete shortcut (that can be inlined) and the appropriate head/torso as loose parts.  Thoughts?

Maybe for the non-technic-figures the assemblies could be done in the same way? I would love to have the figures "done already" in some way as I (personally) prefer building the sets - building/assembling the figures is not my favored action, so I would prefer "having" figures instead of "building" them.

[Orion Pobursky]

Maybe for the non-technic-figures the assemblies could be done in the same way? I would love to have the figures "done already" in some way as I (personally) prefer building the sets - building/assembling the figures is not my favored action, so I would prefer "having" figures instead of "building" them.

This would be awesome but is beyond the scope of this project. Although, an enterprising person could use the cross ref file and the new minifig feature of Rebrickable to do just this.

[Orion Pobursky]

Sorry, not sure to understand what you intend to do exactly?

So basically for torso assemblies, I replace the Rebrickable entry with the appropriately patterned torso, left and right arms, and 2 hands. The Technic action figure, however, is significantly more parts, most of which don't change. Therefore, I think the best solution is to replace the Rebrickable entry with Technic Action Figure shortcut (2698c01) in the dominant color, the appropriately patterned torso (2698pXX), and the appropriately patterned head (2707pXX). 2698c01 can then be inlined and the head and torso replaced with a double click.

[Philippe Hurbain]

Got it. Yes, I think it's a good solution...

[Marc Giraudet]

I finished entering all 526(!) torso assemblies (out of a possible 3700ish) into the cross reference file, I've moved the Set PBG generator out of experimental status.

New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

My next project is hips assemblies then Minidoll torso and leg assemblies followed by Technic action figures. I may sprinkle in other assemblies as I find them.

As always, If you find bugs or oddities, please post them here. If you find a misnumbered single file that's not part of an assembly, submit a change request at Rebrickable.

The current crossref file can always be found here:
https://www.ldraw.org/common/data/pbg_crossref.csv

Found a problem with 298c02 part :
For example set 76042-1 or 7236-1 have 2 of it but in the export there are only 1x4592 +1x4593.

When 298c02 is exported in 2 sub parts (i prefer this solution), why 2429c01 is not ?

[Jaco van der Molen]

I finished entering all 526(!) torso assemblies (out of a possible 3700ish) into the cross reference file, I've moved the Set PBG generator out of experimental status.

New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

My next project is hips assemblies then Minidoll torso and leg assemblies followed by Technic action figures. I may sprinkle in other assemblies as I find them.

As always, If you find bugs or oddities, please post them here. If you find a misnumbered single file that's not part of an assembly, submit a change request at Rebrickable.

The current crossref file can always be found here:
https://www.ldraw.org/common/data/pbg_crossref.csv

Great work Orion. This a a VERY useful tool!

[Orion Pobursky]

Found a problem with 298c02 part :
For example set 76042-1 or 7236-1 have 2 of it but in the export there are only 1x4592 +1x4593.

That is indeed a bug but not a serious one. The cross ref code doesn't take quantity into account but LDCad doesn't do anything with quantity at the moment so it shouldn't affect the ability to use the PBG effectively. I'll push a change shortly.

When 298c02 is exported in 2 sub parts (i prefer this solution), why 2429c01 is not ?

This is because the two parts of 2429c01 never come as different colors. Therefore it is trivial to add the part to the model and then inline it.

[Marc Giraudet]

That is indeed a bug but not a serious one. The cross ref code doesn't take quantity into account but LDCad doesn't do anything with quantity at the moment so it shouldn't affect the ability to use the PBG effectively. I'll push a change shortly.


This is because the two parts of 2429c01 never come as different colors. Therefore it is trivial to add the part to the model and then inline it.

I agree it's not a serious one.
About quantity in LDCad, I use 2 part bin to check inventory of official model after finishing. I found the issue when I check my latest set (76042).
Today I check one by one but maybe LDCad could proposed further an automatic check. Or a new part bin "dynamic" like a box, the number of part decreased when you put a part in the model (at the end, easier to found parts).

Thanks for the explaination of the colour.

[Orion Pobursky]

That is indeed a bug but not a serious one. The cross ref code doesn't take quantity into account but LDCad doesn't do anything with quantity at the moment so it shouldn't affect the ability to use the PBG effectively. I'll push a change shortly.

This bug is fixed.

[Orion Pobursky]

Added a new feature:
If a sticker sheet that isn't in the cross reference is detected, it is skipped and a warning issued.

[Marc Giraudet]

Added a new feature:
If a sticker sheet that isn't in the cross reference is detected, it is skipped and a warning issued.

LDCad Set PBG generator broken ?
Error : {"error":"Error: cross reference file not found","messages":null,"pbg":null}

[Orion Pobursky]

LDCad Set PBG generator broken ?
Error : {"error":"Error: cross reference file not found","messages":null,"pbg":null}

Fixed. I moved some code around and forgot to update the paths.

[Philippe Hurbain]

I tested the PBG generator for the Black Seas Barracuda... Got some glitches
- Some possible X-ref improvements:
970e02 -> 84637c01
2529pb01 -> 2528p30
2559c03 -> 2559c01
2557c03 -> 2557c01
3934a -> 3934?
3660 -> 3660a
42009 -> 42009a
973c11 -> torso+arms+hands? or 973c01 ?


More annoying:
4589 cone was replaced by stickers 004589
4315 hinge was replaced by stickers 004315

[Orion Pobursky]

I tested the PBG generator for the Black Seas Barracuda... Got some glitches
- Some possible X-ref improvements:
970e02 -> 84637c01
2529pb01 -> 2528p30
2559c03 -> 2559c01
2557c03 -> 2557c01
3934a -> 3934?
3660 -> 3660a
42009 -> 42009a
973c11 -> torso+arms+hands? or 973c01 ?


More annoying:
4589 cone was replaced by stickers 004589
4315 hinge was replaced by stickers 004315

Some of these are indeed problems with PBG gen. The rest are Rebrickable database errors which I can submit fixes. I'll have a more detail response once I have access to my desktop.

[Orion Pobursky]

Quick update:

970e02 -> 84637c01
2559c03 -> 2559c01
2557c03 -> 2557c01

Change requests have been submitted for these

3934a -> 3934?
3660 -> 3660a

These need to be researched and a "better" LDraw to Rebrickable map created. Then I can submit change requests.

42009 -> 42009a
2529pb01 -> 2528p30

I can't find either of these

973c11 -> torso+arms+hands? or 973c01 ?

I'll add this to the crossref

More annoying:
4589 cone was replaced by stickers 004589
4315 hinge was replaced by stickers 004315

Still investigating these 2

[Philippe Hurbain]

42009 -> 42009a
2529pb01 -> 2528p30

I can't find either of these

Sorry, typos... was 4209 -> 4209a
and 2528pb01 -> 2528p30
I submitted a change request for both

[Merlijn Wissink]

If I may ask; what is the advantage of this separate pbg exporter compared to the pbg exporter available on Rebrickable itself? It uses the same data, if I understand correctly?

[Orion Pobursky]

If I may ask; what is the advantage of this separate pbg exporter compared to the pbg exporter available on Rebrickable itself? It uses the same data, if I understand correctly?

The biggest advantage is the inclusion of assemblies (e.g. Torsos) that Rebrickable has declined to catalog as separate elements.

Also, it translate patterns that aren't yet included in the LDraw library to the plain versions.

And, when I finally get off my butt to program it, it'll include LDCad flexible elements as well.

[Orion Pobursky]

More annoying:
4589 cone was replaced by stickers 004589
4315 hinge was replaced by stickers 004315

I finally found this bug. Basically its that there's a sticker sheet with the number 004589 and a part with the number 4589. So it's a number conversion problem. It's gonna be a bit before it's fixed since I have to add a new field to the CSV file or improve part match logic (whichever proves to be more efficient)

[Philippe Hurbain]

I finally found this bug. Basically its that there's a sticker sheet with the number 004589 and a part with the number 4589. So it's a number conversion problem. It's gonna be a bit before it's fixed since I have to add a new field to the CSV file or improve part match logic (whichever proves to be more efficient)

A little request (I guess it should rather go to Roland actually...): in LDCad, I can't search for set description, only for set number. That improves if the pbg file name is extended to contain set description...

[Orion Pobursky]

That improves if the pbg file name is extended to contain set description...

This is trivial to implement. I'll have it done shortly.

[Orion Pobursky]

This is trivial to implement. I'll have it done shortly.

Done!

[Orion Pobursky]

I finally found this bug. Basically its that there's a sticker sheet with the number 004589 and a part with the number 4589. So it's a number conversion problem. It's gonna be a bit before it's fixed since I have to add a new field to the CSV file or improve part match logic (whichever proves to be more efficient)

I was even wrong about the source of the bug. It was a problem with the cross reference file itself. This problem is now fixed.

[Philippe Hurbain]

Got a small problem with PBG generator for 42121 set. Generated PBG uses 92694 for linear actuator, but this is actually the body only. AFAIK, Rebrickable inventory lists 92693, and this one correctly maps to 92693c01-f1.

[Orion Pobursky]

Got a small problem with PBG generator for 42121 set. Generated PBG uses 92694 for linear actuator, but this is actually the body only. AFAIK, Rebrickable inventory lists 92693, and this one correctly maps to 92693c01-f1.

Hmm. I don't know what's going on here. The response from Rebrickable doesn't even have 92694 in it and the crossref file isn't overriding anything. I'll have to dig into the code to find out whats up.

[Orion Pobursky]

Hmm. I don't know what's going on here. The response from Rebrickable doesn't even have 92694 in it and the crossref file isn't overriding anything. I'll have to dig into the code to find out whats up.

I found the issue. 92693 is MOVED TO 92694. This means that I need to retool my order of operations regarding redirect substitution. This will give me the opportunity to update and streamline the code.

[Orion Pobursky]

I found the issue. 92693 is MOVED TO 92694. This means that I need to retool my order of operations regarding redirect substitution. This will give me the opportunity to update and streamline the code.

This issue should now be fixed.

[Philippe Hurbain]

This issue should now be fixed.

Thanks Orion!

[Orion Pobursky]

I finished entering all 526(!) torso assemblies (out of a possible 3700ish) into the cross reference file, I've moved the Set PBG generator out of experimental status.

New url is:
https://www.ldraw.org/downloads-2/pbg-generator.html

My next project is hips assemblies then Minidoll torso and leg assemblies followed by Technic action figures. I may sprinkle in other assemblies as I find them.

As always, If you find bugs or oddities, please post them here. If you find a misnumbered single file that's not part of an assembly, submit a change request at Rebrickable.

The current crossref file can always be found here:
https://www.ldraw.org/common/data/pbg_crossref.csv

I'm working on improved sticker detection that will used formed shortcuts if available. The ultimate goal is to do away with the crossref file completely but that will depend on interfacing with Bricklink.

[Orion Pobursky]

I've made a few "under the hood" changes to the code to allow for better code reuse in other projects and to try to ensure that we don't exceed the request/sec limitation imposed by Rebrickable. If you notice anything odd, as always, report it here. I have a path forward for automatic sticker inclusion but have to think more on how to handle sticker shortcuts.

[Philippe Hurbain]

I'm working on improved sticker detection that will used formed shortcuts if available. The ultimate goal is to do away with the crossref file completely but that will depend on interfacing with Bricklink.

What's the best procedure for x-ref update? If that helps, I prepared a csv with the 4 torso assemblies of set 10278 that I recently patterned...

[Stefan Frenz]

I was interested in your file but always get an error "Authorization code mismatch. Are you accessing this function correctly". Did I something wrong or is the file protected against me?
   

[Philippe Hurbain]

I was interested in your file but always get an error "Authorization code mismatch. Are you accessing this function correctly". Did I something wrong or is the file protected against me?

Since I couldn't send directly a csv file, I renamed it .csv.txt so here it opens as a plain text (which it is actually  )

[Stefan Frenz]

Since I couldn't send directly a csv file, I renamed it .csv.txt so here it opens as a plain text (which it is actually  )

Thank you very much, now it works

[Orion Pobursky]

What's the best procedure for x-ref update? If that helps, I prepared a csv with the 4 torso assemblies of set 10278 that I recently patterned...

The best way is what you just did. I have a script that will pull info if there's enough crossref info in the keywords (part of why I started using Rebrickable numbers instead of Bricklink numbers) and then I just have to check arm and hand color. It's still a fairly manual process at the moment.

[N. W. Perry]

Just noticed this (and it's something of a judgment call): the "caption" and "description" tags are reversed from the intended usage—or at least from the usage in the example .pbg files.

The more verbose ("Parts in…") tag should go under description, which appears in the hint bar of the main editing window.

The less verbose tag should be the caption, which appears in the top of the part bin itself. It's meant to be as short as possible since the part bin window is typically not very wide.

Although this is made somewhat moot by the way the generator is set up, since it adds the set name, so you wouldn't want to reverse it completely. Still, I think the expected output should be something like this, to match the "official" example files:

Code:

caption=Set 21057-1
description=Parts in set 21057-1 - Singapore