LDraw.org Discussion Forums
[LDCad] Rebrickable LDCad PBG export - Printable Version

+- LDraw.org Discussion Forums (https://forums.ldraw.org)
+-- Forum: LDraw Programs (https://forums.ldraw.org/forum-7.html)
+--- Forum: LDraw Editors and Viewers (https://forums.ldraw.org/forum-11.html)
+--- Thread: [LDCad] Rebrickable LDCad PBG export (/thread-24001.html)

Pages: 1 2 3 4 5


Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24

I love Rebrickable's PBG export. However, it has the drawback that minifig torsos and legs are listed as assemblies so unless they're generic enough to use the appropriate shortcut they'll always appear as unfound parts in the parts bin. I recently had a forum thread with the dev team about this and was basically told that this will never change since it's too much for them to track arms, hands, torsos, hips, and legs separately. I don't disagree with this decision from their point of view.

I proposed the following solution:
Associate the LDraw part number for the appropriate torso with the assembly. Add in the arms in the same color as the torso and the hands in yellow. This will at least get the correct torso pattern to show up in the parts bin and the if the arm or hand color is different, that's easily changeable by the user. They have yet to get back to me on this suggestion.

Then I got to thinking, we could use the Rebrickable API to provide our own PBG export. Either roll our own cross reference database or, since Bricklink does catalog the torsos, arms, and hands separately and Rebrickable provides the Bricklink number, pull this data from Bricklink via their API. We could also have an option to include the unpatterned version of a part if the patterned version doesn't exist and reference the appropriate flex part template for flexible parts.

Thoughts?


RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24

I certainly wouldn't mind a PBG export tool that includes all component parts of an assembly (perhaps as an option), whether that's minifig parts, hinge assemblies and turntables, etc.

While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc.


RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24

(2020-04-24, 16:58)N. W. Perry Wrote: While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc.

These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy.


RE: Rebrickable LDCad PBG export - Roland Melkert - 2020-04-24

(2020-04-24, 14:56)Orion Pobursky Wrote: Thoughts?

I'm not sure if I can (reliably) find parts based on the information in the generated pbg's. Some seem to have extra characters or different numbers all together.

But if I had some kind of grammar rules it could be done I guess. Could add it as a fallback option when a part can't be found.

Having some sort of reference library would be the preferred way I think.

Could optionally add it to the .dat's themselves trough keywords or a new meta e.g. "!ALIAS"

Or in LDCad's shadow if people don't want to pollute the library with third party stuff.

Then again one big (json?) file might be easier to generate/maintain.


RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24

(2020-04-24, 17:10)Roland Melkert Wrote: I'm not sure if I can (reliably) find parts based on the information in the generated pbg's.

You don't need to do anything. I'm already 30% done with the proof of concept and I might be finished this weekend depending on how much time I get to devote to it. It's really just
2 API calls and some data cross referencing.


RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24

(2020-04-24, 17:05)Orion Pobursky Wrote: These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy.

That, or by using a different database—you mentioned pulling some data from BL instead, for example. But BL has its own quirks and inaccuracies (like using "undetermined" part types, which don't correlate to a real-world element), so yeah, the accuracy of the export can't exceed that of the underlying inventory database without applying some further intelligence, whether human or artificial.

Incidentally, instructions can be misleading in some cases, and so can even the "official" renderings that go onto box artwork, etc. I've noticed cases where these sources will depict a yet-to-be-released mold variation of a part, whereas actual sets still ship with the older version. And in many cases, different variants will appear in different sets, sometimes even commingled, so there's no single "correct" inventory—which BL tries to account for with their Alternate Items section.

Also, mold variations often affect the undersides of parts, which are usually not visible in instructions or box art. The most conclusive sources I've found are pictorial or video reviews of sets (but you have to be sure the reviewer has an actual factory set and didn't just part it out from RB or BL), combined with the handy timeline graphs on RB's part detail pages.

Anyway, sorry for the long tangent. Smile Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories.


RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24

(2020-04-24, 18:24)N. W. Perry Wrote: Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories.

I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases.


RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24

(2020-04-24, 19:22)Orion Pobursky Wrote: I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases.

Yep, I'm right there with you. The only issues I would be thinking of for this are the little systematic quirks—basically, the automatic file clean-up I have to do now with each PBG I export. The cheese slopes thing is in that category: RB has them in their part listings correctly as 54200, but for some reason it always exports them with the alias number 63290 instead.


RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-27

Update on my progress:

This project turned out to be way easier that I envisioned in my head. The bulk of the work is already done by the Rebrickable data. I've have successfully pulled data from Rebrickable (via their API) and converted it to a PBG file. This replicates the functionality already at Rebrickable.

I want to do 3 things initally:
a. Include (by option) the appropriate templates for flex objects instead of the (unflexed) library versions.
b. Include (by option) the unpatterned version of a patterned part if the pattern isn't in the library.
c. Include the individual parts of patterned assemblies (i.e. torsos/arms/hards and hips/legs).
d. Expanding on c. Maybe break apart assemblies into component parts?
e. Fix MovedTo and Alias to point to the right parts.

b) Was easy and is implemented
a and c) Will required a cross reference file which will have to built manually (although some of it could be automated with a careful reading of the library)
d) We'd have to pick and choose since this isn't appropriate for some (if not most) assemblies. Again, something that can go into the cross reference file.
e) Should be easy but isn't preferred. I think the right answer is fix the issue but also throw a warning to submit a change request to the Rebrickable.

The bulk of the work now is to finalize the structure of and add the data to the cross reference file. I should have a beta test version out by the end of the week.

Note: I'm not going to correct individual patterned parts that are referenced incorrectly in Rebrickable. The correct answer is to submit a change request. Same for incorrect part numbers or part number mismatch.

Note to the note: Patterned parts that reference a Bricklink number can be automatically correlated to a Rebrickable number if Rebrickable has a Bricklink listing in it's data. In this case we can probably do the same as e) and correct the reference and also throw a warning to submit a change request.


RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-29

I just finished doing a full comparative scan of the library vs. RB's database. It took quite a bit of time since I had to wait 3 sec between requests to prevent triggering RB's automatic throttling. I broke the results down into 3 categories:
  • Match, this is where  LDraw number was found in the Rebrickable database either directly (e.g. 3001 in both) or via RB's cross reference data (e.g. most patterned parts)
  • Possible error, this is a match that refers to a part whose name begins with ~,=,_,| meaning the RB database is possibly pointing to a moved, obsolete, or alias part and a change request may need to be submitted
  • Not found, the LDraw number does not exist in the RB database either directly or via internal cross reference

There's a 4th category that I'll add when I format the data and that's parts with a Bricklink number in the keywords. This 4th category will most likely overlap with the Found/Not Found categories above since I ran this as a separate scan.

The results can be found here:
https://docs.google.com/spreadsheets/d/1i1tU6rGaUpHeS_AbS50LBvWJ3ekztAm5pjx7uD3PfXw/edit?usp=sharing

I've asked the RB team about the preferred method to submit a large amount change requests.