[LDCad] Rebrickable LDCad PBG export - Printable Version +- LDraw.org Discussion Forums (https://forums.ldraw.org) +-- Forum: LDraw Programs (https://forums.ldraw.org/forum-7.html) +--- Forum: LDraw Editors and Viewers (https://forums.ldraw.org/forum-11.html) +--- Thread: [LDCad] Rebrickable LDCad PBG export (/thread-24001.html) |
Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24 I love Rebrickable's PBG export. However, it has the drawback that minifig torsos and legs are listed as assemblies so unless they're generic enough to use the appropriate shortcut they'll always appear as unfound parts in the parts bin. I recently had a forum thread with the dev team about this and was basically told that this will never change since it's too much for them to track arms, hands, torsos, hips, and legs separately. I don't disagree with this decision from their point of view. I proposed the following solution: Associate the LDraw part number for the appropriate torso with the assembly. Add in the arms in the same color as the torso and the hands in yellow. This will at least get the correct torso pattern to show up in the parts bin and the if the arm or hand color is different, that's easily changeable by the user. They have yet to get back to me on this suggestion. Then I got to thinking, we could use the Rebrickable API to provide our own PBG export. Either roll our own cross reference database or, since Bricklink does catalog the torsos, arms, and hands separately and Rebrickable provides the Bricklink number, pull this data from Bricklink via their API. We could also have an option to include the unpatterned version of a part if the patterned version doesn't exist and reference the appropriate flex part template for flexible parts. Thoughts? RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24 I certainly wouldn't mind a PBG export tool that includes all component parts of an assembly (perhaps as an option), whether that's minifig parts, hinge assemblies and turntables, etc. While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24 (2020-04-24, 16:58)N. W. Perry Wrote: While we're at it, there are a couple of other quirks I wouldn't mind seeing fixed, like RB's insistence on round-top antennas and old-style jumpers long after they stopped being included in current sets, or using the alias number for cheese slopes, etc. These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy. RE: Rebrickable LDCad PBG export - Roland Melkert - 2020-04-24 (2020-04-24, 14:56)Orion Pobursky Wrote: Thoughts? I'm not sure if I can (reliably) find parts based on the information in the generated pbg's. Some seem to have extra characters or different numbers all together. But if I had some kind of grammar rules it could be done I guess. Could add it as a fallback option when a part can't be found. Having some sort of reference library would be the preferred way I think. Could optionally add it to the .dat's themselves trough keywords or a new meta e.g. "!ALIAS" Or in LDCad's shadow if people don't want to pollute the library with third party stuff. Then again one big (json?) file might be easier to generate/maintain. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24 (2020-04-24, 17:10)Roland Melkert Wrote: I'm not sure if I can (reliably) find parts based on the information in the generated pbg's. You don't need to do anything. I'm already 30% done with the proof of concept and I might be finished this weekend depending on how much time I get to devote to it. It's really just 2 API calls and some data cross referencing. RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24 (2020-04-24, 17:05)Orion Pobursky Wrote: These are a matter of correcting either the inventory or the part number at Rebrickable. I've probably submitted 30 change requests in the last couple of weeks to correct the LDraw part number associated with a part or to correct an inventory. The widespread availability of digital instructions makes this easy. That, or by using a different database—you mentioned pulling some data from BL instead, for example. But BL has its own quirks and inaccuracies (like using "undetermined" part types, which don't correlate to a real-world element), so yeah, the accuracy of the export can't exceed that of the underlying inventory database without applying some further intelligence, whether human or artificial. Incidentally, instructions can be misleading in some cases, and so can even the "official" renderings that go onto box artwork, etc. I've noticed cases where these sources will depict a yet-to-be-released mold variation of a part, whereas actual sets still ship with the older version. And in many cases, different variants will appear in different sets, sometimes even commingled, so there's no single "correct" inventory—which BL tries to account for with their Alternate Items section. Also, mold variations often affect the undersides of parts, which are usually not visible in instructions or box art. The most conclusive sources I've found are pictorial or video reviews of sets (but you have to be sure the reviewer has an actual factory set and didn't just part it out from RB or BL), combined with the handy timeline graphs on RB's part detail pages. Anyway, sorry for the long tangent. Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-24 (2020-04-24, 18:24)N. W. Perry Wrote: Per the topic at hand, my thought was that if we did have our own cross-reference DB, could it have certain checks built in, like redirecting alias or obsolete parts, just to avoid having to rely on RB to respond to change requests? I think the value here is in automating the export itself, not concerning ourselves with the accuracies of inventories. I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases. RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-24 (2020-04-24, 19:22)Orion Pobursky Wrote: I'm not really interested in reinventing the wheel. BL, Rebrickable, BrickOwl, even Peeron have extensive set inventory databases. Most of them have API that make querying them easy. What I like about Rebrickable, and why I want to support them, is that they also have a MOC inventory side. I'd rather the bulk of the cross referencing be done by others (e.g. Rebrickable) and "fill in the gaps" when those services are unable (or unwilling) to work with us. Basically, our database would be small and consist of edge cases. Yep, I'm right there with you. The only issues I would be thinking of for this are the little systematic quirks—basically, the automatic file clean-up I have to do now with each PBG I export. The cheese slopes thing is in that category: RB has them in their part listings correctly as 54200, but for some reason it always exports them with the alias number 63290 instead. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-27 Update on my progress: This project turned out to be way easier that I envisioned in my head. The bulk of the work is already done by the Rebrickable data. I've have successfully pulled data from Rebrickable (via their API) and converted it to a PBG file. This replicates the functionality already at Rebrickable. I want to do 3 things initally: a. Include (by option) the appropriate templates for flex objects instead of the (unflexed) library versions. b. Include (by option) the unpatterned version of a patterned part if the pattern isn't in the library. c. Include the individual parts of patterned assemblies (i.e. torsos/arms/hards and hips/legs). d. Expanding on c. Maybe break apart assemblies into component parts? e. Fix MovedTo and Alias to point to the right parts. b) Was easy and is implemented a and c) Will required a cross reference file which will have to built manually (although some of it could be automated with a careful reading of the library) d) We'd have to pick and choose since this isn't appropriate for some (if not most) assemblies. Again, something that can go into the cross reference file. e) Should be easy but isn't preferred. I think the right answer is fix the issue but also throw a warning to submit a change request to the Rebrickable. The bulk of the work now is to finalize the structure of and add the data to the cross reference file. I should have a beta test version out by the end of the week. Note: I'm not going to correct individual patterned parts that are referenced incorrectly in Rebrickable. The correct answer is to submit a change request. Same for incorrect part numbers or part number mismatch. Note to the note: Patterned parts that reference a Bricklink number can be automatically correlated to a Rebrickable number if Rebrickable has a Bricklink listing in it's data. In this case we can probably do the same as e) and correct the reference and also throw a warning to submit a change request. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-29 I just finished doing a full comparative scan of the library vs. RB's database. It took quite a bit of time since I had to wait 3 sec between requests to prevent triggering RB's automatic throttling. I broke the results down into 3 categories:
There's a 4th category that I'll add when I format the data and that's parts with a Bricklink number in the keywords. This 4th category will most likely overlap with the Found/Not Found categories above since I ran this as a separate scan. The results can be found here: https://docs.google.com/spreadsheets/d/1i1tU6rGaUpHeS_AbS50LBvWJ3ekztAm5pjx7uD3PfXw/edit?usp=sharing I've asked the RB team about the preferred method to submit a large amount change requests. RE: Rebrickable LDCad PBG export - Philippe Hurbain - 2020-04-29 (2020-04-29, 16:23)Orion Pobursky Wrote: or alias partWhat's supposed to be the problem with alias part? Shouldn't we create aliases to match design number provided by LEGO for a given set? (for what it is worth, as LEGO has a tendancy to "rewrite history" depending on part availability) RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-29 (2020-04-29, 18:16)Philippe Hurbain Wrote: What's supposed to be the problem with alias part? Shouldn't we create aliases to match design number provided by LEGO for a given set? (for what it is worth, as LEGO has a tendancy to "rewrite history" depending on part availability) Most because they sort wrong in the parts bin but also some have complain about their usage (cough, Willy, cough, cough). RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-30 Here's the alpha version. Still lots of work to do. https://www.ldraw.org/files/pbg/ RE: Rebrickable LDCad PBG export - Philippe Hurbain - 2020-04-30 (2020-04-30, 5:04)Orion Pobursky Wrote: Here's the alpha version. Still lots of work to do.Tested on 45678 Spike set, works rather well! - desciption -> description (I guess that description will be updated to include set name?) - Would be nice to append -1 to set number if no -x is entered. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-30 (2020-04-30, 7:39)Philippe Hurbain Wrote: Tested on 45678 Spike set, works rather well! Thanks for testing. I'll implement those corrections/suggestions. Now that it works as I intend it to, it's a matter of code polish and moving it to the main cms. RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-30 (2020-04-30, 5:04)Orion Pobursky Wrote: Here's the alpha version. Still lots of work to do. Clicking "submit" appears to have no effect… (Tried a few browsers, all the same result.) RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-30 (2020-04-30, 17:27)N. W. Perry Wrote: Clicking "submit" appears to have no effect… (Tried a few browsers, all the same result.) Do you have javascript turned off? Also, there's a delay between submit and display for the code to download and process the inventory. If the set is large, there may be a substantial delay. Eventually I'll put in a visual indicator for this. RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-30 (2020-04-30, 17:32)Orion Pobursky Wrote: Do you have javascript turned off? Javascript is on. Perhaps it's the delay—I'll leave the tab open for a while and see what happens. RE: Rebrickable LDCad PBG export - Magnus Forsberg - 2020-04-30 (2020-04-30, 18:43)N. W. Perry Wrote: Javascript is on. Perhaps it's the delay—I'll leave the tab open for a while and see what happens. Could it be that you have to add the suffix -1 to the set number? RE: Rebrickable LDCad PBG export - N. W. Perry - 2020-04-30 (2020-04-30, 20:04)Magnus Forsberg Wrote: Could it be that you have to add the suffix -1 to the set number? It could be…and it is. ;-) RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-04-30 The new alpha location is here: https://www.ldraw.org/hidden-content/pbg-test.html Note: The download button doesn't do anything (yet) RE: Rebrickable LDCad PBG export - Roland Melkert - 2020-04-30 (2020-04-30, 22:02)Orion Pobursky Wrote: The new alpha location is here: I tried 5580 and it 'hangs' (console xhr inspection shows it fails to fetch the model from rebrickable) It does work when using 5580-1 but it misses the hose templates (check is enabled) needed for the engine. Minor note: the pbg item sourceInv property is only really needed when using a template reference so you could filter those out to save bandwidth RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-01 (2020-04-30, 23:53)Roland Melkert Wrote: I tried 5580 and it 'hangs' (console xhr inspection shows it fails to fetch the model from rebrickable) The fail code isn't very robust. Right now there are cases where the php exits but doesn't send any error message. Also, as noted above, the "-1" is required for now. Templates are part of the cross ref (unless I change my mind) and that file has exactly one test entry. So, odds are, if the template box is checked nothing will happen. RE: Rebrickable LDCad PBG export - Stefan Frenz - 2020-05-01 (2020-04-30, 22:02)Orion Pobursky Wrote: The new alpha location is here:This is awesome! Indeed the Rebrickable-search in LDInspector was meant in the very same way, but I discovered the same drawbacks listed in your early post and I couldn't generalize the template algorithm... So may I ask: is it open source? RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-01 (2020-04-30, 22:02)Orion Pobursky Wrote: The new alpha location is here: - "-1" is automatically appended if not found and a message generated - The download button works now - If a part isn't found, the messages will provide you a link to the part on Rebrickable. Still to be done: - Polish up the UI - Look up the set name - Implement template replacement - Populate the cross ref file RE: Rebrickable LDCad PBG export - Stefan Frenz - 2020-05-01 (2020-05-01, 16:38)Orion Pobursky Wrote: - If a part isn't found, the messages will provide you a link to the part on Rebrickable.Some time ago, Philippe helped me searching minifig parts and mentioned the description. This way I found many correct LDraw numbers as replacement for Rebrickable's PBG-parts and this is what is done in LDInspector when on the web pane "S" is pressed for a part (ranking the results is not always accurate, but sometimes I treat the results being not too bad). Maybe description-lookup could be useful to have a replacement guess? I would love to have a web based Rebrickable and Bricklink part list that is comparable before the export (which I tried in LDInspector but what I am not yet happy with). I would like to offer help in coding, testing, whatever - so please let me know if I may support. RE: Rebrickable LDCad PBG export - Merlijn Wissink - 2020-05-04 (2020-05-01, 16:38)Orion Pobursky Wrote: - "-1" is automatically appended if not found and a message generated I don't know if all of it uses the API, but if you're running into throttling issues, you could always use (for parts of the application) the offline download files: https://rebrickable.com/downloads/ Only drawback is you need to keep them updated. RE: Rebrickable LDCad PBG export - Stefan Frenz - 2020-05-18 Today I tried with set 70618 (Ninjago Destiny's Bounty) and do like the result very much - I think, the corresponding functions at Rebrickable and LDInspector may be obsolete. Two suggestions: Code: 14395.dat is unofficial. Ensure your unofficial parts folder is up to date The not found part listed as "34888" with link points to Rebrickable part 3068bpr0344 - there Rebrickable says "LDraw 34888" and "Printed version of 3068b". So maybe 3068b can be listed instead of not found part? Activating "Use unpatterned version of unfound patterned" does not change the behavior. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-18 (2020-05-18, 13:00)Stefan Frenz Wrote: Today I tried with set 70618 (Ninjago Destiny's Bounty) and do like the result very much - I think, the corresponding functions at Rebrickable and LDInspector may be obsolete. That's a great compliment considering I'm not even half done. (2020-05-18, 13:00)Stefan Frenz Wrote: Two suggestions: The formatting (newline vs. none) this is definitely the "polish" I was referring to. As far as a link in the unoffical warning, should it point to Rebrickable or the PT (I think the PT). For 34888, the script is "dumb" right now in that it looks for the part in the official and unofficial library and then stops there if the number isn't found and the part doesn't have a pattern code. I can have it look at the Rebrickable number to see if that has a pattern code. That's a fairly easy change. (2020-05-18, 13:00)Stefan Frenz Wrote: Activating "Use unpatterned version of unfound patterned" does not change the behavior. I actually just noticed this myself when testing last night. Additionally, it won't catch all of the patterns as the check I use assumes a part name format that isn't necessarily true (this more true for older sets you'll have to be pulling a fairly old set to encounter it). RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-18 (2020-05-01, 19:22)Stefan Frenz Wrote: Some time ago, Philippe helped me searching minifig parts and mentioned the description. This way I found many correct LDraw numbers as replacement for Rebrickable's PBG-parts and this is what is done in LDInspector when on the web pane "S" is pressed for a part (ranking the results is not always accurate, but sometimes I treat the results being not too bad). Maybe description-lookup could be useful to have a replacement guess? Thanks for the offer. I think this is a topic for a separate thread but here's what I want help on: the format for the cross ref file. I keep changing my mind on how I want to structure it. I think some input from other developer will help me finalize things. That said, I strongly prefer it be JSON (or XML if you twist my arm). As far as the actual cross ref data, I've submitted over 400 pattern corrections to Rebrickable. I'm finished through p2<x> torso assembly cross ref via spread sheet. Minifig, minidoll, and Technic action figure cross ref remain my focus for assemblies (since these won't get LDraw equivalents). You can see my efforts here: https://docs.google.com/spreadsheets/d/1i1tU6rGaUpHeS_AbS50LBvWJ3ekztAm5pjx7uD3PfXw/edit?usp=sharing RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-18 I found another bug. Currently, if an LDraw number is listed for a part, the script takes that number unquestioningly. This results in the inclusion of a lot of alias parts which is the opposite of what I made this script to do. Instead it should check if the Rebrickable number is a valid LDraw part number and, if so, use that number. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-18 (2020-05-18, 16:45)Orion Pobursky Wrote: I found another bug. This is now fixed. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-18 New feature that I implemented while trying to fix some part number inconsistencies: The converter will now auto replace Moved To and Alias parts to the referred to part. Also, obsolete parts will generate a warning. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-19 (2020-05-01, 16:38)Orion Pobursky Wrote: - "-1" is automatically appended if not found and a message generated I did a bunch of updates: - Lots of polish done - Set name is now looked up and included - Part number choice is now more intelligent. - Automatically redirects Moved To and Alias parts Known problems: - Option checkboxes don't actually do anything - Template replacement not implemented - Cross ref file has zero entries so effectively doesn't do anything RE: Rebrickable LDCad PBG export - Jaco van der Molen - 2020-05-22 Oh, very nice! I tested with 75274, which gave very good results. The only thing missing was [color=#4183c4]2454b not found in the official library or on the Parts Tracker [/color] Which is brick 1x2x5 that is rather common? Links in this message don't point to a right page 65429, 65426, 30166 are unofficial. Ensure your unofficial parts folder is up to date [size=small]The last 3 messages gave an alternative to patterned parts, which is great! [/size] RE: Rebrickable LDCad PBG export - Jaco van der Molen - 2020-05-22 The next test with 75276 gave 2 errors. [color=#4183c4]28802, 2454b not found in the official library or on the Parts Tracker [/color] Of course the 1x2x5 brick again. And the 28802 = Bracket 1 x 2 - 1 x 4 [Rounded Corners at Bottom, Square Corners at Top] which we do have. And a bit nitpicking perhaps, but: [color=#4183c4]30166 are unofficial. Ensure your unofficial parts folder is up to date. If one unofficial brick is found, it should say "is unofficial". I think this is an easy one to fix? ;-)[/color] Maybe add a link to the latest unofficial zip? RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-22 (2020-05-22, 18:49)Jaco van der Molen Wrote: Oh, very nice! 2454b is a variant mold that we haven't modeled or made an alias I actually posted about it here: https://forums.ldraw.org/thread-23954.html (2020-05-22, 18:49)Jaco van der Molen Wrote: Links in this message don't point to a right page That's a bug. I'll have it fixed tonight. (2020-05-22, 18:49)Jaco van der Molen Wrote: [size=small]The last 3 messages gave an alternative to patterned parts, which is great! This and minifig assemblies is why I started this project. The non-pattern substitution isn't perfect (and probably never will be) but it's right the vast majority of the time. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-24 https://www.ldraw.org/hidden-content/pbg-test.html More updates: - I finally implemented the reason I wrote this generator: assembly replacement. Note that this is a manually created database. Currently it has about half of LDraw's minifg torsos (p codes 0 - a), control sticks (it was an easy test case), and unpattened hips and legs where the hips and leg colors are different (Rebrickable uses a pattern code scheme that allowed me to programmatically generate a list). This will expand out as I chug away at the list. The focus will be minifig/minidoll assemblies. If you have a category you want to see me implement sooner rather than later, let me know. The current, up-to-date list is located here: https://www.ldraw.org/common/data/pbg_crossref.csv - The de-pattern code is more robust - The de-pattern checkbox actually works and is set to replace by default - Fixed some other back end bugs. RE: Rebrickable LDCad PBG export - Vincent Messenet - 2020-05-25 (2020-05-24, 20:40)Orion Pobursky Wrote: https://www.ldraw.org/hidden-content/pbg-test.html Thanks Orion! I just did a test with set 10272-1 and part 2339 is reported as not being official while it is: https://www.ldraw.org/parts/official-part-lookup.html?folder=parts&partid=2339 Otherwise, 3 patterns are replaced by default parts because their Ldraw references are wrong at Rebrickable: 10202p01: https://rebrickable.com/parts/10202pr0007/tile-6-x-6-with-bottom-tubes-with-soccer-field-center-print/ 10202p00: https://rebrickable.com/parts/10202pr0006/tile-6-x-6-with-bottom-tubes-with-soccer-field-side-line-print/ 90498p00: https://rebrickable.com/parts/90498pr0004/tile-8-x-16-with-bottom-tubes-with-soccer-field-goal-area-print/ Did you already submit a change request for these 3 parts? RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-25 (2020-05-25, 5:21)Vincent Messenet Wrote: Thanks Orion! You found a bug! It's a pretty rare bug. Basically this set uses 14395 which is on the PT as an alias. The part is flagged by the script as unofficial. Then, since the part is an alias, the script changes the part to the base reference but never rechecks or unflags the part as unofficial. I'll have to rearrange the order of checks to prevent this. (2020-05-25, 5:21)Vincent Messenet Wrote: Otherwise, 3 patterns are replaced by default parts because their Ldraw references are wrong at Rebrickable: Those are all unofficial and I've been focusing on the official library. RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-25 (2020-05-25, 21:26)Orion Pobursky Wrote: You found a bug! It's a pretty rare bug. Basically this set uses 14395 which is on the PT as an alias. The part is flagged by the script as unofficial. Then, since the part is an alias, the script changes the part to the base reference but never rechecks or unflags the part as unofficial. I'll have to rearrange the order of checks to prevent this. This bug is now fixed. RE: Rebrickable LDCad PBG export - Vincent Messenet - 2020-05-26 (2020-05-25, 22:09)Orion Pobursky Wrote: This bug is now fixed.Thanks! RE: Rebrickable LDCad PBG export - Orion Pobursky - 2020-05-28 Now that I'm done with all 526(!) torso assembly entries (out of a possible 3700ish) in the cross reference file, I've moved the Set PBG generator out of experimental status. New url is: https://www.ldraw.org/downloads-2/pbg-generator.html and a new discussion thread started here: https://forums.ldraw.org/thread-24060.html |