LEGO 2 LDraw 2 BL

[Willy Tschager]

I'd like to discuss the argument opened here:

The LDD ldraw.xml file has this structure:

<!-- Arch  3 x  6 x  5  30613.dat -->
<!-- PAVILION 6X3X5  19063 -->
<Brick ldraw="30613.dat" lego="19063"/>
<!-- PAVILION 6X3X5  30613 -->
<Brick ldraw="30613.dat" lego="30613"/>

<!-- Animal Dolphin  6228b.dat -->
<!-- DOLPHIN 9M  6228 -->
<Brick ldraw="6228b.dat" lego="6228"/>

How does it look like in Studio?

How would an information like this treated in an editor? Would it show also the alternate number or just jump to the right part is an element number is entered say in a search?

w.

[Roland Melkert]

How would an information like this treated in an editor? Would it show also the alternate number or just jump to the right part is an element number is entered say in a search?

Could be as simple as XML:

Code:

<items>
 <item src="6228b" dst="6228"/>
</items>

or JSON:

Code:

[{src: '6228b', dst: '6228'}]

or even CSV

Code:

6228b,6228

An editor would use it as a translation table.

Any format could be generated if the 3rd party names would be stored in the parts tracker as metadata.

We could also introduce a new LDraw meta for this, and add it to the parts themselves. Eliminating the need for an index file altogether.

[Lutz Gehlen]

Could be as simple as XML:

Code:

<items>
 <item src="6228b" dst="6228"/>
</items>

or JSON:

Code:

[{src: '6228b', dst: '6228'}]

or even CSV

Code:

6228b,6228

An editor would use it as a translation table.

Any format could be generated if the 3rd party names would be stored in the parts tracker as metadata.

We could also introduce a new LDraw meta for this, and add it to the parts themselves. Eliminating the need for an index file altogether.

Hi all,

I think the two of you are talking about two different aspects. Roland is talking about how to store the information in files. One important aspect is whether to store the reference system of a part number explicitly. In the ldraw.xml file that Willy quoted the attribute names encode explicitly which is the lego number and which is the ldraw number. In my opinion this is preferable over src/dest. In the latter case you need meta information on the file in order to know how to use it. For example you need to know that this is an ldraw to lego translation file and not the other way around or ldraw to bricklink or whatever. In the former case, this information is stored in the content of the file itself. You could even have something like

Code:

<item ldraw="foo" lego="bar" bricklink="baz" />

in one go. Whether the file format is XML or JSON is not that important I think, but CSV might be too limited.

I think the last idea is interesting to store the information in the files themselves and eliminate the need for a translation file altogether. Roland, could you elaborate on that idea? Do you think of something like
0 ALIAS LEGO 6228
or what is the idea?

On the other hand, if I understand correctly, Willy is talking about how to expose this information (no matter how it is stored) to the user in an editor. I think from a model author's point of view there are several aspects. If I want to add a part to my model it would be most convenient if I can find it under any of its part numbers. Maybe I have found the part in a Lego model or on Bricklink or wherever and now I want to use it. On the other hand, in some situations I might want to use its number in a specific reference system. For example, I might want to look up on Bricklink how expensive the part is. Or I want to export a Bricklink XML file. It just happened to me that I exported such a file from LeoCAD and could not import it to Bricklink before I manually translated some of the part numbers. Ideally, this is to be avoided by an editor if such translation information is available.

Cheers,
Lutz

[David Manley]

We could also introduce a new LDraw meta for this, and add it to the parts themselves. Eliminating the need for an index file altogether.

If I'm understanding this correctly, this suggestion would be to specify the part mapping within the individual part files. Given that BrickLink's part number change over time, wouldn't the implication of doing this be that then the individual LDraw part files would need to be updated? If so, I think an index file would be more preferable.

Also, if something along these lines sees the light of day, it might be worth liaising with rebrickable.com. They have a pretty good database mapping between LDraw, LEGO, BrickLink and BrickOwl part ids and a published API which might simplify the generation of any mapping file.

David

[Milan Vančura]

If I'm understanding this correctly, this suggestion would be to specify the part mapping within the individual part files. Given that BrickLink's part number change over time, wouldn't the implication of doing this be that then the individual LDraw part files would need to be updated? If so, I think an index file would be more preferable.

The index file has more advantages:
* any utility can read just one file than thousands of files
* any standard format (JSON, YAML, XML...) allows us to use standard libraries immediately and easily
* there is a room for extensions (I'd like to have a database of existing colors for each part, one day in future... - I know, I know. It's hard to get and maintain. But let me dream, OK? )
* any change is a problem of this index file, not a certification of the part

[Magnus Forsberg]

I think we have to have this information outside the individual dat-files. The same dat-file would be referenced in many different ElementID. The same dat-file would be used in a red brick as in a blue brick, but have different ElementID.

ElementID = DesignID(shape of the brick) + colour of the brick.


And to make it even worse, sometimes the DesignID mean the shape of the brick, and sometimes it means the design of the print on the brick. The same DesignID could be printed on both a 3626c.dat and on 3626b.dat. But they would have uniqe ElementID numbers.

I think it would quickly become a nightmare to sort all this info correct.

And why do we want our own database when all of this is already at Rebrickable or Brickset?
Lets use what is already done and build on that.

[Philippe Hurbain]

Fully agree with Magnus !

[Willy Tschager]

Last time we recycled all parts through the PT to add the licence we had a stilstood of almost two years so adding this kind of information to the single parts is a no-go.

w.

[Johann Eisner]

I think we have to have this information outside the individual dat-files. The same dat-file would be referenced in many different ElementID. The same dat-file would be used in a red brick as in a blue brick, but have different ElementID.

ElementID = DesignID(shape of the brick) + colour of the brick.


And to make it even worse, sometimes the DesignID mean the shape of the brick, and sometimes it means the design of the print on the brick. The same DesignID could be printed on both a 3626c.dat and on 3626b.dat. But they would have uniqe ElementID numbers.

I think it would quickly become a nightmare to sort all this info correct.

And why do we want our own database when all of this is already at Rebrickable or Brickset?
Lets use what is already done and build on that.

I followed this article with great excitement and did not understand anything at the beginning.
Now that I know what is meant, I also think that we do not have to reinvent the wheel.

[Jaco van der Molen]

I'd like to discuss the argument opened here:

The LDD ldraw.xml file has this structure:

<!-- Arch  3 x  6 x  5  30613.dat -->
<!-- PAVILION 6X3X5  19063 -->
<Brick ldraw="30613.dat" lego="19063"/>
<!-- PAVILION 6X3X5  30613 -->
<Brick ldraw="30613.dat" lego="30613"/>

<!-- Animal Dolphin  6228b.dat -->
<!-- DOLPHIN 9M  6228 -->
<Brick ldraw="6228b.dat" lego="6228"/>

How does it look like in Studio?

How would an information like this treated in an editor? Would it show also the alternate number or just jump to the right part is an element number is entered say in a search?

w.

Following the discussion... have to think about contributing.
After all it was kind of "my idea..." 

[Orion Pobursky]

I'd like to discuss the argument opened here:

The LDD ldraw.xml file has this structure:

<!-- Arch  3 x  6 x  5  30613.dat -->
<!-- PAVILION 6X3X5  19063 -->
<Brick ldraw="30613.dat" lego="19063"/>
<!-- PAVILION 6X3X5  30613 -->
<Brick ldraw="30613.dat" lego="30613"/>

<!-- Animal Dolphin  6228b.dat -->
<!-- DOLPHIN 9M  6228 -->
<Brick ldraw="6228b.dat" lego="6228"/>

How does it look like in Studio?

How would an information like this treated in an editor? Would it show also the alternate number or just jump to the right part is an element number is entered say in a search?

w.

I think this a fairly trivial problem to solve as long as the APIs to Rebrickable and Bricklink are robust. A simple script that runs every so often to update a database and then another script to pull info from the database to auto-generate a file in the format of choice (I think JSON is appropriate). Manual entries to the database would also be available to suppliment the API data. We could even do an initial population of the data by scraping already available KEYWORD data. Maybe a front end to allow searching. I'm super worried about it being 100% accuarate, just "good enough".

[Willy Tschager]

I think this a fairly trivial problem to solve as long as the APIs to Rebrickable and Bricklink are robust. A simple script that runs every so often to update a database and then another script to pull info from the database to auto-generate a file in the format of choice (I think JSON is appropriate). Manual entries to the database would also be available to suppliment the API data. We could even do an initial population of the data by scraping already available KEYWORD data. Maybe a front end to allow searching. I'm super worried about it being 100% accuarate, just "good enough".

Could you find out if this is doable? As for 100% accuracy - we are working on this project for ... some 25 years ... and are still far from perfection ;-)

w.

[David Manley]

Could you find out if this is doable? As for 100% accuracy - we are working on this project for ... some 25 years ... and are still far from perfection ;-)

w.

It is do-able. I've just knocked something together to process the rebrickable data using their published API. My attempt at producing a reasonable output is visible here.

There are a couple of rough edges to remove from my solution but I'll post my approach and scripts on here in the morning, should there be any interest.

David

[Orion Pobursky]

It is do-able. I've just knocked something together to process the rebrickable data using their published API. My attempt at producing a reasonable output is visible here.

There are a couple of rough edges to remove from my solution but I'll post my approach and scripts on here in the morning, should there be any interest.

David

Very nice. I think maybe we should seek a closer partnership with Rebrickable since they seem to be very LDraw friendly.

[Jaco van der Molen]

I think this a fairly trivial problem to solve as long as the APIs to Rebrickable and Bricklink are robust. A simple script that runs every so often to update a database and then another script to pull info from the database to auto-generate a file in the format of choice (I think JSON is appropriate). Manual entries to the database would also be available to suppliment the API data. We could even do an initial population of the data by scraping already available KEYWORD data. Maybe a front end to allow searching. I'm super worried about it being 100% accuarate, just "good enough".

[David Manley]

It is do-able. I've just knocked something together to process the rebrickable data using their published API. My attempt at producing a reasonable output is visible here.

There are a couple of rough edges to remove from my solution but I'll post my approach and scripts on here in the morning, should there be any interest.

David

If anyone wants to tutu with this, this zip file contains the necessary scripts. The requirements are;

• Windows PowerShell
  
• curl (free to download from https://curl.haxx.se/)
  
• saxon (free to download the Home edition from http://www.saxonica.com)
  
• you have a rebrickable account and already have an API key or you obtain a free API key for your account from rebrickable 
  

Comments:

1. I have been running this on Windows 10, using PowerShell 5. It should run on earlier versions of Windows, depending on which version of PowerShell is installed on the OS. Try it and see.
   
2. I'd suggest you alter the PowerShell script part_mapping_generate.ps1 on line 48 and update it with the path to your installation of saxon. You can pass this value in as a command line parameter if you so wish but I suggest this will quickly become tiresome
   
3. While the output format appears to be okay, I need to do some checks on whether all the parts have been output. According to Rebrickable's website, they have 32 000+ parts in their database. The generated XML file currently contains 5 000+ parts. I'll to a bit of digging into this.
   
4. Some rebrickable part ids map to multiple part ids in other systems. For example this part lists two LDraw part numbers. This can occur for systems other than just LDraw. My unconfirmed suspicion is that the first part number listed when there are multiple matches should be taken as the "best" match.
   
5. I don't know whether rebrickable differentiates between the part id being the same between two systems (e.g. rebrickable and LDraw) and a part being only in rebrickable but not in LDraw. When the part id is the same, I think it does not explicitly identify the LDraw part id but is it doing the same thing when there is no equivalent LDraw part? Something I shall be digging in to.
   
6. I keep forgetting how much I dislike PowerShell scripting.
   

Usage:

1. The command is invoked from the Powershell command line thusly;
   .\part_mapping_generate.ps1 your-api-key xml-output-file-name json-output-file-name
   
2. The parameter values influence its behaviour
   

• .\part_mapping_generate.ps1 your-api-key
  Queries rebrickable and output XML to standard output
  
• .\part_mapping_generate.ps1 your-api-key abc.xml
  Queries rebrickable and output XML to the file abc.xml
  
• .\part_mapping_generate.ps1 your-api-key abc.xml abc.jsn
  Queries rebrickable, output XML to the file abc.xml and JSON interim output to abc.jsn
  
• .\part_mapping_generate.ps1 "" abc.xml abc.jsn
  Takes its JSON input from abc.jsn rather than querying rebrickable and output XML to the file abc.xml
  
• .\part_mapping_generate.ps1 "" "" abc.jsn
  Takes its JSON input from abc.jsn rather than querying rebrickable and output XML to standard output
  

If you have issues with getting this to run, please drop me a line.

Regards,

David

[David Manley]

The PowerShell script in the zip file has been updated. It is a bit "gentler" when querying Rebrickable and the shortage of parts being retrieved has been resolved.

Comments:
2. I suggest you alter the PowerShell script part_mapping_generate.ps1 on line 48 69 and update it with the path to your installation of saxon. You can pass this value in as a command line parameter if you so wish but I suggest this will quickly become tiresome.
Do something similar for the curl executable location on line 68.
3. While the output format appears to be okay, I need to do some checks on whether all the parts have been output. According to Rebrickable's website, they have 32 000+ parts in their database. The generated XML file currently contains 5 000+ parts. I'll to a bit of digging into this.
This has been resolved. I believe all parts are now being retrieved.

Usage:

1. The command is invoked from the Powershell command line thusly;
   .\part_mapping_generate.ps1 your-api-key xml-output-file-name json-output-file-name
   
2. The parameter values influence its behaviour
   

• .\part_mapping_generate.ps1 your-api-key
  Queries rebrickable and output XML to standard output
  
• .\part_mapping_generate.ps1 your-api-key abc.xml -output_progress 1
  Queries rebrickable and output XML to the file abc.xml and displays progress when querying Rebrickable.
  
• .\part_mapping_generate.ps1 your-api-key abc.xml abc.jsn -output_progress 1
  Queries rebrickable, output XML to the file abc.xml and JSON interim output to abc.jsn and displays progress when querying Rebrickable.
  
• .\part_mapping_generate.ps1 "" abc.xml abc.jsn
  Takes its JSON input from abc.jsn rather than querying rebrickable and output XML to the file abc.xml
  
• .\part_mapping_generate.ps1 "" "" abc.jsn
  Takes its JSON input from abc.jsn rather than querying rebrickable and output XML to standard output
  

Regards,

David

[David Manley]

Very nice. I think maybe we should seek a closer partnership with Rebrickable since they seem to be very LDraw friendly.

I made a couple of posts over in the Rebrickable forum regarding the mapping of part ids.  The potential implications of the replies is that it would be beneficial to be running some form of confirmatory query of the existence of a LDraw part id as part of the process of creating a mapping file.

Is there (he wrote optimistically) any form of API available to query the LDraw parts database e.g. to obtain a list of the part ids? This could be used as part of the transformational process of Rebrickable's data to create better matching part ids.

Regards,

David

[Willy Tschager]

I made a couple of posts over in the Rebrickable forum regarding the mapping of part ids.  The potential implications of the replies is that it would be beneficial to be running some form of confirmatory query of the existence of a LDraw part id as part of the process of creating a mapping file.

Is there (he wrote optimistically) any form of API available to query the LDraw parts database e.g. to obtain a list of the part ids? This could be used as part of the transformational process of Rebrickable's data to create better matching part ids.

Regards,

David

I guess the only person who can gives a proper answer is Chris. Beside that I'd like to hear from the code-gods how the drawn information can be used in their progs?

w.

[Orion Pobursky]

I guess the only person who can gives a proper answer is Chris. Beside that I'd like to hear from the code-gods how the drawn information can be used in their progs?

w.

There is no formal API. All the individual parts are accessible online but the expection now is for people to download the library and collate thier own database.

[Jaco van der Molen]

Today I was browsing through my LDraw installation folders and found this one: LdSetsConversion.

I have no idea where this came from (Word document origin looks like Italian)
It still is on http://zaxxonit.altervista.org/ldsetsconversion.htm
Dates from March 2016 (?)

Who made this? I cannot find a name or author.

It sure looks like something we are trying to achieve here?

[Jaco van der Molen]

Something worth looking at, I think?

LPub3D has element-ID integration too in the BOM!
See https://github.com/trevorsandy/lpub3d/issues/196
It can get the element-ID based on color too:



And, using some LDView features it can create a partslist with Designnumber and Elemen-ID!

[David Manley]

There is no formal API. All the individual parts are accessible online but the expection now is for people to download the library and collate thier own database.

In reply to one of the questions I posted on the rebrickable forum, the rebrickable team came up with a question/request which, if we were able to assist with here at LDraw.org, would help them to improve their part id mapping data. This would in turn benefit us as their improved mapping would be exposed via their API and hence any transformation of the rebrickable data to LDraw/LEGO/BrickLink ids would have better information.

Fundamentally, they would like to know if there is a way on the ldraw.org website to see the image associated with an official part? While using the Prts Tracker one can see an image for unofficial parts, the same is not true for official parts. The example they give is http://www.ldraw.org/cgi-bin/ptscan.cgi?...ebrickable, with the image for the unofficial part being visible in the "Details" link. There is no equivalent link for an official part and while the narrative of the approval process for the unofficial part may not be of value, perhaps the file header and imgae information could be displayed for the "detail" of an official part?

A related question (from me, rather than the rebrickable team). Could the parts tracker be enhanced to add an optional parameter meaning list the part exactly matching the lookup value. For example, at the moment the link http://www.ldraw.org/cgi-bin/ptscan.cgi?q=3010 will match multiple parts, which is desirable. But could the interface be enhanced to support something like http://www.ldraw.org/cgi-bin/ptscan.cgi?q=3010.dat&exact, so the tracker would list just the single part?

Regards,

David

[Orion Pobursky]

I replied to this on Rebrickable.

[Jaco van der Molen]

Something worth looking at, I think?

LPub3D has element-ID integration too in the BOM!
See https://github.com/trevorsandy/lpub3d/issues/196
It can get the element-ID based on color too:


And, using some LDView features it can create a partslist with Designnumber and Elemen-ID!

LPub3D has implemented Rebrickable.com and Brickset.com to HTML Part List lookup sites
See https://github.com/trevorsandy/lpub3d/issues/239

Perhaps we are looking at the same thing here, but only in the other LDraw software.

[Willy Tschager]

I replied to this on Rebrickable.

Is there going on some work in the background I'm not aware of? Could you please update us on the progress?

w.

[Orion Pobursky]

Is there going on some work in the background I'm not aware of? Could you please update us on the progress?

w.

So basically I think Rebrickable has done this work for us already. The added bonus is that they also have a easily usable API as well. Therefore, I don’t think we need to do anything except work with them by finding errors and reporting them via their mechanism for doing so.