Insert Related Database

[Ben Supnik]

Hi Y'all,

I am working on "Insert Related…" in BrickSmith. The idea is simple: select a door frame, and BrickSmith can tell you which doors fit, and put them into the correct location.*

Insert Related solves two problems:

- If you haven't memorized every single Lego Part, it can be hard to find the right matched part, e.g. the right tire for the wheel, the right door for the frame, the right glass for the window. Some of these are manageable (e.g. windows), some I still don't know after tens of thousands of parts inserted (e.g. tires). Insert related is a database that "knows" which parts fit for certain special roles.

- Some parts have relative origin offsets that aren't on the major brick grid> This typically happens when a part has its origin set for easy rotation instead of easy relative placement. So even once I find the right door for the frame, how does it 'snap' in?

I have two questions:
1. Is anyone else interested in the data for other modeling programs?
2. Does anyone have any feedback on the proposed data format? (This matters mainly if anyone else wants to use the data.)

There's nothing BrickSmith specific going on, hence the inquiry.

Here's what I have right now:

122c01.dat 3641.dat -31 6 0 0 0 1 0 1 0 -1 0 0 Left Tire
122c02.dat 3641.dat -31 6 0 0 0 1 0 1 0 -1 0 0 Left Tire
122c01.dat 4084.dat -31 6 0 0 0 1 0 1 0 -1 0 0 Left Tire
122c02.dat 4084.dat -31 6 0 0 0 1 0 1 0 -1 0 0 Left Tire
122c01.dat 3641.dat 31 6 0 0 0 -1 0 1 0 1 0 0 Right Tire
122c02.dat 3641.dat 31 6 0 0 0 -1 0 1 0 1 0 0 Right Tire
122c01.dat 4084.dat 31 6 0 0 0 -1 0 1 0 1 0 0 Right Tire
122c02.dat 4084.dat 31 6 0 0 0 -1 0 1 0 1 0 0 Right Tire

Basically the format is:

<parent file> <child file> <relative transform> <placement name>

The relative transform is the matrix that a 1 directive would have for the child if the parent is at 0,0,0, unrotated, and the parts are put together according to the relationship described to the right. The relationships names help disambiguate (for UI purposes) multiple attach points, e.g. do we want to put this tire on the left or right wheel, do we want the door to open to the left or right, is this the left or right shutter?

It is intentional that the relationship name is the same for multiple lines with different parts - the idea is that the user can say two things:

"If I want to add part 3641.dat to 122c01.dat, what are my choices?"

(The answer is that 3641.dat can act as a "Left Tire" or "Right Tire", and there are different transforms to do that.)

or the user can ask:

"If I want to add a Left Tire to 122c01.dat, what parts can do this?"

(The answer is that both 3641.dat and 4084.dat fit as tires - by luck the transform is the same for both but this won't be true some of the time.)

The data is 'flat' - that is, relationships are provided only for final parts, not for sub-parts.

To create the data, I created a very simple .mpd file with the parts assembled 'correctly' (using various steps and sub-models to structure the parts) and then ran a Python script over it to get the final flat results. This means that, to create the data, I simply modeled each part once, visually checking to see if the part was positioned correctly.

Anyway, if this is of interest to anyone else working on modeling software, please let me know - I'm happy to share the data.

Cheers
Ben

* My previous post on this:
forums.ldraw.org/showthread.php?tid=8008,8025
got side-tracked into a discussion of 'true connectivity'. This is not that at all - it's just a way to insert parts. I would like to do connectivity some-day, but connectivity is an ocean-liner of a feature, and this is a rubber raft. Or something like that.

[Roland Melkert]

I'd be interested, even with the ocean liner in place this information would be helpful.

I was thinking about something similar like this a while back but decided to push it to the next next LDCad version thinking it's part of connectivity which I'm planning for a later version.

You could probably collect some information from scanning the headers of the library, most companion parts use a identity matrix anyway and share similar descriptions or !help lines.

For example the wheel and tyre parts usually have the same diameter in their description (you could setup regex rules for this in the cfg file). The list would then automatically assimilate new parts in new library's and you only have to add information about less obvious parts to the config file.

I was also thinking about a self learning feature at some point using grouping.

[Philippe Hurbain]

About matching parts, Bricklink contains useful data, placed at the bottom of part description. Eg. here http://www.bricklink.com/catalogItem.asp?P=6870 we learn that 6859 matches 6870 part.
You can see the database here: http://www.bricklink.com/catalogRel.asp
SR3D builder has a tire matching tool.

[Ben Supnik]

Hi Philippe,

The tire matching tool was my inspiration. :-) :-) But since S3RD has connectivity, the tire matching list contains only parent/child and no coordinates...it doesn't need them.

Re: Bricklink, thanks -- that's really helpful for seeing if I missed combinations. Sadly sometimes the offset is not published, not in the HELP tags and not obvious. But...that's why such a related parts list might be useful. :-)

Cheers
Ben

[Steffen]

the idea of using a mpd file to store the relationships between related parts is a great one here IMHO.

it simply allows to provide a list of part combinations,
and this list easily can be used by the tool to insert the related parts.

my suggestion would be to provide this mpd file together with the tool,
and make the tool parse that list at runtime

[Ben Supnik]

Hi Stephen,

An email thread with Roland convinced me to use the .ldr file directly too -- this is what I have for the current format (an excerpt):

Code:

0 !PARENT
1 4    0 0 0    1 0 0 0 1 0 0 0 1 122c01.dat
1 4    0 0 0    1 0 0 0 1 0 0 0 1 122c02.dat
0 !CHILD Left Tire
1 1    -31.000000 6.000000 0.000000    0.000000 0.000000 1.000000  0.000000 1.000000 0.000000  -1.000000 0.000000 0.000000  3641.dat
1 1    -31.000000 6.000000 0.000000    -0.000000 0.000000 1.000000  -0.000000 1.000000 -0.000000  -1.000000 -0.000000 -0.000000  4084.dat
0 !PARENT
1 4    100 0 0    1 0 0 0 1 0 0 0 1 122c01.dat
1 4    100 0 0    1 0 0 0 1 0 0 0 1 122c02.dat
0 !CHILD Right Tire
1 1    131.000000 6.000000 0.000000    -0.000000 0.000000 -1.000000  -0.000000 1.000000 -0.000000  1.000000 0.000000 -0.000000  3641.dat
1 1    131.000000 6.000000 0.000000    -0.000000 0.000000 -1.000000  -0.000000 1.000000 -0.000000  1.000000 0.000000 -0.000000  4084.dat

- The comments tell the parser what the relationship names are and who is a parent-child.
- Parents can be offset from 0,0,0 to make the file more viewable.
- The file is currently 'flat' - steps and mpd sub-files are ignored - so an author can use them for organization but they are not needed to make the relationships.

Cheers
Ben

[Steffen]

a good approach, I've some more questions/ideas/inspirations to share:

(a)
could you get rid of the unnecessary trailing zeros to save filesize and parsing time?

(b)
using LDR as file extension instead of MPD for this contents seems wrong to me.
your contents is a list of (PARENT+CHILDREN) assemblies, and the natural way to store something like that would be MPD
, as far as I can see, because LDR is/should be/should be thought of "1 assembly", not "a list of assemblies".
to word it more abstract: LDR is "an entity", and MPD is "a list of entities". you need the list. what were the reasons to use LDR instead of MPD?

[Ben Supnik]

Hi Steffen,

(a)
could you get rid of the unnecessary trailing zeros to save filesize and parsing time?

I think so -- the output is just what BrickSmith natively saves.

(b)
using LDR as file extension instead of MPD for this contents seems wrong to me.
your contents is a list of (PARENT+CHILDREN) assemblies, and the natural way to store something like that would be MPD
, as far as I can see, because LDR is/should be/should be thought of "1 assembly", not "a list of assemblies".
to word it more abstract: LDR is "an entity", and MPD is "a list of entities". you need the list. what were the reasons to use LDR instead of MPD?

[/quote]

Hrm - there's a few issues here.
- The file itself I think is an MPD file, because it contains multiple MPD parts.
- The extension is therefore definitely wrong - it should have been an .mpd file. I think BrickSmith auto-suggests .ldr in the file-save dialog box for all files, so I just didn't notice.
- My format may not match the "natural way" to store such a suggestion database. Since .mpd files aren't really meant to store suggestion/related part lists, the format was going to require some extra conventions or metas no matter what.

The format which I have so far (which allows mpd parts but does not depend on them for the structuring of the actual relationship data) was chosen to simplify editing and maintaining a suggestion file in-editor. The original format was a simple flat list for easy parsing and databasing; Roland suggested to me that if the file itself was an LDraw file then it would be easier for users to add part relationships. So the current format is designed to create a 'workspace', with one mpd sub-model containing several relationships, and each relationship containing whole sets of parents and children. The setup is pretty arbitrary - it's what I found to be straight-forward to work with and may not be usable to anyone else.

(In particular, I wanted to be able to work with whole sets of related parts on one screen and not have to change MPD sub-models for every single relationship.)

Cheers
Ben

[Travis Cobbs]

Your excerpt isn't an MPD. Maybe the file at large is, but nothing in your excerpt suggests that this would be the case. Also, please see this recent decision by the LSC. It explicitly states that any LDraw file may make use of the MPD language extension (although usage of the MPD extension is forbidden in official parts). My personal opinion is that LDR should be used as the file extension for all files visible to an end user.

[Allen Smith]

a) Removing "unnecessary zeroes" is not a meaningful optimization. This is not where the processing time is spent when opening a file. Also, filesize is essentially irrelevant. This is an aesthetic consideration(*) and nothing more.

b) The default extension for an MPD file is .ldr. (It is also acceptible to use .mpd or .dat, but in my opinion, you shouldn't.)
http://www.ldraw.org/article/218.html

(*) Bricksmith has an internal option to output formatted columns for easier reading, which is what you're seeing in Ben's file. It's off by default because LDrawers seem to hate it, so I don't know how he managed to obtain it.

[Ben Supnik]

Hi Y'all,

@Travis - you are correct - the excerpt is from a sub-section of a single MPD sub-model, and the format doesn't require MPD to encode relationships - the parser just isn't killed by it. The whole file has a few MPD sections to divide relations by theme - that's a human-readable-sanity-thing.

@Allen - I think that output may have gone through a Python post-processor, which might be responsible for the atypical formatting. As I've changed the format, I re-process my work in progress.

Cheers
Ben

[Sergio Reano]

Hi Ben,
as told by Phil, in my SR3DBuilder I'm using a very similar system to manage tires and the idea to extend it to windows frame and glasses.

Furthermore it could be have sense to create a more general management system allowing all (or at least most commonly used) parts having some kind of corresponding ones to be included in this kind of management (thinking to many hinges in the parts database).

We can also use your proposed encoding, since I can simply ignore the addition informations...

[Ben Supnik]

Hi Sergio,

My original format - a "flat" file with one line per relation - was inspired by your tires file. But of course, the tires file has a lot less numbers (since the coordinates are not needed) so it is an easier file to work with.

At this point I am not sure what I think...I think an .ldr/.mpd file is more useful for users editing such data, but a flat file would be easier for interchange between actual 3-d editors (in particular, if a 3-d editor wants a proprietary format, it would be easier to get there from a flat file).

Cheers
Ben

[Sergio Reano]

Hi,
it is not a problem for my application to strip off unnecessary information from a text file, even if you name it .ldr, .mpd or what else, especially if the required information for my application are in the first 2 fields.
The only "important" thing is to choose a common field separator: I'm using the <TAB> character for this.

Also the process could easily be a step by step trying to include in the management most commonly used parts before and adding the other at a second time...

Now the file format is totally your choice. IMPO a simple text file could be more indicated since it does not contains information about parts definition or model, but it is a "system" file for the application.
As I experimented (please nobody upset reading the following: it is just my experience up to now), it is quite difficult that someone apart you (or me, if we are going to work together on it, and maybe Phil Urbain ) are going to maintain the information in this file

Just for information, the wheel/tyre positioning is nearly always at coords 0,0,0 so the work to include them in your program is quite easy.

[Steffen]

I clearly favor normal LDR or MPD files here because then a user can easily
add part relations to it using the existing editors.
Additionally, the parts can be positioned relatively to another this way, which is nice for
windows+glasses (or: +hinges), wheels+tyres, etc.

I see something nice forthcoming here, especially because of the synergy between Ben and Sergio, NICE!

[Allen Smith]

I also agree that an LDraw model is the best way to represent this information. The ease of editing is unparalleled. It's one less parser to write too.

[Roland Melkert]

I like to tread this kind of information as 'examples' of part usage, so it's (imho) logical to use the ldr format it self.

@ben

Have you thought about my suggestion of using the STEP meta for separation instead of the child/parent ones? It will lower the threshold for adaption in other programs even more.

You could use normal comment metas to assign names (like the child does in your code) to the groups separated by the step metas and an editor only needs to add support for exclusive step viewing in order to make these files editable for everyone.

[Ben Supnik]

Hi Guys,

@Allen, I've been meaning to push my implementation to SVN for you (but SF appears to be borked??) - my code is its own parser using your string utilities - if I had realized I was going to use a real .mpd file, I probably could have opened it as a file object and traversed it for parts...the code was born out of parsing a simple text list. Well, we can clean that up. :-)

@Roland, the only problem with using steps is that relation names are (currently) fundamental to the schema of the system. In other words, it's sort of important to know that a part is a shutter or a window pane.

I suppose a program that wants to ignore relation names could _read_ the file without the meta comments, but as long as the central schema is to have relation names, saving work compliant with that program (and risking not having the child name metas) would result in useless data.

In other words, my thought was that if we want to _require_ names, I should make a format that won't function at all without them.

@All: are relation names necessary? I think they are. For example, most two-wheel holders can take the same wheel hub part in two locations; the relation names are the only user-identifiable way of picking which one to use.

cheers
Ben

[Travis Cobbs]

I'm not sure if names are necessary or not.

Having said that, one thing I didn't notice when I initially looked at your file is that I would argue that it goes against best practices for meta-commands. If you feel it is necessary to introduce new meta-commands to make this work, ideally you should only introduce a single one, and have it have one ENUM argument and then optional info after that. The LDraw spec states that you should use the program name of your program, but since you're trying to make this useful for everyone, that doesn't seem appropriate. However, I still feel that it's better to have related functionality be grouped under a single meta-command, instead of two like you have. It makes it obvious that the statements are related.

For example, you could do the following:

Code:

0 !RELATION PARENT
1 4    0 0 0    1 0 0 0 1 0 0 0 1 122c01.dat
1 4    0 0 0    1 0 0 0 1 0 0 0 1 122c02.dat
0 !RELATION CHILD Left Tire
1 1    -31.000000 6.000000 0.000000    0.000000 0.000000 1.000000  0.000000 1.000000 0.000000  -1.000000 0.000000 0.000000  3641.dat
1 1    -31.000000 6.000000 0.000000    -0.000000 0.000000 1.000000  -0.000000 1.000000 -0.000000  -1.000000 -0.000000 -0.000000  4084.dat
0 !RELATION PARENT
1 4    100 0 0    1 0 0 0 1 0 0 0 1 122c01.dat
1 4    100 0 0    1 0 0 0 1 0 0 0 1 122c02.dat
0 !RELATION CHILD Right Tire
1 1    131.000000 6.000000 0.000000    -0.000000 0.000000 -1.000000  -0.000000 1.000000 -0.000000  1.000000 0.000000 -0.000000  3641.dat
1 1    131.000000 6.000000 0.000000    -0.000000 0.000000 -1.000000  -0.000000 1.000000 -0.000000  1.000000 0.000000 -0.000000  4084.dat

In the above, you would be introducing the !RELATION meta-command. PARENT and CHILD are required as the first argument, and every other argument after CHILD is the child's name. At least two current official meta-commands behave like this (BFC and !TEXMAP).

[Ben Supnik]

Hi Travis,

That makes sense - I can push the comment/metas into a namespace. Are the metas supposed to have ! escaping them or no? (The BFC appears to not have that.)

cheers
Ben

[Roland Melkert]

Everything after the meta keyword (eg !RELATION) is up to you as long it's utf8.

That said, I would recommend escaping or some other way of separation to reserve the possibility to add parameters later on.

[Travis Cobbs]

All new official meta-commands are required to have a ! prefix, and have the command name be in all caps. (It is strongly recommended that unofficial ones have the same format.) BFC (0 BFC ...) is too old to have that. Same with MPD (0 FILE ...). Comments are recommended to have a // prefix (0 // Some comment...).

See the Line Type 0 section of the LDraw 1.0.2 spec.

[Travis Cobbs]

I don't see any need to escape in this case. The new meta command (!RELATION) in my example has one required argument, which is the relation type. Right now, there are two possible relation types: PARENT and CHILD. The PARENT relation type has no further arguments. The CHILD relation type is required to be followed by a free-form string which acts as the name of the relation. If additions are needed in the future, new relation types can be introduced. Adding other arguments to the existing relation types would just cause problems, so any changes at all would need to be in the form of new types with new names.

For example, if in the future it was decided that PARENT could stand to have a name also, then instead of simply adding a name to the existing PARENT type, a new type would be create that requires a name (like PARENT2, or something).

[Roland Melkert]

With escaping I mean the name string, to avoid problems like we are having (sort of) with the type 1 line.

We can't extend that line's format (eg with group info) without breaking backwards comp.

[Tim Gould]

You can add more sub-metas

Code:

0 !RELATION PARENT Darth Vader
0 !RELATION GROUP Sith
1 ...
0 !RELATION CHILD Luke
1 ...

[Ben Supnik]

Hi Guys,

I just want to bring up one point. I _think_ y'all already know this, but just to be sure:

- It is _not_ my intention to introduce a new meta command for _general_ use in MPD or LDR files.
- It is _not_ my intention to allow authors to specify relations within their working documents.

The goal here is to make the relation data file user-editible, not an extension to .ldr files.

With that in mind, I agree that it makes sense to have any custom metas conform to the .ldr guidelines; some client programs may wish to use a single parser for all cases.

But I would say that having these metas in an .mpd file other than the one that is used by a particualr editor to specify relations (e.g. similar to Sergio's "Tires" file) is an error and the metas should be ignored.

cheers
Ben

[Steffen]

may I bring your attention to this thread

http://forums.ldraw.org/showthread.php?t...7#pid11117

, where the connectivity / snapping feature was brought up again