Standardize official search path

[Travis Cobbs]

It has been requested that we standardize the official search path. So I'm creating this thread to do that. Right now, I think that most programs follow in LDView's footsteps, but I'm open for the official search path being different. LDView by default looks for files in the following order:

MPD subfiles
Model Directory (the directory that the top-level model resides in)
<LDraw>\p
<LDraw>\parts
<LDraw>\models
<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

If we agree that the above order is correct, we can then discuss where to put it in the specification. If we want a different default search path, we can discuss that.

Note: If your name isn't Travis Cobbs, Chris Dee, or Roland Melkert, please don't respond to this thread. Read the rules for this board. Other than the three of us, only admins and mods can post here, but they aren't supposed to do so. If you want to bring up concerns, do so in the Parts Authoring board.

[Chris Dee]

It has been requested that we standardize the official search path. So I'm creating this thread to do that. Right now, I think that most programs follow in LDView's footsteps, but I'm open for the official search path being different. LDView by default looks for files in the following order:

MPD subfiles
Model Directory (the directory that the top-level model resides in)
<LDraw>\p
<LDraw>\parts
<LDraw>\models
<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

If we agree that the above order is correct, we can then discuss where to put it in the specification. If we want a different default search path, we can discuss that.

Note: If your name isn't Travis Cobbs, Chris Dee, or Roland Melkert, please don't respond to this thread. Read the rules for this board. Other than the three of us, only admins and mods can post here, but they aren't supposed to do so. If you want to bring up concerns, do so in the Parts Authoring board.

That sequence seems logical to me.

[Roland Melkert]

MPD subfiles
Model Directory (the directory that the top-level model resides in)
<LDraw>\p
<LDraw>\parts
<LDraw>\models

I have no problem with this part (it basically just adds models to the spec.)

But I never really liked the following

<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

Because it breaks the 'package', also imho you should look in the unofficial location before the official one because part and primitives can be updated.
With the current order the old ones will still load.


I also would like to make clear this order is to be followed within the context of the currently begin processed ldraw file.

Meaning a stud.dat in the mpd's subfolder (or as a submodel) will not be used for each part in the scene.

It will only be used if it's actually referenced inside the mpd itself trough a "1 ..... stud.dat" line.

This because all references inside the mpd will be tested against the parent folder.

But once e.g. 3001.dat start loading it will look inside it's own folder (parts) first before going to the library p folder etc.

This will also ensure you can cache meshes.

[Travis Cobbs]

MPD subfiles
Model Directory (the directory that the top-level model resides in)
<LDraw>\p
<LDraw>\parts
<LDraw>\models

I have no problem with this part (it basically just adds models to the spec.)

But I never really liked the following

<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

Because it breaks the 'package', also imho you should look in the unofficial location before the official one because part and primitives can be updated.
With the current order the old ones will still load.

I'm OK if Unofficial is left out of the official spec, but I am not OK with putting it somewhere else in the spec. (Also, I'd prefer if it were documented in the spec as an optional pair of paths that can be checked after everything else.) The Unofficial directory was something that I created for LDView. If LDView cannot find a part, it checks ldraw.org for an unofficial part. If it finds it, it downloads it and puts it into Unofficial. Note that this was not designed to allow users to have the latest possible unofficial version of existing parts. It was designed purely to find (and auto-download) missing parts. LDView never deletes things it has downloaded into Unofficial. However, since Unofficial comes at the end of the search path, that doesn't really matter, since as soon as a part becomes official, the official version will be loaded instead of the unofficial version. (LDView also keeps track of when it downloaded unofficial files so that after a configurable time has passed, it can check to see if the file has updated. I only does this if it loads an unofficial file due to not finding it anywhere else in the search path.)

I also would like to make clear this order is to be followed within the context of the currently begin processed ldraw file.

Meaning a stud.dat in the mpd's subfolder (or as a submodel) will not be used for each part in the scene.

It will only be used if it's actually referenced inside the mpd itself trough a "1 ..... stud.dat" line.

This because all references inside the mpd will be tested against the parent folder.

But once e.g. 3001.dat start loading it will look inside it's own folder (parts) first before going to the library p folder etc.

This will also ensure you can cache meshes.

I'm not sure I completely understand what you are saying. My expectation (based on the proposed search path) is that if the user puts a custom stud.dat in the directory alongside the main model (or inside the mpd), the proposed path order (and what I feel is correct) would use that stud.dat for the entire model. LDView doesn't support having two different versions of the same filename at once. Once LDView loads a file (from disk, or while processing an MPD), that loaded copy goes into its in-memory cache, and all future references to that filename come from the cache. The cache isn't hierarchical either (other than that if the type 1 line includes any slashes, those are part of the filename key in the cache).

Just to be clear, I'm not saying that we have to define the spec based on how LDView currently behaves. I am saying that I think it behaves fine, though. If we define the spec to be different, we should do so because we don't think the way I proposed is right.

Your last sentence seems to say that you are arguing in favor of what LDView already does. However, the rest of that section is arguing for something else, which is why I said that I don't completely understand what you are saying.

In case I'm not clear here, LDView does the following:

1. Starts at the model the user opens (the main model). Records the directory that this model resides in for use in search path lookups.
   
2. Reads the model file.
   
3. If the model file is an MPD, parses it enough to know what MPD submodels are inside. Each of these goes into its global "loaded models" cache keyed off of the filename in the 0 FILE line.
   
4. Second pass full parse of the file (as well as parsing the MPD submodels). For each type 1 line, it first checks to see if there is an entry in its global "loaded models" cache matching the filename portion of the type 1 line. If not, it uses the search path to find the specified file and jumps back to step 2 for that file, then loads it and inserts it into the global "loaded models" cache based on the type 1 filename, then references the result. If the model was already in the global "loaded models" cache, it instead references the cache entry.
   

[Roland Melkert]

It was designed purely to find (and auto-download) missing parts. LDView never deletes things it has downloaded into Unofficial. However, since Unofficial comes at the end of the search path, that doesn't really matter

If that is the intend of those folders I'm ok with it.

My expectation (based on the proposed search path) is that if the user puts a custom stud.dat in the directory alongside the main model (or inside the mpd), the proposed path order (and what I feel is correct) would use that stud.dat for the entire model.

I disagree on this, a stud.dat should not replace all studs in the whole scene if it's located in the mpd itself or its folder.

For example d:\test contains only these two files.
d:\test\awsomeModel.mpd
d:\test\stud.dat

The mpd contains:

Code:

0 FILE main.ldr
1 4 0 0 0 1 0 0 0 1 0 0 0 1 3001.dat
1 4 0 0 0 1 0 0 0 1 0 0 0 1 myPart.dat

0 FILE myPart.dat
1 16 0 0 0 1 0 0 0 1 0 0 0 1 stud.dat

In this case the references should translate like so:

For main.ldr
1 4 0 0 0 1 0 0 0 1 0 0 0 1 3001.dat  --> <lib>\parts\3001.dat
1 4 0 0 0 1 0 0 0 1 0 0 0 1 myPart.dat  -->  <mpd>\myPart.dat

Then loading for 3001.dat starts within its own context, meaning it does nothing with the mpd content nor files inside that mpd's folder.
So any stud.dat will be loaded as <lib>\p\stud.dat  unless 3001.dat has an embeded stud.dat (should never happen but could have been edited.) or another stud.dat is also present in the <lib>\parts folder.

Then for myPart.dat it will translate like
1 16 0 0 0 1 0 0 0 1 0 0 0 1 stud.dat  --> d:\test\stud.dat  (because its in the current context's parent folder 'd:\test'.)

Hope this make it more clear on how I always assumed search resolving should go.

The real question here is do we WANT a local stud.dat to affect the global scope or should it only affect its local context (like I assumed).

imho the local context approach is much more logical, global stuff should be handled by prefixing extra search locations to the global order.

edit: I assumed 3001.dat has no further dependencies. Otherwise it would recursively keep applying the context approach)

[Travis Cobbs]

As mentioned in the thread that's open to public content, I still feel that my proposed behavior is in fact the intended behavior.

[Willy Tschager]

[SteeCo hat, Admin hat] ON

Please bring this to an end and ratify.

[SteeCo hat, Admin hat] OFF

w.

[Roland Melkert]

[SteeCo hat, Admin hat] ON

Please bring this to an end and ratify.

[SteeCo hat, Admin hat] OFF

I haven't had the time to research the 'defacto' scoping behavior in the wild some more.

So maybe we should only vote on the folder order, as the scoping stuff really shouldn't be rushed.

I'm ok with:

<LDraw>\p
<LDraw>\parts
<LDraw>\models
<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

Given <LDraw> is a 'library', meaning you could have multiples of those.

[Chris Dee]

Because it breaks the 'package', also imho you should look in the unofficial location before the official one because part and primitives can be updated.
With the current order the old ones will still load.

It would be wrong, and dangerous, to look in the Unofficial library before the official one. A updated primitive could be added to the unofficial library without corresponding changes to the other files in which it is used. The Parts Update build process should ensure that the Official lbrary has a coherent set of dependent files - updated primtives and subfiles are only released along with the necessary concommitant changes to their parents.

[Willy Tschager]

[SteeCo hat, Admin hat] ON

Any progress on this?

[SteeCo hat, Admin hat] OFF

w.

[Roland Melkert]

Any progress on this?

I'm still having very big problems with the proposed (some say defacto) scoping rules as they will completely ruin the (beautiful) recursive nature of LDraw.

Not to mention they are a complete nightmare to implement in a multi document environment.

I still want to gather information on how the existing tools handle this, but most of them have only one open model to deal with so the big issue doesn't even concern them.

That said I do think we could vote on the path order alone, like I wrote in post #8

[Roland Melkert]

I still want to gather information on how the existing tools handle this, but most of them have only one open model to deal with so the big issue doesn't even concern them.

Did a quick test, and the proposed scoping is far from defacto.

   

LDView and brciksmith seem to ignore it altogether.

MLCad seem to apply the 'expected' scoping, but it crashes while opening another model afterwards. It's  complaining about cross model references so I'm guessing it is trying to use the stud.dat from the previous loaded mpd.

Given this is far from defacto I see no real reason to make it the standard way of doing things.

I think we should keep the format as recursive as possible, only compromise might be to handle libraries different so .e.g a stud.dat in unooficial/p would be used for all bricks loaded but a stud.dat in a mpd should, imho, NEVER override a stud.dat in a library location.

Because that would mean bricks can look different depending on which models loaded. So when you want multiple modles to be loaded at once, you would need to keep track of a 'brick context' which is a whole lot of administration and requires lots of extra memory etc.

Not to mention the mess it would cause with large recursive models (e.g. datsville)

And I seriously wonder why anyone would want this behavior. Why not just use a unofficial library when working with new parts etc?

my 2cts, excuse the rant

[Roland Melkert]

Did a quick test...

This is how it looks in LDCad, thought it was already posted above but apparently that was in another thread.

   

This is how I always understood the LDraw format to work, recursive at a per file level. So the embedded stud.dat is only used by models refering stud.dat inside the mpd file itself

[Travis Cobbs]

This is how it looks in LDCad, thought it was already posted above but apparently that was in another thread.



This is how I always understood the LDraw format to work, recursive at a per file level. So the embedded stud.dat is only used by models refering stud.dat inside the mpd file itself

I know this is old, but I'm replying to this before replying to another thread.

Two things:

1. LDView behaves like you see only because you chose to use a primitive that LDView can substitute for your sample. When primitive substitution is enabled in LDView, it always ignores any data in the associated dat file of a substituted primitive. So, "stud.dat" is directly recognized by LDView as being a stud primitive, and it ignores any and all on-disk definitions for that when primitive substitution is enabled. Not all files in the <LDraw>\p directory are directly supported by LDView's primitive substitution, so some primitives could be included in an MPD and have that definition show up in LDView.
   
2. If you turn off primitive substitution in LDView, I think you will see all the studs become yellow. I can change its behavior regarding MPD-embedded files to make them local to the MPD.
   

[Willy Tschager]

I hope you reach agreement on this.

w.

[Roland Melkert]

Working on LDCad 2's file loader made me revisit this discussion.

We never voted on formalizing the order so I wanted to remind people about that, and also would like to suggest one additional change:

<LDraw>\p
<LDraw>\parts
<LDraw>\models
<LDraw>\
<LDraw>\Unofficial\p
<LDraw>\Unofficial\parts

Above is like last mentioned order, but with library root folder inserted.

This will make the LDConfig.ldr file findable using a standard search, which imho makes the loading mechanics cleaner as it basically is assumed to be at the root of the model tree.

edit: It will also uniform user locations as those would just be libraries without the p and parts folders.

Thoughts?

Or just vote one the original order (without scoping rules, as those are still not agreed upon).