Search path discussion

[Orion Pobursky]

Since non-LSB members can't comment on the LSB forum, this thread is for discussion of the search path proposal there.

To kick this off I'd like to comment on something Roland posted:

I disagree on this, a stud.dat should not replace all studs in the whole scene if it's located in the mpd itself or its folder.

This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.

[Philippe Hurbain]

This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.

Fully agree. Embedded parts should have precedence so a model using unofficial parts won't be broken inadvertantly. I remember a time when embedding parts was little used and it was a struggle to find again the unofficial parts used by author.

[Roland Melkert]

Fully agree. Embedded parts should have precedence so a model using unofficial parts won't be broken inadvertantly. I remember a time when embedding parts was little used and it was a struggle to find again the unofficial parts used by author.

I don't understand. If the parts are embedded in the MPD they will be loaded instead of the library ones.

But only withing the context of the model.

So if you put a stud.dat inside the mpd with e.g. a yellow 4-4disc.dat only the embedded files actually referencing stud.dat will use that stud variant.

All the other non embedded files will use the normal stud.dat as found in <lib>\p

Like so:
   

If I understand you correctly you would want all studs to be yellow in that image ?

LDView seems to ignore it and use the library one for everything.
   


mpd code:

Code:

0 FILE main.ldr
1 7 0 0 0 -1 0 0 0 1 0 0 0 -1 819.dat
1 10 40 -24 -30 -1 0 0 0 1 0 0 0 -1 3065.dat
1 10 0 -24 -30 -1 0 0 0 1 0 0 0 -1 3065.dat
1 10 -30 -24 -20 0 0 -1 0 1 0 1 0 0 3065.dat
1 10 -40 -48 20 -1 0 0 0 1 0 0 0 -1 brick.dat

0 FILE brick.dat
0 UNOFFICIAL PART
0 BFC CERTIFY CCW
1 16 -10 0 0 1 0 0 0 1 0 0 0 1 stud.dat
1 16 10 0 0 1 0 0 0 1 0 0 0 1 stud.dat
1 16 0 24 0 20 0 0 0 24 0 0 0 20 box.dat

0 FILE stud.dat
0 UNOFFICIAL PART
0 BFC CERTIFY CCW
1 16 0 0 0 6 0 0 0 1 0 0 0 6 4-4edge.dat
1 16 0 -4 0 6 0 0 0 1 0 0 0 6 4-4edge.dat
1 16 0 0 0 6 0 0 0 -4 0 0 0 6 4-4cyli.dat
1 14 0 -4 0 6 0 0 0 1 0 0 0 6 4-4disc.dat
0 1 16 0 -4 0 1 0 0 0 1 0 0 0 1 logo3.dat

[Orion Pobursky]

That's not the way it's supposed to work. If stud.dat is embedded in the mpd or located in the same directory as the model file then it should replace all instances of stud.dat for the entire model. This allows, say, a part author to test out a fix on a bunch of different parts simultaneously or a model author to include a custom version of some commonly file (e.g. a stud with a custom logo).

[Roland Melkert]

That's not the way it's supposed to work. If stud.dat is embedded in the mpd or located in the same directory as the model file then it should replace all instances of stud.dat for the entire model. This allows, say, a part author to test out a fix on a bunch of different parts simultaneously or a model author to include a custom version of some commonly file (e.g. a stud with a custom logo).

You could do that by adding your custom/development library paths in front of the official ones.

The main thing I'm disliking about the global scope thing is the fact an embeded file which happens to have the same name as some official part would break all dependencies.

The 'context' method will only affect its local scope so there will never be problems with duplicate names, it's comparable with local variables in a programming language.

[Travis Cobbs]

The only reason you saw what you did in LDView is because you had primitive substitution enabled (which is the only way to see stud logos in LDView). I didn't mention it in the search path discussion, but LDView's primitive substitution trumps the search path when it is enabled. If LDView recognizes a file to be a primitive when primitive substitution is enabled, it never even bothers to look at the file system.

[Travis Cobbs]

You could do that by adding your custom/development library paths in front of the official ones.

The main thing I'm disliking about the global scope thing is the fact an embeded file which happens to have the same name as some official part would break all dependencies.

The 'context' method will only affect its local scope so there will never be problems with duplicate names, it's comparable with local variables in a programming language.

At this point in time, I think it's safe to say that if a user embeds a *.dat file in their MPD, they intend it to replace the *.dat file of that same name in the LDraw parts library. Nobody should be using *.dat for personal files anymore. So I still feel that this global replacement is correct.

Note that putting files alongside the main model to do the same thing is a leftover from the days before MPDs. However, I still think that it is consistent to continue with this behavior.

[Roland Melkert]

At this point in time, I think it's safe to say that if a user embeds a *.dat file in their MPD, they intend it to replace the *.dat file of that same name in the LDraw parts library. Nobody should be using *.dat for personal files anymore. So I still feel that this global replacement is correct.

So to be clear, the way you expect it to work is like so:

given a mpd containing main.ldr, brick.dat and stud.dat  with a reference tree like so:

d:\model.mpd
  main.ldr
    3001.dat
       s\3001s01.dat
         stud.dat
   brick.dat

  brick.dat
    box.dat
    stud.dat

  stud.dat


The main.ldr reference will look in these
<mpd>    <--hit
d:\
<lib>\p
<lib>\parts

The 3001.dat reference will look in these:
<mpd>
d:\
<lib>\p
<lib>\parts    <-- hit

The s\3001s01.dat reference in 3001.dat will look in these:
<lib>\parts  (because its the parent folder of 3001.dat, just like d:\ is for the mpd)    <-- hit
<mpd>
d:\
<lib>\p 
<lib>\parts 

The stud.dat reference in s\3001s01.dat will look in these:
<lib>\parts\s  (parent folder of 3001s01.dat)
<mpd>     <-- hit
d:\
<lib>\p
<lib>\parts 


So in short the mpd content and parent folder of the file being loaded will be prefixed to ALL following search requests?

And every recursively loaded file will add to that by also prefixing their folder/mpd location?

Do note I make no distinction between .dat, .ldr, .mpd or if its located in parts / p etc, they are all just ldraw files being processed using a generic method.

Or are we suggesting that models have different kind of rules for resolving references then parts (which is even more messy imho).

so what would happen if you got a second mpd also containing a stud.dat which references the first mpd? 

e.g.

Code:

a.mpd
  main.ldr
     3001.dat
     b.mpd
  stud.dat

b.mpd
  main.ldr
     3001.dat
  stud.dat

If I understand the proposed system correctly it should use b.mpd's stud.dat for every brick used  in b.mpd's model's and it should use a.mpd's stud.dat for everything else.

You will end up with 2 different versions of <lib>parts\3001.dat this way.

The context approach wouldn't have this problem because every file on disk lives in it's own private container, which is very easy to cache as they never change, even when you load additional ldraw files.

In the context approach you would only end up with two different 3001.dat's if they have different ldraw sources much easier to maintain (especially in editors).

Hope this is somewhat understandable as I'm mainly trying to show we really should look beyond the single mpd impact of this 'global' system.

[Trevor Sandy]

The real question here is do we WANT a local stud.dat to affect the global scope or should it only affect its local context (like I assumed).

imho the local context approach is much more logical, global stuff should be handled by prefixing extra search locations to the global order.

Since LDraw.org does not author any applications why not define specifications in a manner that support both scenarios as they are both valid? This way, application developers can decide if to support either or both approaches and LDraw.org could focus on qualifying how unofficial file load/parse under each scope should be implemented, pros, cons etc...

Cheers,

[Travis Cobbs]

So to be clear, the way you expect it to work is like so:

given a mpd containing main.ldr, brick.dat and stud.dat  with a reference tree like so:

d:\model.mpd
  main.ldr
    3001.dat
       s\3001s01.dat
         stud.dat
   brick.dat

  brick.dat
    box.dat
    stud.dat

  stud.dat


The main.ldr reference will look in these
<mpd>    <--hit
d:\
<lib>\p
<lib>\parts

The 3001.dat reference will look in these:
<mpd>
d:\
<lib>\p
<lib>\parts    <-- hit

The s\3001s01.dat reference in 3001.dat will look in these:
<lib>\parts  (because its the parent folder of 3001.dat, just like d:\ is for the mpd)    <-- hit
<mpd>
d:\
<lib>\p 
<lib>\parts 

This last part is wrong. While logical, "current subfile directory" is never supposed to be in the part search path. In other words, 3001.dat will only be able to find s\3001s01.dat because <lib>\parts is in the standard search path. In the same way, s\3001s01.dat would not find a hypothetical s\3001s02.dat if it simply referred to it as 3001s02.dat. It is required to use s\3001s02.dat.

The stud.dat reference in s\3001s01.dat will look in these:
<lib>\parts\s  (parent folder of 3001s01.dat)
<mpd>     <-- hit
d:\
<lib>\p
<lib>\parts 

Edited.

So in short the mpd content and parent folder of the file being loaded will be prefixed to ALL following search requests?

No: the mpd content and the parent folder of the top-level file that the user opened will be prefixed to all search requests.

And every recursively loaded file will add to that by also prefixing their folder/mpd location?

As stated above, only the top-level folder is in the path. We should perhaps decide what rules should be followed if an mpd loads another mpd. I think a better solution to that may be to state that the behavior is undefined, and consider mpds loading other mpds to be invalid. (The whole point of mpds is to encapsulate a bunch of files into one, so having one load another defeats their very purpose, IMO.

Do note I make no distinction between .dat, .ldr, .mpd or if its located in parts / p etc, they are all just ldraw files being processed using a generic method.

Or are we suggesting that models have different kind of rules for resolving references then parts (which is even more messy imho).

From the standpoint of loading the files, all should be treated equal, no matter what they are or where they come from. Once the file is loaded, programs are free (and even encouraged) to detect parts and treat them differently. LDView does this (for seams), and I'm pretty sure LDCad does also.

so what would happen if you got a second mpd also containing a stud.dat which references the first mpd? 

e.g.

Code:

a.mpd
  main.ldr
     3001.dat
     b.mpd
  stud.dat

b.mpd
  main.ldr
     3001.dat
  stud.dat

If I understand the proposed system correctly it should use b.mpd's stud.dat for every brick used  in b.mpd's model's and it should use a.mpd's stud.dat for everything else.

You will end up with 2 different versions of <lib>parts\3001.dat this way.

The context approach wouldn't have this problem because every file on disk lives in it's own private container, which is very easy to cache as they never change, even when you load additional ldraw files.

In the context approach you would only end up with two different 3001.dat's if they have different ldraw sources much easier to maintain (especially in editors).

Hope this is somewhat understandable as I'm mainly trying to show we really should look beyond the single mpd impact of this 'global' system.

We can define what happens if one mpd reference another, but as I said above, I would prefer to state that it should be avoided at all costs and explicitly make the behavior undefined. I'm not entirely sure what LDView does in this situation right now. Either stud.dat from a.mpd will be used for the entire model, or stud.dat from b.mpd will be used for the entire model. (Probably a.mpd, but I'm not sure.) I'm pretty sure that it will never have two different files with the same "reference filename" in memory at one time, since my "loaded model" cache is keyed off the lower case version of that "reference filename". (Note: by "reference filename", I mean the filename listed in the type 1 line that caused the file to be loaded.)

What I'm not sure of is what it does when it encounters the second stud.dat. I think it just ignores it, but if it doesn't, it might in fact be broken.

[Travis Cobbs]

Since LDraw.org does not author any applications why not define specifications in a manner that support both scenarios as they are both valid? This way, application developers can decide if to support either or both approaches and LDraw.org could focus on qualifying how unofficial file load/parse under each scope should be implemented, pros, cons etc...

Ideally we want the spec to fully specify exactly what should happen when loading any LDraw file. Having two possible results, where the file looks completely different depending on which program loaded it is bad.

[Trevor Sandy]

Ideally we want the spec to fully specify exactly what should happen when loading any LDraw file.

Indeed. This is precisely what LDraw.org's specification should state - for each scenario.

Having two possible results, where the file looks completely different depending on which program loaded it is bad.

Isn't this the case at this moment ? I believe valid use cases should drive the specifications. Otherwise the specifications will simply not be followed and you will most certainly continue with bad.

If two scenarios are specified, then it would be wise for application developers to respect both - this is what preferences are for. Doing so will ensure consistency across applications.

Cheers,

[Orion Pobursky]

I believe that local files (either embedded within the mpd or in the local directory) overriding library files is the de facto standard and has been since, at least, the early days of MLCad. Any default behavior the doesn't follow that standard would, in my opinion, result in unexpected output. Of course, any reasonable program can choose to allow the user to override this default with the caveat that unintended output could result.

[Roland Melkert]

I believe that local files (either embedded within the mpd or in the local directory) overriding library files is the de facto standard and has been since, at least, the early days of MLCad. Any default behavior the doesn't follow that standard would, in my opinion, result in unexpected output. Of course, any reasonable program can choose to allow the user to override this default with the caveat that unintended output could result.

That's ok, but the thing I'm getting from this discussion is the feeling there are different rules for parts and model's.

Also the examples given are all single model ones.

O would like to define the whole spectrum of behaviour.

Including references to another mpd (sorry Travis I think this should be supported, e.g. a city made out of several houses all contained in their own mpd's)

For example what does mlcad do if you load a different model also containing a stud.dat, will all parts change or will they retain the previous version.

I'll have to do some additional research to get a better insight on the 'de-facto' way of things.

[Travis Cobbs]

That's ok, but the thing I'm getting from this discussion is the feeling there are different rules for parts and model's.

Also the examples given are all single model ones.

O would like to define the whole spectrum of behaviour.

Including references to another mpd (sorry Travis I think this should be supported, e.g. a city made out of several houses all contained in their own mpd's)

For example what does mlcad do if you load a different model also containing a stud.dat, will all parts change or will they retain the previous version.

I'll have to do some additional research to get a better insight on the 'de-facto' way of things.

I don't understand what you mean by different rules for parts and models. The rules are the same for all files, but the directory containing the top-level file is treated as being special (being the second entry in the path lookup right after mpd-contained files).

We can come up with rules for mpds inside of other mpds.

[Roland Melkert]

I don't understand what you mean by different rules for parts and models. The rules are the same for all files, but the directory containing the top-level file is treated as being special (being the second entry in the path lookup right after mpd-contained files).

I always assumed every ldraw file looks in its own mpd content and folder before diving into the search path (even files loaded from parts\s etc)

Then the above made me think this should only be done for models.

And now you write it should only be done for the top level file.

The one things made clear inside this thread: its about time we standardize this

That said it would be very complicated and messy, imho, to enforce the proposed global approach in an editor which works with multiple open models etc.

I, for example, would need to completely rewrite the caching system of LDCad (Something I'm not looking forward to).

[Orion Pobursky]

In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.

[Roland Melkert]

In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.

That's why I added the "Given <LDraw> is a 'library', meaning you could have multiples of those. " bit.

So it only follows that order within a self contained library.

This way you can use multiple libraries and custom files like e.g.:

<myMainLDrawFolder>
<unoff-202100326>\p
<unoff-202100326>\parts
<unoff-202100326>\models
<unoff-202100326>\Unofficial\p
<unoff-202100326>\Unofficial\parts
<lib-2003>\p
<lib-2003>\parts
<lib-2003>\models
<lib-2003>\Unofficial\p
<lib-2003>\Unofficial\parts

It doesn't matter if the unofficial zip doesn't have the model folder etc, it's just another library and a program should search within it using the given 5 folders (if they exist).

LDView already does this but only for a single Library.
As I understand it some others (LPub?) also use the, by LDView created, Unoffical p and parts folder (hence the request to standardize it).
LDCad is easily adjusted to add the models and Unoffical/* ones.

[Travis Cobbs]

In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.

What about if we standardize everything except the Unofficial, but also note that some programs then add the two Unofficial paths to the end of their search path.

[Roland Melkert]

What about if we standardize everything except the Unofficial, but also note that some programs then add the two Unofficial paths to the end of their search path.

So just add 'Models' to the current p and parts folders?

[Travis Cobbs]

So just add 'Models' to the current p and parts folders?

Yes. Specifically, this would be the official search path:

<Directory containing top-level file opened by user>
<LDraw>\p
<LDraw>\parts
<LDraw>\models

Then, note that some programs look for files in <LDraw>\Unofficial\p and <LDraw>\Unofficial\parts if they can't find them in the official locations.

[Roland Melkert]

Then, note that some programs look for files in <LDraw>\Unofficial\p and <LDraw>\Unofficial\parts if they can't find them in the official locations.

Sorry for the late reply, I've been very busy with work lately.

I still would like to put some time in analyzing the 'top level only' thing as used in current tools.

Because I really don't like what it suggests (library parts can look different when used in different models).

It will make multi file/model editing very hard to maintain.

It will also mess with your part inventory overviews, as the images of parts can differ per model while reality has only one version.

In short could we postpone voting on this, unless we limit it to just the folder locations and leave scoping/context out of it for the time being.

[Magnus Forsberg]

I remember an experiment I did way back when I started editing parts.

A default set-up of libraries have two folders, Parts and Unofficial/Parts.
A full dowload of all unofficial files will create a set of duplicated parts. A recycled partnumber will end up both in Parts and Unofficial Parts. 2 versions of the same partnumber.

To understand which file I was seeing in MLCad I edited the version in Unofficial. I gave it a different description and I coloured sections of it red. 
When I opened MLCad and generated a new part.lst and updated the library, I saw the old description from the official file in the Parts-folder, but the red geometry from the file in the Unofficial/Parts-folder. (... or vice versa. I can't remember now.)

It's a long time since I used MLCad, and a new version is released now, but I think this illustrate very well what could/will happen.

I have since then been removing old files from the official Parts-folder, and placed the updated version in Unofficial/Parts-folder.
Every official part release ends with me sitting at my desk and sorting all the files so that I don't have any duplicated partnumbers present.
I want to know that the geometry I see, in any editor, is the updated version.

[Travis Cobbs]

I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.

[Orion Pobursky]

I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.

It does. I modeled the search path dialog in LDDP after MLCad's

[Willy Tschager]

I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.

You're not reading my tutorials? Shame on you

http://www.holly-wood.it/mlcad/unofficial1-en.html

w.

[Philippe Hurbain]

You're not reading my tutorials? Shame on you

http://www.holly-wood.it/mlcad/unofficial1-en.html

w.

No, it's just that nobody uses it now 

[Travis Cobbs]

It does. I modeled the search path dialog in LDDP after MLCad's

Based on the tutorial in Willy's response below, it sounds like MLCad supports having the user add the Unofficial\p and Unofficial\parts directories to the search path, but does not have these by default. What am I missing?

[Orion Pobursky]

Based on the tutorial in Willy's response below, it sounds like MLCad supports having the user add the Unofficial\p and Unofficial\parts directories to the search path, but does not have these by default. What am I missing?

You're not missing anything. This is the same thing all the programs do since there's no defined path. That said, I stand by my opinion that we should not officially define an unofficial part path. It should always be explicitly user set.

[Willy Tschager]

Since non-LSB members can't comment on the LSB forum, this thread is for discussion of the search path proposal there.

To kick this off I'd like to comment on something Roland posted:

This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.

Could we please bring this to a happy end?

w.

[Orion Pobursky]

Could we please bring this to a happy end?

w.

I think we're waiting on Roland to do some investigating/testing

[Cam's Bricks]

I hate to bring up a topic that is 9 months old, but it seems that this never got a final conclusion. 

I would agree with Roland in this post here https://forums.ldraw.org/thread-24490-po...l#pid41951 that the flow should be that the subfiles pulled from the library should NOT be over written by the subfiles within the MPD itself. 

I have worked with a lot of cad programs and done a lot of modeling work, often using the basic structure concept of primitives in my own work making assemblies for manufacturing. Roland's approach is the most logical from a library perspective and it is how other cad packages handle similar circumstances. 

I have seen several references to the OMR. I think that is a moot point because I seem to recall that all the parts must be part from the official library to be OMR compliant. This would mean that the use case for non-standard "stud.dat" would never be an issue there. 

The issue comes up in testing new or altering existing parts. The new/altered parts ideally would be in a Development library that is referenced after the official and unofficial library locations.

This would be the ideal stack of libraries for a developer:

\p
\parts
\models
\Unofficial\p
\Unofficial\parts
\Development\p
\Development\parts

Then all programs would follow the order that the user chooses for the libraries. Then whichever is loaded LAST is ruling file. I say last because one of the leading CAD packages in the world, AutoCAD, has user loadable LISP files and if I have 4 files with a command call for "MAKEWALL", only the last loaded file that has the command "MAKEWALL" will work. 

Also in AutoCAD, anything loaded in an Xref (cross reference) is never affected by the current file. There is a secondary type of reference that is called a Block. And Xref is always loaded from an outside source and a Block is local to the drawing. I can have the same block referenced in multiple files then Xref them into the current file. If the block definition is defined differently in the current file, it will not affect any Xref versions of the same block unless I go to each one and update it. Its not quite the same but the general concept is the same in that parts that are referenced into the current file should retain their reference paths but for new subfiles created within the current file, they should look to the MPD before the library. 

Hopefully this rambling makes some sense.

[N. W. Perry]

To offer my point of view strictly as a user, without any development expertise:

I think the behavior I'd expect is the same as what Roland describes: a subfile embedded within (or placed alongside) a referencing file should be limited to the scope of the referencing file, at whatever recursive level that is. This to me is compatible with the problem statement:

Local files should always override library parts.

…with the added understanding that a "local" (sub)file is indeed local to its containing file. Conversely, local files should not override library parts in contexts to which they're not local.


All that being said, as a user I don't think I'd want to (inadvertently) create a situation where I had same-named subfiles existing at different levels of the tree, and I think I'd want the program to notify me if I did. At that point, maybe I'd want to be given the option to rename the local subfile, or to choose whether it should apply globally or locally.

The question of whether a non-official update to an official file should be given precedence is, I think, not really in dispute and in any case should be user configurable. It only becomes relevant because, if there is way to globally replace a subfile by changing the search path order, and a way to locally replace it by embedding it at a local level, then it seems to me that both use cases are accommodated and all is well. (You could even have both situations occurring at once, with a subfile globally replaced by an unofficial update, and with yet a third version appearing in specific parts—though again, as a user, I would tend not to want to work that way.)

[Travis Cobbs]

I'm writing this as a reply to the original post instead of replying to one of the sub-posts. What I say will probably refer to multiple threads. Based on everything written here, I would like to make the following suggestion for the official LDraw search path:

1. 0 FILE references in the currently active MPD. Note: an MPD in this context is any LDraw file that contains 0 FILE lines. "Currently active" in this context means that the files are only visible inside the MPD, and are not visible to any other files.
   
2. The directory containing the top-level model opened by the user. (Note: I think this dates to ldraw.exe. We can decide that it's not a good idea, but I don't personally think we should do that.)
   
3. <LDraw Directory>\p
   
4. <LDraw Directory>\parts
   
5. <LDraw Directory>\models
   

I could see adding an entry either before or after 2 that is the directory containing the current LDraw file.

The first hit found when searching for files in the above order would be used. 

Additionally, it should be noted that many tools add the following two entries to the end of their search path, but they are not part of the LDraw standard:

1. <LDraw Directory>\Unofficial\p
   
2. <LDraw Directory>\Unofficial\parts
   

Just as a note, to the best of my knowledge, LDView was the first program to support the Unofficial directory, and it did so as part of its feature that automatically downloads unofficial files from the LDraw Parts Tracker. Downloading unofficial parts into the main LDraw directories was clearly a bad idea, and I chose "Unofficial" as a directory to hold them. As far as I know, no other programs automatically download files into the Unofficial directory, but various other programs include it in their search path (presumably due to LDView's auto-download feature).

[Roland Melkert]

I'm writing this as a reply to the original post instead of replying to one of the sub-posts. What I say will probably refer to multiple threads. Based on everything written here, I would like to make the following suggestion for the official LDraw search path:

1. 0 FILE references in the currently active MPD. Note: an MPD in this context is any LDraw file that contains 0 FILE lines. "Currently active" in this context means that the files are only visible inside the MPD, and are not visible to any other files.
   
2. The directory containing the top-level model opened by the user. (Note: I think this dates to ldraw.exe. We can decide that it's not a good idea, but I don't personally think we should do that.)
   
3. <LDraw Directory>\p
   
4. <LDraw Directory>\parts
   
5. <LDraw Directory>\models
   

I could see adding an entry either before or after 2 that is the directory containing the current LDraw file.

The first hit found when searching for files in the above order would be used. 

Additionally, it should be noted that many tools add the following two entries to the end of their search path, but they are not part of the LDraw standard:

1. <LDraw Directory>\Unofficial\p
   
2. <LDraw Directory>\Unofficial\parts
   

Just as a note, to the best of my knowledge, LDView was the first program to support the Unofficial directory, and it did so as part of its feature that automatically downloads unofficial files from the LDraw Parts Tracker. Downloading unofficial parts into the main LDraw directories was clearly a bad idea, and I chose "Unofficial" as a directory to hold them. As far as I know, no other programs automatically download files into the Unofficial directory, but various other programs include it in their search path (presumably due to LDView's auto-download feature).

I still think #2 should be changed into something like "The directory containing the current file being processed."

I understand the ldraw.exe legacy but that program assumed simple self contained models (basically an mpd on disk).

The format as grown beyond that imho.

Enforcing the rule today will cause problems loading recursive multi folder models.

For example you open model "a.ldr" containing references to "subFolder\b.ldr"

Now when b.ldr contains a line referencing "c.ldr" the original #2 rule would dictate it looks for c.ldr inside the folder containing a.ldr.

It will not be found (unless there is another c.ldr there) so the loader goes on to 'p', 'parts' etc probably never finding it.

My suggestion would make it look inside the folder 'b.ldr' sits in which is, imho, much more modular and makes it a fully recursive system which is easier to write loaders for. (most current loaders probably already do it this way).

The original rule would mean you need to reference c.ldr like "subFolder\c.ldr" inside b.ldr which will break the model when loading just b.ldr

It would also create scooping/instances nightmare as parts/models can appear different depending from where they are referenced.


I have no objection to adding the unofficial folders though.

[Travis Cobbs]

I still think #2 should be changed into something like "The directory containing the current file being processed."

I understand the ldraw.exe legacy but that program assumed simple self contained models (basically an mpd on disk).

The format as grown beyond that imho.

Enforcing the rule today will cause problems loading recursive multi folder models.

For example you open model "a.ldr" containing references to "subFolder\b.ldr"

Now when b.ldr contains a line referencing "c.ldr" the original #2 rule would dictate it looks for c.ldr inside the folder containing a.ldr.

It will not be found (unless there is another c.ldr there) so the loader goes on to 'p', 'parts' etc probably never finding it.

My suggestion would make it look inside the folder 'b.ldr' sits in which is, imho, much more modular and makes it a fully recursive system which is easier to write loaders for. (most current loaders probably already do it this way).

The original rule would mean you need to reference c.ldr like "subFolder\c.ldr" inside b.ldr which will break the model when loading just b.ldr

It would also create scooping/instances nightmare as parts/models can appear different depending from where they are referenced.


I have no objection to adding the unofficial folders though.

I'm dubious that multi-folder models exist, but I will concede your point. Having said that, the top-level model folder still makes sense if you think of that folder as the current working directory for the whole model. Adding the current file's directory to the search path is reasonable, but I don't think it should replace the top-level model's directory. What about this:

1. 0 FILE references in the currently active MPD. Note: an MPD in this context is any LDraw file that contains 0 FILE lines. "Currently active" in this context means that the files are only visible inside the MPD, and are not visible to any other files.
   
2. The directory of the currently active file.
   
3. The directory containing the top-level model opened by the user.
   
4. <LDraw Directory>\p
   
5. <LDraw Directory>\parts
   
6. <LDraw Directory>\models
   

[Roland Melkert]

Having said that, the top-level model folder still makes sense if you think of that folder as the current working directory for the whole model. Adding the current file's directory to the search path is reasonable, but I don't think it should replace the top-level model's directory. What about this:

What would be the advantage of that?

I can only see it causing potential problems, expanding on my above example.

If the b.ldr model want to use something from the higher folder it should reference it using "..\d.ldr" otherwise it will (again) fail to load when opening just b.ldr.

And also it will complicate things for programs working with multiple open models at once.

Because you will need to keep track of how a model was opened.

It will cause two different 'b.ldr' instances when a user first opens b.ldr standalone and later also opens 'a.ldr' in the same program.

The first b.ldr will indicate missing references and the second one will display fully keeping track of this will need lots of house keeping without gaining, imho, any advantage over not including the working folder at all.

In short every model should live in it's own little universe fully isolated from anything else loaded.

This way you can work with multiple open models without the scoping problems a 'working folder' would cause.

Also when will it even be needed?

You say you think there are no real multi folder models out there, which would mean the <current file folder> and <working folder> would be the same for almost all models (at least in your experience).

So what would be the point of adding it to the search order causing all these headache's for developers

[Travis Cobbs]

What would be the advantage of that?

The advantage would be that this is that this is how it's always been (in ldraw.exe, ledit.exe, LDView, l3p, and probably other LDraw programs). Obviously, since there has been no official documentation of the search path, this behavior might not be widely known. But I don't think that's an excuse not to include it in the official search path now that we are documenting it.

Lars C. Hassing's LDrawIni code (which LDView and l3p for sure use, and possibly other programs) have always used it as the first entry in the default search path (ignoring MPD subfiles which have to be dealt with separately).

[Roland Melkert]

Lars C. Hassing's LDrawIni code (which LDView and l3p for sure use, and possibly other programs) have always used it as the first entry in the default search path (ignoring MPD subfiles which have to be dealt with separately).

When you are working with a single folder this is really only a naming thing.

So maybe Lars wrote it with only that usage in mind.

Can we get Lars into this discussion?  I really doubt he purposely wanted to restrict the format this way?

Also standardizing the search order is the perfect opportunity to address this (ihmo) problem/bug so it won't be an issue of confusion again.

I made a small test model demonstrating the problem.

LDView an MLCad both can't find roof.ldr but I really think it should load ok using the (new) standard search path rules.

And again this change won't break the single folder approach everyone (else) is using, it only introduces recursion to the loading process.

And it also handles mpd references because the recursion will automatically support that.

So the spec would be:

<current file's MPD content>
<current file's folder content>
<library>/p
<library>/parts
<library>/models
<library>/Unofficial/p
<library>/Unofficial/parts

And a loader just needs to push/pop the 'current ...' items while walking the model tree.

And if your program only supports the existing 'working folder' you don't have to change a thing, all (new) single folder models will still load.

Sorry if I appear to be stubborn about this, but I truly believe it will hurt the format if we keep the top level folder approach.

Imho it will ruin the (potential) modular/recursive nature of the format. (especially for software working with multiple open models).

[Travis Cobbs]

I indicated that I was willing to add the current directory of the current file to the search path right after MPD. I just want to also keep the other entry there for historical reasons (as the next entry). Your file would load the same with my suggested search path and yours, since I put your suggested entry before the top-level model directory entry.

[Roland Melkert]

I indicated that I was willing to add the current directory of the current file to the search path right after MPD. I just want to also keep the other entry there for historical reasons (as the next entry). Your file would load the same with my suggested search path and yours, since I put your suggested entry before the top-level model directory entry.

Yes, but I think it is not needed. Because the proposed order without recursion already describes the current  usage.

Adding it will only make things more difficult for any program working with multiple open models while it isn't even used in real-life.

The alternative for those programs would be to just ignore it (not loosing anything by doing so), but technically you wouldn't be following the spec any more.

Anyone has had recent contact with Lars? I would really like to know his opinion on this.

[Orion Pobursky]

So the spec would be:

<current file's MPD content>
<current file's folder content>
<library>/p
<library>/parts
<library>/models
<library>/Unofficial/p
<library>/Unofficial/parts

Recursion debate aside, I do not want to have the "Unofficial" part paths officially supported. The precedence of this path should always be up to the user.

[Max Martin Richter]

The precedence of this path should always be up to the user.

+1

as a reviewer I have the unoffical parts sometimes above the official ones. So the order should be just a template that can be overwritten by the (experienced) user.

For the normal user I'm fine with this order.

[Roland Melkert]

Recursion debate aside, I do not want to have the "Unofficial" part paths officially supported. The precedence of this path should always be up to the user.

I don't mind ether way.

But a user could always just leave those folders empty and just use a second library or additional  search folders depending on the program used.

[Travis Cobbs]

Can we get Lars into this discussion?  I really doubt he purposely wanted to restrict the format this way?

I contacted Lars. He's having LDraw forum password issues, but hopefully will reply when he gets them resolved.

[Orion Pobursky]

I contacted Lars. He's having LDraw forum password issues, but hopefully will reply when he gets them resolved.

I confirm he's in contact with me and I'm working to resolve the issue.

[Lars C. Hassing]

Hi guys,
it's been a while, I got involved in other projects in 2008
and haven't really followed LDraw since then,
though I did spend a lot of time with LDraw 1998-2008.


l3p was released in August 1998.
For October the change log says:

  1998.10.04 lch File search also in directory of model
  1998.10.05 lch Support for MPD files

so these two features have existed from the very early days.


l3p (and L3Lab) treats MPD subfiles as public/global, which I think was the original idea.

If an .mpd file with "0 FILE xxx.dat" is met and xxx.dat has already been seen and loaded,
you will get a warning and the second xxx.dat is skipped.
The first xxx.dat can be referenced from anywhere.

l3p is of course tightly coupled to the POV-Ray data model.

If it were to support MPD subfiles as local to the MPD and its children,
it would require adding some extra naming convention 
in stead of the straight forward _xxx_dot_dat.


l3p is a simpler app handling all loading in one run.
L3Lab uses caching, so parts loaded for the first model file are kept
readily available for reference by a second model file.
However, if the model file was an MPD file, its subfiles are wiped before loading the second model file.

In the source code I found this:

   /* Prepare for the day MPD is allowed in P and PARTS so this MPD subfile doesn't get cleared in FreeModel() */

which means that subfiles of an MPD file loaded from P and PARTS are not wiped.
Note however they are publicly available.


I never saw any MPD files in P and PARTS,
and don't know if they exist today.
l3p and L3Lab cannot cope with two MPD files in PARTS both having the same "0 FILE xxx.dat" line.

But, MPD files in PARTS could solve the namespace problem themselves using prefixing:

   PARTS\3001.mpd:  0 FILE 3001_xxx.dat
   PARTS\3002.mpd:  0 FILE 3002_xxx.dat

3001_xxx.dat and 3002_xxx.dat would still be public, but who cares/realizes ?

l3p would continue to generate correct POV-Ray code.


I also found this comment:

      /* Hmm, MPD scoping says that MPD subfiles are public. However,
        L3xx allows nested MPDs which makes scoping somewhat unclear.
        Right above you are warned if the same filename appeared in
        another MPD (or elsewhere). MPD subfiles of a part or a
        primitive should be kept private...
        -and not cleared in FreeModel()                                                  */

which suggests that I had some thoughts about scoping back then
(for technical reasons/conflicts from a programmers perspective, not about the logical application/usage)
but at that time it was never an issue.
/Lars

[Roland Melkert]

Hi guys,
it's been a while, I got involved in other projects in 2008
and haven't really followed LDraw since then,
though I did spend a lot of time with LDraw 1998-2008.

Welcome back.

...........
If an .mpd file with "0 FILE xxx.dat" is met and xxx.dat has already been seen and loaded,
you will get a warning and the second xxx.dat is skipped.
The first xxx.dat can be referenced from anywhere.
...........
l3p and L3Lab cannot cope with two MPD files in PARTS both having the same "0 FILE xxx.dat" line.
...........
/* Hmm, MPD scoping says that MPD subfiles are public. However,
L3xx allows nested MPDs which makes scoping somewhat unclear.
Right above you are warned if the same filename appeared in
another MPD (or elsewhere). MPD subfiles of a part or a
primitive should be kept private...
-and not cleared in FreeModel() */

which suggests that I had some thoughts about scoping back then
(for technical reasons/conflicts from a programmers perspective, not about the logical application/usage)
but at that time it was never an issue.

So basically everything loaded is public/global?

This is fine when loading a single model tree with the purpose to generate pov code.

But I really don't think it should become the standard.

Strict local scoping ensuring every ldraw file (mpd or plain) lives in a closed container so it only has to be loaded once and always looks the same (except for colors) no matter what file references it.

Otherwise working with multiple models/files in a single program will become an instance administrative nightmare.

[Travis Cobbs]

So basically everything loaded is public/global?

This is fine when loading a single model tree with the purpose to generate pov code.

But I really don't think it should become the standard.

Strict local scoping ensuring every ldraw file (mpd or plain) lives in a closed container so it only has to be loaded once and always looks the same (except for colors) no matter what file references it.

Otherwise working with multiple models/files in a single program will become an instance administrative nightmare.

It's my opinion that the top level model's directory has always been searched by various LDraw programs since day one (although apparently not LDCad), and because of this we should maintain this behavior in the standard. However, it doesn't seem like we're going to agree on that, so I'd like to propose that we ask Orion's opinion (since he's the third vote), and we go with whatever he recommends.

Note: I'm fine with adding the local directory of the currently processed model to the search path.

[Roland Melkert]

so I'd like to propose that we ask Orion's opinion (since he's the third vote), and we go with whatever he recommends.

I guess that would be a solution to the deadlock

But (again) I want to make clear the only issue I'm having here, is we are (imho) in danger of standardizing something that only make things more difficult while never or hardly been used in practice.

Just because "It's always been there" is not a good enough reason for me.

Most people just trow all files in a single folder where the whole issue doesn't matter, because local==model folder.

Things get messy when subfolder models start referencing the higher ones without using "../". But I doubt people even use that.

Keeping the top level model folder in the (temporary) search list influences the context of all recursively loaded files. This means if another model uses the same submodel it (in theory, but I suspect never for real) might be (non col16) visibly different and will need instance management.

Hence my reluctance to adding this to the standard. Because it would force developers to support something (potentially extremely complicated) that is never used and easily fixable in the few cases that do need it.

One could fix it by referencing using the parent folder(s) or by simply adding a global (model) search folder to program's settings.

Both would prevent the whole messy context dependency thing and help  keep the standard clean and easy to implement.