Search path discussion


Search path discussion
#1
Since non-LSB members can't comment on the LSB forum, this thread is for discussion of the search path proposal there.

To kick this off I'd like to comment on something Roland posted:
Quote:I disagree on this, a stud.dat should not replace all studs in the whole scene if it's located in the mpd itself or its folder.

This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.
Reply
RE: Search path discussion
#2
(2021-03-16, 13:26)Orion Pobursky Wrote: This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.
Fully agree. Embedded parts should have precedence so a model using unofficial parts won't be broken inadvertantly. I remember a time when embedding parts was little used and it was a struggle to find again the unofficial parts used by author.
Reply
RE: Search path discussion
#3
(2021-03-16, 14:48)Philippe Hurbain Wrote: Fully agree. Embedded parts should have precedence so a model using unofficial parts won't be broken inadvertantly. I remember a time when embedding parts was little used and it was a struggle to find again the unofficial parts used by author.

I don't understand. If the parts are embedded in the MPD they will be loaded instead of the library ones.

But only withing the context of the model.

So if you put a stud.dat inside the mpd with e.g. a yellow 4-4disc.dat only the embedded files actually referencing stud.dat will use that stud variant.

All the other non embedded files will use the normal stud.dat as found in <lib>\p

Like so:
   

If I understand you correctly you would want all studs to be yellow in that image ?

LDView seems to ignore it and use the library one for everything.
   


mpd code:
Code:
0 FILE main.ldr
1 7 0 0 0 -1 0 0 0 1 0 0 0 -1 819.dat
1 10 40 -24 -30 -1 0 0 0 1 0 0 0 -1 3065.dat
1 10 0 -24 -30 -1 0 0 0 1 0 0 0 -1 3065.dat
1 10 -30 -24 -20 0 0 -1 0 1 0 1 0 0 3065.dat
1 10 -40 -48 20 -1 0 0 0 1 0 0 0 -1 brick.dat

0 FILE brick.dat
0 UNOFFICIAL PART
0 BFC CERTIFY CCW
1 16 -10 0 0 1 0 0 0 1 0 0 0 1 stud.dat
1 16 10 0 0 1 0 0 0 1 0 0 0 1 stud.dat
1 16 0 24 0 20 0 0 0 24 0 0 0 20 box.dat

0 FILE stud.dat
0 UNOFFICIAL PART
0 BFC CERTIFY CCW
1 16 0 0 0 6 0 0 0 1 0 0 0 6 4-4edge.dat
1 16 0 -4 0 6 0 0 0 1 0 0 0 6 4-4edge.dat
1 16 0 0 0 6 0 0 0 -4 0 0 0 6 4-4cyli.dat
1 14 0 -4 0 6 0 0 0 1 0 0 0 6 4-4disc.dat
0 1 16 0 -4 0 1 0 0 0 1 0 0 0 1 logo3.dat
Reply
RE: Search path discussion
#4
That's not the way it's supposed to work. If stud.dat is embedded in the mpd or located in the same directory as the model file then it should replace all instances of stud.dat for the entire model. This allows, say, a part author to test out a fix on a bunch of different parts simultaneously or a model author to include a custom version of some commonly file (e.g. a stud with a custom logo).
Reply
RE: Search path discussion
#5
(2021-03-16, 19:35)Orion Pobursky Wrote: That's not the way it's supposed to work. If stud.dat is embedded in the mpd or located in the same directory as the model file then it should replace all instances of stud.dat for the entire model. This allows, say, a part author to test out a fix on a bunch of different parts simultaneously or a model author to include a custom version of some commonly file (e.g. a stud with a custom logo).
You could do that by adding your custom/development library paths in front of the official ones.

The main thing I'm disliking about the global scope thing is the fact an embeded file which happens to have the same name as some official part would break all dependencies.

The 'context' method will only affect its local scope so there will never be problems with duplicate names, it's comparable with local variables in a programming language.
Reply
RE: Search path discussion
#7
(2021-03-16, 19:43)Roland Melkert Wrote: You could do that by adding your custom/development library paths in front of the official ones.

The main thing I'm disliking about the global scope thing is the fact an embeded file which happens to have the same name as some official part would break all dependencies.

The 'context' method will only affect its local scope so there will never be problems with duplicate names, it's comparable with local variables in a programming language.

At this point in time, I think it's safe to say that if a user embeds a *.dat file in their MPD, they intend it to replace the *.dat file of that same name in the LDraw parts library. Nobody should be using *.dat for personal files anymore. So I still feel that this global replacement is correct.

Note that putting files alongside the main model to do the same thing is a leftover from the days before MPDs. However, I still think that it is consistent to continue with this behavior.
Reply
RE: Search path discussion
#8
(2021-03-16, 22:11)Travis Cobbs Wrote: At this point in time, I think it's safe to say that if a user embeds a *.dat file in their MPD, they intend it to replace the *.dat file of that same name in the LDraw parts library. Nobody should be using *.dat for personal files anymore. So I still feel that this global replacement is correct.

So to be clear, the way you expect it to work is like so:

given a mpd containing main.ldr, brick.dat and stud.dat  with a reference tree like so:

d:\model.mpd
  main.ldr
    3001.dat
       s\3001s01.dat
         stud.dat
   brick.dat

  brick.dat
    box.dat
    stud.dat

  stud.dat


The main.ldr reference will look in these
<mpd>    <--hit
d:\
<lib>\p
<lib>\parts

The 3001.dat reference will look in these:
<mpd>
d:\
<lib>\p
<lib>\parts    <-- hit

The s\3001s01.dat reference in 3001.dat will look in these:
<lib>\parts  (because its the parent folder of 3001.dat, just like d:\ is for the mpd)    <-- hit
<mpd>
d:\
<lib>\p 
<lib>\parts 

The stud.dat reference in s\3001s01.dat will look in these:
<lib>\parts\s  (parent folder of 3001s01.dat)
<mpd>     <-- hit
d:\
<lib>\p
<lib>\parts 


So in short the mpd content and parent folder of the file being loaded will be prefixed to ALL following search requests?

And every recursively loaded file will add to that by also prefixing their folder/mpd location?

Do note I make no distinction between .dat, .ldr, .mpd or if its located in parts / p etc, they are all just ldraw files being processed using a generic method.

Or are we suggesting that models have different kind of rules for resolving references then parts (which is even more messy imho).

so what would happen if you got a second mpd also containing a stud.dat which references the first mpd? 

e.g.
Code:
a.mpd
  main.ldr
     3001.dat
     b.mpd
  stud.dat

b.mpd
  main.ldr
     3001.dat
  stud.dat


If I understand the proposed system correctly it should use b.mpd's stud.dat for every brick used  in b.mpd's model's and it should use a.mpd's stud.dat for everything else.

You will end up with 2 different versions of <lib>parts\3001.dat this way.

The context approach wouldn't have this problem because every file on disk lives in it's own private container, which is very easy to cache as they never change, even when you load additional ldraw files.

In the context approach you would only end up with two different 3001.dat's if they have different ldraw sources much easier to maintain (especially in editors).

Hope this is somewhat understandable as I'm mainly trying to show we really should look beyond the single mpd impact of this 'global' system.
Reply
RE: Search path discussion
#10
(2021-03-16, 23:43)Roland Melkert Wrote: So to be clear, the way you expect it to work is like so:

given a mpd containing main.ldr, brick.dat and stud.dat  with a reference tree like so:

d:\model.mpd
  main.ldr
    3001.dat
       s\3001s01.dat
         stud.dat
   brick.dat

  brick.dat
    box.dat
    stud.dat

  stud.dat


The main.ldr reference will look in these
<mpd>    <--hit
d:\
<lib>\p
<lib>\parts

The 3001.dat reference will look in these:
<mpd>
d:\
<lib>\p
<lib>\parts    <-- hit

The s\3001s01.dat reference in 3001.dat will look in these:
<lib>\parts  (because its the parent folder of 3001.dat, just like d:\ is for the mpd)    <-- hit
<mpd>
d:\
<lib>\p 
<lib>\parts 

This last part is wrong. While logical, "current subfile directory" is never supposed to be in the part search path. In other words, 3001.dat will only be able to find s\3001s01.dat because <lib>\parts is in the standard search path. In the same way, s\3001s01.dat would not find a hypothetical s\3001s02.dat if it simply referred to it as 3001s02.dat. It is required to use s\3001s02.dat.


(2021-03-16, 23:43)Roland Melkert Wrote: The stud.dat reference in s\3001s01.dat will look in these:
<lib>\parts\s  (parent folder of 3001s01.dat)
<mpd>     <-- hit
d:\
<lib>\p
<lib>\parts 

Edited.

(2021-03-16, 23:43)Roland Melkert Wrote: So in short the mpd content and parent folder of the file being loaded will be prefixed to ALL following search requests?

No: the mpd content and the parent folder of the top-level file that the user opened will be prefixed to all search requests.


(2021-03-16, 23:43)Roland Melkert Wrote: And every recursively loaded file will add to that by also prefixing their folder/mpd location?

As stated above, only the top-level folder is in the path. We should perhaps decide what rules should be followed if an mpd loads another mpd. I think a better solution to that may be to state that the behavior is undefined, and consider mpds loading other mpds to be invalid. (The whole point of mpds is to encapsulate a bunch of files into one, so having one load another defeats their very purpose, IMO.


(2021-03-16, 23:43)Roland Melkert Wrote: Do note I make no distinction between .dat, .ldr, .mpd or if its located in parts / p etc, they are all just ldraw files being processed using a generic method.

Or are we suggesting that models have different kind of rules for resolving references then parts (which is even more messy imho).

From the standpoint of loading the files, all should be treated equal, no matter what they are or where they come from. Once the file is loaded, programs are free (and even encouraged) to detect parts and treat them differently. LDView does this (for seams), and I'm pretty sure LDCad does also.


(2021-03-16, 23:43)Roland Melkert Wrote: so what would happen if you got a second mpd also containing a stud.dat which references the first mpd? 

e.g.
Code:
a.mpd
  main.ldr
     3001.dat
     b.mpd
  stud.dat

b.mpd
  main.ldr
     3001.dat
  stud.dat


If I understand the proposed system correctly it should use b.mpd's stud.dat for every brick used  in b.mpd's model's and it should use a.mpd's stud.dat for everything else.

You will end up with 2 different versions of <lib>parts\3001.dat this way.

The context approach wouldn't have this problem because every file on disk lives in it's own private container, which is very easy to cache as they never change, even when you load additional ldraw files.

In the context approach you would only end up with two different 3001.dat's if they have different ldraw sources much easier to maintain (especially in editors).

Hope this is somewhat understandable as I'm mainly trying to show we really should look beyond the single mpd impact of this 'global' system.

We can define what happens if one mpd reference another, but as I said above, I would prefer to state that it should be avoided at all costs and explicitly make the behavior undefined. I'm not entirely sure what LDView does in this situation right now. Either stud.dat from a.mpd will be used for the entire model, or stud.dat from b.mpd will be used for the entire model. (Probably a.mpd, but I'm not sure.) I'm pretty sure that it will never have two different files with the same "reference filename" in memory at one time, since my "loaded model" cache is keyed off the lower case version of that "reference filename". (Note: by "reference filename", I mean the filename listed in the type 1 line that caused the file to be loaded.)

What I'm not sure of is what it does when it encounters the second stud.dat. I think it just ignores it, but if it doesn't, it might in fact be broken.
Reply
RE: Search path discussion
#6
The only reason you saw what you did in LDView is because you had primitive substitution enabled (which is the only way to see stud logos in LDView). I didn't mention it in the search path discussion, but LDView's primitive substitution trumps the search path when it is enabled. If LDView recognizes a file to be a primitive when primitive substitution is enabled, it never even bothers to look at the file system.
Reply
RE: Search path discussion
#9
(2021-03-15, 20:05)Roland Melkert Wrote: The real question here is do we WANT a local stud.dat to affect the global scope or should it only affect its local context (like I assumed).

imho the local context approach is much more logical, global stuff should be handled by prefixing extra search locations to the global order.

Since LDraw.org does not author any applications why not define specifications in a manner that support both scenarios as they are both valid? This way, application developers can decide if to support either or both approaches and LDraw.org could focus on qualifying how unofficial file load/parse under each scope should be implemented, pros, cons etc...

Cheers,
Reply
RE: Search path discussion
#11
(2021-03-17, 1:54)Trevor Sandy Wrote: Since LDraw.org does not author any applications why not define specifications in a manner that support both scenarios as they are both valid? This way, application developers can decide if to support either or both approaches and LDraw.org could focus on qualifying how unofficial file load/parse under each scope should be implemented, pros, cons etc...

Ideally we want the spec to fully specify exactly what should happen when loading any LDraw file. Having two possible results, where the file looks completely different depending on which program loaded it is bad.
Reply
RE: Search path discussion
#12
(2021-03-17, 2:01)Travis Cobbs Wrote: Ideally we want the spec to fully specify exactly what should happen when loading any LDraw file.

Indeed. This is precisely what LDraw.org's specification should state - for each scenario.

(2021-03-17, 2:01)Travis Cobbs Wrote: Having two possible results, where the file looks completely different depending on which program loaded it is bad.

Isn't this the case at this moment ? I believe valid use cases should drive the specifications. Otherwise the specifications will simply not be followed and you will most certainly continue with bad.

If two scenarios are specified, then it would be wise for application developers to respect both - this is what preferences are for. Doing so will ensure consistency across applications.

Cheers,
Reply
RE: Search path discussion
#13
I believe that local files (either embedded within the mpd or in the local directory) overriding library files is the de facto standard and has been since, at least, the early days of MLCad. Any default behavior the doesn't follow that standard would, in my opinion, result in unexpected output. Of course, any reasonable program can choose to allow the user to override this default with the caveat that unintended output could result.
Reply
RE: Search path discussion
#14
(2021-03-17, 4:24)Orion Pobursky Wrote: I believe that local files (either embedded within the mpd or in the local directory) overriding library files is the de facto standard and has been since, at least, the early days of MLCad. Any default behavior the doesn't follow that standard would, in my opinion, result in unexpected output. Of course, any reasonable program can choose to allow the user to override this default with the caveat that unintended output could result.
That's ok, but the thing I'm getting from this discussion is the feeling there are different rules for parts and model's.

Also the examples given are all single model ones.

O would like to define the whole spectrum of behaviour.

Including references to another mpd (sorry Travis I think this should be supported, e.g. a city made out of several houses all contained in their own mpd's)

For example what does mlcad do if you load a different model also containing a stud.dat, will all parts change or will they retain the previous version.

I'll have to do some additional research to get a better insight on the 'de-facto' way of things.
Reply
RE: Search path discussion
#15
(2021-03-17, 7:08)Roland Melkert Wrote: That's ok, but the thing I'm getting from this discussion is the feeling there are different rules for parts and model's.

Also the examples given are all single model ones.

O would like to define the whole spectrum of behaviour.

Including references to another mpd (sorry Travis I think this should be supported, e.g. a city made out of several houses all contained in their own mpd's)

For example what does mlcad do if you load a different model also containing a stud.dat, will all parts change or will they retain the previous version.

I'll have to do some additional research to get a better insight on the 'de-facto' way of things.

I don't understand what you mean by different rules for parts and models. The rules are the same for all files, but the directory containing the top-level file is treated as being special (being the second entry in the path lookup right after mpd-contained files).

We can come up with rules for mpds inside of other mpds.
Reply
RE: Search path discussion
#16
(2021-03-17, 19:27)Travis Cobbs Wrote: I don't understand what you mean by different rules for parts and models. The rules are the same for all files, but the directory containing the top-level file is treated as being special (being the second entry in the path lookup right after mpd-contained files).

I always assumed every ldraw file looks in its own mpd content and folder before diving into the search path (even files loaded from parts\s etc)

Then the above made me think this should only be done for models.

And now you write it should only be done for the top level file.

The one things made clear inside this thread: its about time we standardize this Big Grin

That said it would be very complicated and messy, imho, to enforce the proposed global approach in an editor which works with multiple open models etc.

I, for example, would need to completely rewrite the caching system of LDCad (Something I'm not looking forward to).
Reply
RE: Search path discussion
#17
In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.
Reply
RE: Search path discussion
#18
(2021-03-25, 23:05)Orion Pobursky Wrote: In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.

That's why I added the "Given <LDraw> is a 'library', meaning you could have multiples of those. " bit.

So it only follows that order within a self contained library.

This way you can use multiple libraries and custom files like e.g.:

<myMainLDrawFolder>
<unoff-202100326>\p
<unoff-202100326>\parts
<unoff-202100326>\models
<unoff-202100326>\Unofficial\p
<unoff-202100326>\Unofficial\parts
<lib-2003>\p
<lib-2003>\parts
<lib-2003>\models
<lib-2003>\Unofficial\p
<lib-2003>\Unofficial\parts

It doesn't matter if the unofficial zip doesn't have the model folder etc, it's just another library and a program should search within it using the given 5 folders (if they exist).

LDView already does this but only for a single Library.
As I understand it some others (LPub?) also use the, by LDView created, Unoffical p and parts folder (hence the request to standardize it).
LDCad is easily adjusted to add the models and Unoffical/* ones.
Reply
RE: Search path discussion
#19
(2021-03-25, 23:05)Orion Pobursky Wrote: In reply to the latest comment by Roland in the LSB forum:

I reallly don't think we should define an "unofficial" path. Instead of unofficial it could be "user defined paths" or simply left off.

What about if we standardize everything except the Unofficial, but also note that some programs then add the two Unofficial paths to the end of their search path.
Reply
RE: Search path discussion
#20
(2021-03-27, 1:54)Travis Cobbs Wrote: What about if we standardize everything except the Unofficial, but also note that some programs then add the two Unofficial paths to the end of their search path.

So just add 'Models' to the current p and parts folders?
Reply
RE: Search path discussion
#21
(2021-03-29, 22:04)Roland Melkert Wrote: So just add 'Models' to the current p and parts folders?

Yes. Specifically, this would be the official search path:

<Directory containing top-level file opened by user>
<LDraw>\p
<LDraw>\parts
<LDraw>\models

Then, note that some programs look for files in <LDraw>\Unofficial\p and <LDraw>\Unofficial\parts if they can't find them in the official locations.
Reply
RE: Search path discussion
#22
(2021-03-30, 1:06)Travis Cobbs Wrote: Then, note that some programs look for files in <LDraw>\Unofficial\p and <LDraw>\Unofficial\parts if they can't find them in the official locations.

Sorry for the late reply, I've been very busy with work lately.

I still would like to put some time in analyzing the 'top level only' thing as used in current tools.

Because I really don't like what it suggests (library parts can look different when used in different models).

It will make multi file/model editing very hard to maintain.

It will also mess with your part inventory overviews, as the images of parts can differ per model while reality has only one version.

In short could we postpone voting on this, unless we limit it to just the folder locations and leave scoping/context out of it for the time being.
Reply
RE: Search path discussion
#23
I remember an experiment I did way back when I started editing parts.

A default set-up of libraries have two folders, Parts and Unofficial/Parts.
A full dowload of all unofficial files will create a set of duplicated parts. A recycled partnumber will end up both in Parts and Unofficial Parts. 2 versions of the same partnumber.

To understand which file I was seeing in MLCad I edited the version in Unofficial. I gave it a different description and I coloured sections of it red. 
When I opened MLCad and generated a new part.lst and updated the library, I saw the old description from the official file in the Parts-folder, but the red geometry from the file in the Unofficial/Parts-folder. (... or vice versa. I can't remember now.)

It's a long time since I used MLCad, and a new version is released now, but I think this illustrate very well what could/will happen.

I have since then been removing old files from the official Parts-folder, and placed the updated version in Unofficial/Parts-folder.
Every official part release ends with me sitting at my desk and sorting all the files so that I don't have any duplicated partnumbers present.
I want to know that the geometry I see, in any editor, is the updated version.
Reply
RE: Search path discussion
#24
I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.
Reply
RE: Search path discussion
#25
(2021-04-14, 18:09)Travis Cobbs Wrote: I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.

It does. I modeled the search path dialog in LDDP after MLCad's
Reply
RE: Search path discussion
#28
(2021-04-14, 21:04)Orion Pobursky Wrote: It does. I modeled the search path dialog in LDDP after MLCad's

Based on the tutorial in Willy's response below, it sounds like MLCad supports having the user add the Unofficial\p and Unofficial\parts directories to the search path, but does not have these by default. What am I missing?
Reply
RE: Search path discussion
#29
(2021-04-15, 20:14)Travis Cobbs Wrote: Based on the tutorial in Willy's response below, it sounds like MLCad supports having the user add the Unofficial\p and Unofficial\parts directories to the search path, but does not have these by default. What am I missing?

You're not missing anything. This is the same thing all the programs do since there's no defined path. That said, I stand by my opinion that we should not officially define an unofficial part path. It should always be explicitly user set.
Reply
RE: Search path discussion
#26
(2021-04-14, 18:09)Travis Cobbs Wrote: I didn't think that MLCad supported Unofficial parts directly. It may be that the AIOI configures it to support them, or that you did that.

You're not reading my tutorials? Shame on you Big Grin

http://www.holly-wood.it/mlcad/unofficial1-en.html

w.
LEGO ergo sum
Reply
RE: Search path discussion
#27
(2021-04-15, 6:31)Willy Tschager Wrote: You're not reading my tutorials? Shame on you Big Grin

http://www.holly-wood.it/mlcad/unofficial1-en.html

w.

No, it's just that nobody uses it now  Big Grin Big Grin Big Grin
Reply
RE: Search path discussion
#30
(2021-03-16, 13:26)Orion Pobursky Wrote: Since non-LSB members can't comment on the LSB forum, this thread is for discussion of the search path proposal there.

To kick this off I'd like to comment on something Roland posted:

This is precisely the point of embedding such a file in the MPD and why we do it for the OMR. Local files should always override library parts. Otherwise the OMR breaks and parts authoring become morse inconvenient since you'd have to overwrite the official file if you're using or creating a fix.

Could we please bring this to a happy end?

w.
LEGO ergo sum
Reply
RE: Search path discussion
#31
(2021-04-30, 18:50)Willy Tschager Wrote: Could we please bring this to a happy end?

w.

I think we're waiting on Roland to do some investigating/testing
Reply
« Next Oldest | Next Newest »



Forum Jump:


Users browsing this thread: 2 Guest(s)