subfiles in MPD files

[Michael Heidemann]

I am still unhappy with subfiles in MPD files and what to do.

Example:
I have a file "testpart.dat" like this:
0 Testfile
0 Name: testpart.dat
1 16 0 0 0 1 0 0 0 1 0 0 0 1 3001.dat
1 16 0 32 0 1 0 0 0 1 0 0 0 1 s\testparts.dat

"testparts.dat" like this:
0 Testfile subfile blue
0 Name: s\testparts.dat
1 1 0 0 0 1 0 0 0 1 0 0 0 1 3001.dat

If I put the testparts.dat in a subfolder relative to the folder where testpart.dat resists I see with MLCad and LDView both parts.

Now I open testpart.dat in MLCad.
After that I import testparts.dat.
I save that content in the new file testpart.mpd

I would expect that I can now give this testpart.mpd to another user and that he can see what I have build, but that is not the case.
If I open the file from another location on my system MLCad claims "File s\testparts.dat not found! Continue loading?"
After I answered yes only the main color brick has been displayed.
One more bad thing with MLCad - if you export the files ("Multipart" - "Export Models...") the reference "s\testparts.dat" is changed to "Unknown".

With this behaviour it is impossible with the current tools to show the complete model direct out of the MPD file nor to export all files in a good shape.

For this reason I have in MPDCenter implemented a full export of the files in the correct foldertree and then all is shown correctly.

BUT MPDCenter is not designed as a viewer tool (build in viewer is ldvlib.dll).

To have also good experience with LDView I suggest to also implement something that avoid these mistakes.

Or, if we want to do it completely different, we need to forbit subparts to be in the MPDfiles. This can be archieved by changing all references to those subparts and delete the leading "s\".
But then one problem still remains - '48\' primitives. We can not just cut the 48 away, because then we have a different part (16').

Any other thoughts on this?

[Alex Taylor]

It's an error in your .mpd file - the second file should start with "0 FILE s\testparts.dat" not "0 FILE testparts.dat". Change that line and LDView and MLCad open it correctly.

Don't forget that parsers will look at the filename, not the 'Name:' meta-command, when trying to resolve a sub-file reference.

This is basically a bug in MLCad, although to be fair it wasn't designed with this in mind.

[Michael Heidemann]

Ok, then I have to adjust MPDCenter again

What you discribed was also my intend, but as MLCad did not work this way I decided to emulate the behaviour of MLCad for the import of files as MPD files has been introduced by MLCad!

One more bad thing with MLCad in this context:
If I use "Multipart" - "Change Model Info..." and add the missing "s\" in front of the filename the Name: entry that has been correct until my change will be changed to not carry the leading "s\" anymore.
You have to change the Name: entry line manually after you changed the file info.

STRANGE!!

[Michael Heidemann]

As a result of all these findings we should NOT use MLCad if any kind of "s\" or "48\" files needs to be nested in a MPD file.
Because once you get a file not found error this reference is changed to "Unkown" if you safe the file.

[Alex Taylor]

I think what it comes down to is that MLCad's primary function is a model-editor. Sub-models (note: not sub-parts) do not have prefixes on their names, so MLCad simply doesn't support them. It has no way of knowing that the file you imported was a subpart and thus should have the 's\' prefix.

Given the above it's not too surprising that trying to use it like this has unexpected results :-)

My own parser takes this into account: it reads the 'FILE', 'Name:' and '!LDRAW_ORG' lines and when they disagree it attempts to work out the most probable solution. In the case of your file, although the 'FILE' line just gives the filename as 'testparts.dat', because the 'Name:' line reads 's\testparts.dat' the parser concludes that it is in fact a subpart and fixes the 'FILE' line accordingly. Hence if I load the file into my parser it renders both bricks first time :-)

[Alex Taylor]

As a result of all these findings we should NOT use MLCad if any kind of "s\" or "48\" files needs to be nested in a MPD file.
Because once you get a file not found error this reference is changed to "Unkown" if you safe the file.

That I think is an old bug which has been fixed in v3.4.

[Michael Heidemann]

That is exactly what I thought our current application would do also.

In the specs for the MPD file it is stated that after FILE the real filename should be written and not with a leading "s\".
The implementation that you made is perfect, because it make sence to have the Name: entry line.
For MLCad and LDView it seems to me that this line could be cancelled.

I prefer very much your implementation.

Will the spec for MPD files be updated to reflect this - tiny - difficulties?

[Alex Taylor]

In the specs for the MPD file it is stated that after FILE the real filename should be written and not with a leading "s\".

Actually, it doesn't. It says:

Format: 0 FILE <model>

Where:
<model> is the name of the following LDraw file.


The name of the file is the value that would be used with a type-1 line: the value of the 'Name:' meta-command, not the literal filename it would have if written to disk. 'FILE' and 'Name:' are thus required to have the same value (something which the OMR spec appears to be trying to break, incidentally).

[Michael Heidemann]

Stupid me - did not yet realized that version! - Thanks.

[Michael Heidemann]

"(something which the OMR spec appears to be trying to break, incidentally)"

Can you please explain that to me and where this should happen?

[Roland Melkert]

Filename is a misleading name for (as is model for that matter) all this.

The thing we are using is a reference, a reference is always relative and can include a subfolder.

The biggest mistake a parser can make is to simply split the absolute pathname into a folder and filename. And thus internally loosing the relative references. I suspect that's what's going on inside mlcad in regards to mpd's

[Alex Taylor]

Under the section 'MPD File Structure' it says:

Unofficial parts are allowed to be used. The filename of the unofficial part is subject to the following naming rules:

<Set Number>[-<Optional Qualifier>] - <Unofficial Part Number>.dat


And then under 'File Headers' we have:

Each individual model file in the MPD that represents an unofficial part must use the standard header format of unofficial parts on the parts tracker. Its 0 FILE entry will reflect the modified MPD version of the part's filename, but the rest of the header will reflect the original filename.


This is saying that if I import the part '12345.dat' into a model called '8888-1.mpd' then the part's header should be:

0 FILE 8888-1 - 12345.dat
0 Part Title
0 Name: 12345.dat


This is inconsistent with the MPD spec, which expects both 'FILE' and 'Name:' to have the same value.

The OMR spec also makes no mention of what you should do if you import a part and its subpart. For example, suppose I have the unofficial part '12345.dat' which references the unofficial subpart 's\12345s01.dat'. Presumably the subpart's 'FILE' line will have to read 's\8888-1 - 12345.dat' (although the spec doesn't state anything about this), and so the part will have to have its type-1 lines fixed-up to point to the new name.

I strongly disagree with the renaming given in the current spec, not least because no editing tool currently available will handle it correctly. I also cannot see why it was even considered necessary or desirable.

[Roland Melkert]

Seeing this thread also made me think about an old wish of me, namely the possibility to have nested submodels.

But seeing all the problems with normal mpd's I think the world is not ready for it

[Alex Taylor]

Filename is a misleading name for (as is model for that matter) all this.

The thing we are using is a reference, a reference is always relative and can include a subfolder.

The biggest mistake a parser can make is to simply split the absolute pathname into a folder and filename. And thus internally loosing the relative references. I suspect that's what's going on inside mlcad in regards to mpd's

Yes, that's quite plausible. As I said, I don't believe MLCad was really designed with this in mind :-) It is first and foremost a model-editor, and models do not have the concept of relative paths in the way that parts do.

My parser calls these things 'TargetName'. A TargetName consists of three components: the Prefix, the Name and the Suffix. The Prefix is optional, and when present is either 's\' or '48\'. The Name is the filename without its extension, and the Suffix is simply the extension. From an editing perspective, the user can only set the Name. The Prefix and Suffix are set automatically from the 'Type' value (the thing that gets written to the !LDRAW_ORG line): for example, if Type is 'Model' then the Prefix is omitted and the Suffix is set to '.ldr'.

[Michael Heidemann]

I have brought exact this into discussion to the LSC!!

http://forums.ldraw.org/showthread.php?t...47#pid6047

As a conclusion of that discussion I have both entries the same, except for the "s\" parts based on MLCad behaviour.

[Roland Melkert]

I think this is mainly a textual formatting problem, the idea behind the spec is to modify ANY imported file with the prefix.

This is logical (I think) since there are very few unofficial parts that have only a single .dat, so you have to import the whole thing recursively for it to be useful. In such a scenario adjusting the type1 lines seems implied.

Doing all this while leaving the name tag alone gives unpacking software the possibility to restore the original file names.

But you are right the spec could mention all this better.

[Michael Heidemann]

That is a good approach. Maybe I'll use it in the future if needed.

I have mainly to deal with files that might be somewhat correct and I have to found out what is missing to be conform to official specs. :-)
This is a quite different task.

[Michael Heidemann]

I will have a look at the specs later on and see what can be improved/clarified on that and make a suggestion.

[Michael Heidemann]

I just saw, that you can speed things up regarding official specs because you currently are a member of the current LSC. :-)

[Michael Heidemann]

Sounds interesting.

I think the main discussion about files references in MPD files is finished with a good end.

So please explain your idea.

[Roland Melkert]

Well basically it would mean you could put one mpd inside another without loosing the search order or creating naming conflicts. In such a document you could put the whole unofficial part recursively while only having to rename the top level .dat. The bigest problem meta wise is the fact the file meta has no mandatory ending so a following file can never start a nested block.

To solve this I was thinking about an additional meta indicating the nesting level, thus creating additional 'namespaces'.

[Alex Taylor]

That is a good approach. Maybe I'll use it in the future if needed.

I have mainly to deal with files that might be somewhat correct and I have to found out what is missing to be conform to official specs. :-)
This is a quite different task.

Oh, there's a lot more going on in my parser than just that :-) It looks at as much information as is available to try and determine the correct 'type' for the file, and recognises all the outdated and 3rd-party values that can turn up in places of the !LDRAW_ORG line. It's not foolproof - that would be impossible - but it does detect and fix every single known error in the official library, plus some nasties I found in .mpd files I've downloaded from various sources.

[Alex Taylor]

Well basically it would mean you could put one mpd inside another without loosing the search order or creating naming conflicts. In such a document you could put the whole unofficial part recursively while only having to rename the top level .dat. The bigest problem meta wise is the fact the file meta has no mandatory ending so a following file can never start a nested block.

To solve this I was thinking about an additional meta indicating the nesting level, thus creating additional 'namespaces'.

I don't think this is something that needs to be addressed in the file-format - what you're describing is an application-level problem, and it's not particularly difficult to solve.

[Roland Melkert]

I've started a thread:

http://forums.ldraw.org/showthread.php?t...91#pid6691

Let's see what the other think about all this

[Roland Melkert]

I don't think so, an application has no way of kowing how to resolve e.g.

Code:

0 FILE main.ldr
1 <params> a.dat
1 <params> imported.dat

0 FILE a.dat
1 <params> 3001.dat

0 FILE imported.dat
1 <params> a.dat

0 FILE a.dat
1 <params> 3001.dat

where the second a.dat is nested from wherever imported came from.

It could be solved by something like:

Code:

0 FILE main.ldr
1 <params> a.dat
1 <params> imported.dat

0 FILE a.dat
1 <params> 3001.dat
0 NOFILE

0 !MPDLEVEL ++

0 FILE imported.dat
  1 <params> a.dat

0 FILE a.dat
  1 <params> 3001.dat
0 NOFILE

0 !MPDLEVEL --

By putting the mpdlevel meta inbetween valid mpd content it should be backwards compatible without polluting extracted dat's, though incorrectly resolved, in non supporting apps

This is just a quick draft though, It might be solved in different / better ways.

[Alex Taylor]

Well, your example is illegal - you cannot have two 'FILE' statements with the same value in a file.

[Roland Melkert]

Yes, but that's exactly what I'm hoping to improve upon.

[Michael Heidemann]

That is, in a little different interpretation, also a goal of mine with MPDCenter.

The different level are indicated by adding the filename from the container to the filename itself.

By doing this it still remains a valid ldraw file and the you can refer to each file by an individual filename.

Once finished I will store in then .xmpd called files the filename as mpdname&&filename.dat. All file handling has to be done by the application, but can also be done manually.

[Michael Heidemann]

Maybe we should discuss/compare/combine our approaches.

At present I am sure yours is much better than mine :-), so I can improve DATHeader, MPDCenter etc.

[Alex Taylor]

Right, but your solution is for a problem that does not currently exist.

I can't see a situation arising where importing an unofficial part with all its dependencies could in fact cause this. By definition there can only be a single entity in the parts-library with a given TargetName. The only times that you would have two files with the same filename in your library are if one file is in 'parts' and the other in 'parts\s' (or similar), in which case they have unique TargetNames; or if one file was in 'parts' and the other in 'models' (or 'unofficial\parts' if you're into that), in which case the one in 'models' takes precedence when doing a lookup and the one in 'parts' is simply ignored (or vice-versa if your search-path is configured the other way around).

The closest I can see to your example would be if you imported two parts which both had a dependency on the same subpart (or primitive); in that case it would be redundant to import the subpart twice and it's the application's problem to ensure that this does not happen. In fact, your solution would not only allow duplicate imports, it would actually make them more likely to happen.

I've just checked, and MLCad actually has a nasty bug here: it actually loaded the above code! This absolutely should not have happened, since it merrily created two pages with the name 'a.dat' - something that it will not allow you to do if you try and create or rename a page. I'm mildly surprised that it didn't crash on loading, in fact. (There's a second bug, too: it seems to think that TargetNames are case-sensitive - 'A.dat' and 'a.dat' are regarded as being different, which is wrong.)

[Michael Heidemann]

Nothing to add at the moment, only that also MPDCenter does not claim a double FILE entry. It currently seems to ignore the first or the second entry. I'll have to check that.

[Michael Heidemann]

Just read that thread.

Code:

"It's unclear (jet somewhat implied imho) all the unofficial file's recursively used (also unofficial) files should be included in the mpd using the same prefix and as a result all the affected type 1 lines should be updated."

As far as I have understand (german is my native language) the goal should be:
Import every! file (part, subpart, primitive, 48-primitive) completely like it is and put in the 0 FILE entry for the MPD file exactly the same entry than in the Name: entry for the file.

So why you are asking for updating a type 1 line ??

According to Alex's post we can think of the Name: entry to be build from three components
<[optional] prefix><basefilename><extension> = <Targetname>
ex: <s\><test><.dat>

Therefore the specs have to be read for an MPD file entry to look like:
0 FILE <Targetname>
0 file description
0 Name: <Targetname>

[Roland Melkert]

So why you are asking for updating a type 1 line ??

Because if if you add a prefix to all recursively imported used unofficial files, type 1 lines in any of those files referring to other ones need to be updated to also include that prefix. Or the client software will still try to look outside the mpd for those files.

[Roland Melkert]

but your solution is for a problem that does not currently exist.

I'm not suggesting this as a solution to the main subject of this thread, but like I wrote above it's about a feature/extension I would think could be handy in some cases.

[Roland Melkert]

I like that approach, it doesn't need any new meta's just the && token. I'm not very familier with MPDCenter is this feature in the stable releases?

Maybe I'll support it in LDCad at some point (if you don't mind).

[Alex Taylor]

Yup. I would thus regard the imposition of the OMR-prefix onto imported unofficial parts as a problem.

If the model-author creates their model (let's call it "12345 - my model") and needs to import the unofficial part "1000.dat" from the Parts Tracker, the current OMR specification requires that the part be called "12345 - 1000.dat" once imported into the .mpd.

Suppose now that "1000.dat" is released into the official parts-library, and the model-author decides that they want to remove their imported copy - perhaps because the official release is of superior quality to the pre-release version they had imported, or simply to reduce the size of the .mpd file. Once they've deleted the imported part, they will then have to go through every type-1 line which referenced it and change it to read "1000.dat".

If we drop the requirement for the OMR-prefix on imported parts then it is sufficient to delete the import - the next time the user loads their model, the viewer software will simply load the new official part "1000.dat".

To be clear: I have no issue with the OMR-prefix being required for sub-models in the file. I just do not see the need for it on imported parts/subparts/primitives, and its presence is something of a nuisance.

If we are keeping the requirement then the spec should be updated to make it clear that when you import an unofficial subpart or hi-res primitive, its name needs to be changed like this:

"s\1000s01.dat" --> "s\12345 - 1000s01.dat"

and not:

"s\1000s01.dat" --> "12345 - s\1000s01.dat"

As it is currently written the latter interpretation, while wrong, is what's implied by the language.

[Alex Taylor]

Careful though...the '&' character isn't allowed in official files at present.

[Alex Taylor]

It might be, but I would be very wary about any change to the 'document structure'.

LDraw files are a flat-format, which causes some problems when trying to impose a structure on them. The closest we have in the official standards is the 'STEP' meta-command; as Allen suggested a couple of weeks ago, you could look at an LDraw document as being a collection of 'steps', each of which contains other elements - quads, references, other meta-commands and so on. I quite like this approach :-)

MLCad has an implementation of 'groups', albeit a rather idiosyncratic one - it indicates group-membership by putting a meta-command in front of each element which is a member of that group. This prevents groups from being nested, which is sometimes rather limiting, although it does allow you to group together elements from different steps.

Texmap is somewhat structural in that the elements between !TEXMAP START and !TEXMAP END could be seen as being members of a sub-collection. Unlike MLCad's groups, the members of a texmap must all be from the same step.

MPDs add an extra layer: you have a document, which contains multiple 'pages'. Each 'page' is simply a regular LDraw file; in fact, you can look at a simple LDraw file as being an MPD with a single 'page'; the only functional difference is that the file does not have an explicit '0 FILE' meta-command at the start.

So you end up with a possible document-tree like this:

Code:

Document.mpd
  |
  +--- Page1.ldr
  |        |
  |        +--- Step
  |        |      |
  |        |      +--- 1 16 0 0 0 1 0 0 0 1 0 0 0 1 3001.dat
  |        |      +--- 2 24 0 0 0 1 1 1
  |        |
  |        +--- Step
  |               |
  |               +--- 3 16 0 0 0 1 1 1 2 2 2
  |  
  +--- Page2.ldr
           |
           +--- Step
                  |
                  +--- 4 16 0 0 0 1 1 1 2 2 2 3 3 3
                  +--- Texmap
                           |
                           +--- 1 16 0 0 0 1 0 0 0 1 0 0 0 1 s\12345s01.dat

I'm not quite sure where your proposed syntax would fit into such a tree.

[Roland Melkert]

I'm not quite sure where your proposed syntax would fit into such a tree.

I agree on most of the structural stuff you mention, but the meta I was playing with lives outside the tree and is only ment to aid filelocation resolving by supplying an extra (virtual) namespace to first look in.

Please understand I'm not trying to launch such an extension at this moment, there are much more important things in need of official extensions imho.

Like the grouping functionality you mentioned. Although that's where we probably differ in opinion because I wouldn't want it to be limited to a single step. But that's stuff for a future thread (maybe when all the texture stuff is put to bed )

[Roland Melkert]

"s\1000s01.dat" --> "s\12345 - 1000s01.dat"

or

"s\1000s01.dat" --> "12345 - s\1000s01.dat"

wouldn't really matter in the end I think, both lead to valid path names. The second (simple prefix) is easier to manipulate for software though.

[Alex Taylor]

The second one would cause some interesting issues if the user were to try and export it to a file :-) I'd expect the tool to create a folder called "12345 - s" and stick the file in there.

[Michael Heidemann]

It does matter!

Thanks Alex for pointing that out.

The "s\" in the above example is a folder and therefore not part of the filename, but part of the reference!

So the first example has to be used in any case!

[Michael Heidemann]

Code:

Special characters, such as &, #, |, and ?, should be avoided as they may also cause cross-platform issues and create problems when used in URLs.

I have read this as 'it is allowed' because it is not 'forbidden' - translation error ?

On the other hand this delimiter "&&" is only used inside an MPD file. So I do not expect problems with URLs. On Windows and Linux machines this does not break any filename.

Is there anything that I should have been aware of additional to the above mentioned?

I am not limited to this delimiter, but it seemed reasonably to me.

[Michael Heidemann]

This feature is not yet in MPDCenter aktive (basics already coded).

Once it will work like I expect I would be proud to have this feature build also into other applications.

I have some other issues to solve before introducing something new. That's why I startet this thread.

[Roland Melkert]

I was to understand the prefix should go in front of the file meta value, which is a reference (the thing all the confusion in this thread come from).

I really think we should try to sort this mess out in the above mentioned lsc thread. I'm waiting on the other members for that.

[Alex Taylor]

No, no, you're quite correct in saying that they are permitted. However, they are not permitted in Parts Library files, and I wouldn't be surprised if a similar restriction applied to OMR files for the same reason: the files need to be cross-platform compatible.

Parts Library files are restricted to a-z, A-Z, 0-9, _ and . (plus \ or / for s\ and 48\ names, of course). I would expect OMR files to be the same, although I don't think this is stated in the spec (yet).

[Michael Heidemann]

Ok, I have no problem with being out of any existing specification with the delimiter.

I do not expect my solution will ever be of an official status. But with this solutin you **can** use the files with the extenstion .xmpd as normal mpd files (viewing) and additionally they are a container for multiple mpd files itself.