Proposal: New !DATA meta-command for embedded textures

[Travis Cobbs]

Note: this is in response to this thread.

This is a proposal to add a new !DATA meta-command that can be used to embed texture data into model files (for use with unofficial parts in an OMR model). The proposed specification for the new meta-statement follows. Please discuss.

Purpose

This document describes the !DATA meta-statement, which contains a set of commands for embedding binary data into LDraw files.

New meta-statement token

!DATA

This is the main data-embedding meta-command.

Reuse of existing meta-statement token

!:

This will be used to specify a BASE-64 encoded data stream. This token was introduced with the !TEXMAP meta-statement, and will be reused here for three reasons: its reuse will not interfere with its use in !TEXMAP; it is useful for the same reasons that it was useful in !TEXMAP; and !DATA is only useful right now for embedding texture data for use with !TEXMAP, so the two meta-statements are related to each other.

Syntax

0 !DATA START
0 !: <data1>
0 !: <data2>
...
0 !: <datan>
0 !DATA END

Description

This meta-statement allows for BASE-64 encoded data to be used inside a multi-part-dat (MPD) file. The decoded binary data replaces of the embedded MPD sub-file.

1. This meta-statement must come after the 0 FILE meta-statement signifying a sub-file in a multi-part-dat (MPD) file.
   
2. The FILE that the DATA belongs to must not contain any geometry.
   
3. The FILE must contain one and only one DATA section.
   
4. The FILE may contain header information.
   
5. All characters after 0 !: that are not valid BASE-64 characters must be ignored. (This is intended to allow white-space characters to be placed in the line, but still applies to all other characters that are not valid BASE-64 characters.)
   
6. Each 0 !: line except for the last must contain exactly 76 encoded characters.
   
7. The last 0 !: line must contain no more than 76 encoded characters.
   

Example

0 FILE main.ldr
0 Author: Roland Melkert
1 15  -40 -20 0  1 0 0  0 0 1  0 1 0 plane.dat

0 FILE plane.dat
0 plane texmap test
0 UNOFFICIAL PART
0 BFC NOCERTIFY

0 !TEXMAP START PLANAR  0 0 0  40 0 0  0 0 40  checker123yyz.png
4 16  0 0 0  40 0 0  40 0 40  0 0 40
0 !TEXMAP END

0 FILE checker123yyz.png
0 !DATA START
0 !:iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAABmJLR0QA/wD/AP+gvaeTAAAACXBI
0 !:WXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH3gcbDhUzUyNxeQAAArVJREFUeNrt3cENwzAMBMFjoM7d
0 !:p6vIX2lBeRECZzo4GFjIL9ZOdhpVeu3dOj9VZb/9bT4BxhIAEABAAAABAAQAEABAAAABAAQAEABA
0 !:AAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEADgX5Wk9UC6+/T22+8F
0 !:AAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgACAAgAIAAAAIACAAgAIAAAAIA
0 !:CAAgAIAAAAIACABwgeU+u/32z93vBQB+AQABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA
0 !:EABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAI4t99l79z9p3v/6/pP3ewGAXwBAAAAB
0 !:AAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA
0 !:EABAAIBjy3324fu/8f0H7/cCAL8AgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAA
0 !:gAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAxypJ64H06ffZ7bffCwAQAEAAAAEABAAQAEAA
0 !:AAEABAAQAEAAAAEABAAQAEAAAAEABAAQABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAA4ArL
0 !:fXb77Z+73wsA/AIAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAA
0 !:AgAIACAAgAAAAgAIACAAgAAAAgAcqyStB9Ldp7fffi8AQAAAAQAEABAAQAAAAQAEABAAQAAAAQAE
0 !:ABAAQAAAAQAEABAAQABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAIAL/AB/0mJQmotO+gAA
0 !:AABJRU5ErkJggg==
0 !DATA END

[Roland Melkert]

So basically this is a encoded sub-subFile?

If so why limited to one block per mpd subfile?


I would propose to ether:....


EDIT: sorry based my comment on the wrong version.

[Roland Melkert]

Note: this is in response to this thread.

I like this proposal, only thing I would like to add is to maybe use the DATA meta itself as an alternative to the FILE meta.

That way you won't end up with empty models in non supported programs (they would just ignore the unknown 0 lines, as if they were part of the current FILE meta's content).

Also you wont need the "0 !DATA END" line as normal FILE rules would apply (meaning the next FILE or !DATA meta will split subfiles)

Code:

0 FILE main.ldr
0 Author: Roland Melkert
1 15  -40 -20 0  1 0 0  0 0 1  0 1 0 plane.dat

0 FILE plane.dat
0 plane texmap test
0 UNOFFICIAL PART
0 BFC NOCERTIFY

0 !TEXMAP START PLANAR  0 0 0  40 0 0  0 0 40  checker123yyz.png
4 16  0 0 0  40 0 0  40 0 40  0 0 40
0 !TEXMAP END

0 !DATA checker123yyz.png
0 !:iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAABmJLR0QA/wD/AP+gvaeTAAAACXBI
0 !:WXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH3gcbDhUzUyNxeQAAArVJREFUeNrt3cENwzAMBMFjoM7d
0 !:p6vIX2lBeRECZzo4GFjIL9ZOdhpVeu3dOj9VZb/9bT4BxhIAEABAAAABAAQAEABAAAABAAQAEABA
0 !:AAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEADgX5Wk9UC6+/T22+8F
0 !:AAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgACAAgAIAAAAIACAAgAIAAAAIA
0 !:CAAgAIAAAAIACABwgeU+u/32z93vBQB+AQABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA
0 !:EABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAI4t99l79z9p3v/6/pP3ewGAXwBAAAAB
0 !:AAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA
0 !:EABAAIBjy3324fu/8f0H7/cCAL8AgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAA
0 !:gAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAxypJ64H06ffZ7bffCwAQAEAAAAEABAAQAEAA
0 !:AAEABAAQAEAAAAEABAAQAEAAAAEABAAQABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAA4ArL
0 !:fXb77Z+73wsA/AIAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAA
0 !:AgAIACAAgAAAAgAIACAAgAAAAgAcqyStB9Ldp7fffi8AQAAAAQAEABAAQAAAAQAEABAAQAAAAQAE
0 !:ABAAQAAAAQAEABAAQABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAIAL/AB/0mJQmotO+gAA
0 !:AABJRU5ErkJggg==

[Travis Cobbs]

I like this proposal, only thing I would like to add is to maybe use the DATA meta itself as an alternative to the FILE meta.

That way you won't end up with empty models in non supported programs (they would just ignore the unknown 0 lines, as if they were part of the current FILE meta's content).

Also you wont need the "0 !DATA END" line as normal FILE rules would apply (meaning the next FILE or !DATA meta will split subfiles)

Would having a sub-file in an MPD with no geometry really hurt anything in programs that don't support !DATA?

I can see plus and minus points both ways. Having !DATA behave just like FILE is arguably cleaner, since it doesn't involve replacing a file that is being loaded with completely different contents. But putting it inside a FILE is much less of a departure from established practice. Putting it inside a FILE allows for header information to go in. However, I'm not sure if this is a plus or a minus, since these are intended as a drop-in for actual texture files, and such files obviously cannot contain LDraw header information.

My general leaning is toward having !DATA go inside a FILE (as indicated in the proposal). However, I'm not strongly in favor of that, so if you and Chris both feel that !DATA should be used instead of FILE, I can go with that.

[Roland Melkert]

Would having a sub-file in an MPD with no geometry really hurt anything in programs that don't support !DATA?

Not during viewing.

But most editors have a list of submodels somewhere.

Navigating to them will present an empty model which you can then 'mess up' if not careful.

And doing so would break the mpd for software which does understand the new meta.

[Willy Tschager]

Could you please bring this to an end?

Thx, w.

[Travis Cobbs]

Not during viewing.

But most editors have a list of submodels somewhere.

Navigating to them will present an empty model which you can then 'mess up' if not careful.

And doing so would break the mpd for software which does understand the new meta.

I don't understand your final point, but I'll bow to your superior wisdom with respect to editors. (Note, since it's often not obvious in posts, I am not being sarcastic.) Please write up a proposal (like mine above) for your proposed version of !DATA, and we can discuss it, and then vote on it.

[Travis Cobbs]

I don't understand your final point, but I'll bow to your superior wisdom with respect to editors. (Note, since it's often not obvious in posts, I am not being sarcastic.) Please write up a proposal (like mine above) for your proposed version of !DATA, and we can discuss it, and then vote on it.

Unless I'm misunderstanding, I think your are suggesting we go with my original proposal. Since you were the one to suggest using MPD submodels in the first place I want to make sure you haven't forgotten your original reasoning.

[Roland Melkert]

Unless I'm misunderstanding, I think your are suggesting we go with my original proposal. Since you were the one to suggest using MPD submodels in the first place I want to make sure you haven't forgotten your original reasoning.

I was trying to make it an hybrid.

With !DATA acting like FILE non supporting editors will ignore it all, and supporting ones could hide those subfiles and/or give give them special treatment while reusing the search mechanism already in place for the FILE meta.

Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go? 

I can imagine adding more stuff to mpd's (e.g. LDCad scripts   )

[Orion Pobursky]

Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go?

This actually worries me a bit. LDraw file could, potentially, be a security risk making me want to scrap the idea completely.

[Travis Cobbs]

I was trying to make it an hybrid.

With !DATA acting like FILE non supporting editors will ignore it all, and supporting ones could hide those subfiles and/or give give them special treatment while reusing the search mechanism already in place for the FILE meta.

I don't understand what you mean here. My original proposal would be ignored by non supporting programs, since every line is a comment. If you want to treat it like a file entry in LDCad, that seems like an implementation detail, and not something that should be in the spec. Having said that, if we have !DATA be independent of FILE, then it should be valid in a non-MPD LDraw file.

Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go? 

I can imagine adding more stuff to mpd's (e.g. LDCad scripts   )

I see your point, but given Orion's comment, perhaps we should not include the MIME type.

[Orion Pobursky]

I see your point, but given Orion's comment, perhaps we should not include the MIME type.

Yes. We should allow one format and one format only. PNG is probably the right choice.

However, the more I think about this, the more I'm leaning more towards just distributing images alongside the model file and not embedding them. The original reason for this meta was to embed unofficial textures in OMR models and I'm starting to become unconvinced that we need a new meta to do something that is solved just by distributing an extra file.

[Philippe Hurbain]

I still find the all inclusive single file much more elegant (and easier for the not so much computer savvy user).

[Roland Melkert]

I don't understand what you mean here. My original proposal would be ignored by non supporting programs, since every line is a comment. If you want to treat it like a file entry in LDCad, that seems like an implementation detail, and not something that should be in the spec.

Sorry for the confusion.

I liked that version of the proposal except its sub-subfile nature.

I wanted to solve that by making !DATA a special version of FILE so they are 'normal' subfiles for programs whom support it.

And there won't be a need for scoping rules etc as it can use the normal FILE search order methods already available in MPD supporting software.

Having said that, if we have !DATA be independent of FILE, then it should be valid in a non-MPD LDraw file.

I thought we wanted to drop the mpd/ldr distinction of files anyway?

And because it's all comments like you said the file (using !DATA as an alt FILE) won't even register as mpd in non supporting software.

[Roland Melkert]

I still find the all inclusive single file much more elegant (and easier for the not so much computer savvy user).

I agree.

Everything can be misused, even plain 1.0 LDraw files (buffer overflow bugs etc).

Alternative would be to zip multiple file stuff and put it in a standardized '.ldz' file or something.

[Travis Cobbs]

Sorry for the confusion.

I liked that version of the proposal except its sub-subfile nature.

I wanted to solve that by making !DATA a special version of FILE so they are 'normal' subfiles for programs whom support it.

And there won't be a need for scoping rules etc as it can use the normal FILE search order methods already available in MPD supporting software.

I see where complications might show up. In LDView, once I load a file, it always wins when any other reference looks for a file of the same name. In order to support MPDs, I use a two-pass approach, where a placeholder for each sub-file is stored in memory on the first pass, so on the second pass any reference will find the placeholder before looking for the file on disk.

So, what we need to do is specify that !DATA entries behave just like MPD FILE entries for binary data, but are not otherwise related.

I thought we wanted to drop the mpd/ldr distinction of files anyway?

And because it's all comments like you said the file (using !DATA as an alt FILE) won't even register as mpd in non supporting software.

I wan't referring to the extension. Aren't MPD files supposed to begin with 0 FILE?

[Roland Melkert]

So, what we need to do is specify that !DATA entries behave just like MPD FILE entries for binary data, but are not otherwise related.

Yes, but they are related though, I would even call the !DATA meta an mpd extension if we go this route.

Because a !DATA will end the current FILE block if any. It could also be the first subfile in the mpd (unless we mandate it can't).

Aren't MPD files supposed to begin with 0 FILE?

They do.

And this is another argument for !DATA being an MPD extension, because there needs to be at least one FILE block along side the !DATA one to have a meaningful model.

Although you could have a ldr with just the !DATA block and reference that from another ldr (just like an mpd with only one FILE block), but that seems a bit wasteful

[Roland Melkert]

I took the liberty of editing the current MPD spec to include the !DATA one to see how it feels.
----------------------

Purpose
MPD files or "Multi-Part Documents" are a way to combine several LDraw and encoded binary files into one consolidated source. This allows for ease in posting or emailing a model made up of many subparts.

Structure
A MPD file consists out of blocks of LDraw code separated by 0 FILE or 0 !DATA statements. Each block is considered a separate file and can be referenced by the other ones as needed.

Blocks starting with the 0 FILE statement are normal type 0..5 line LDraw code.

Blocks starting with the 0 !DATA statement contain binary data encoded using a multitude of 0 !: lines. Each of those lines contain a chunk of base64 encoded data which a parser must combine before decoding. No other LDraw statements may be used inside a !DATA block.

The end of each block, or just the last block in the MPD, may be marked with a 0 NOFILE line. The 0 NOFILE command is only required if the file's contents are followed by non-LDraw content (such as the poster's signature lines). LDraw parsers must then ignore all content until a new 0 FILE or 0 !DATA is found.

In order to support the inclusion of MPD files in message systems (like email), any text lines before the first 0 FILE or 0 !DATA statement will be discarded. It is considered to be an error for any LDraw code (other than comment lines) to appear before the first 0 FILE or 0 !DATA statement.

The first block in the MPD is treated as the 'main model' -- all other files in the MPD will only be rendered if they are referenced by the main model, directly or indirectly. For this reason it is recommended (but not required) to always use a 0 FILE block as the first entry in the MPD.

So far, there are no clear scoping or namespace rules on MPD files. If you put a file named stud.dat in your MPD file, don't be surprised to see your stud.dat file appear on the top of every single brick in your scene.

MPD META Statements

FILE
Format: 0 FILE <filename>
Where:
<filename> is the name of the following LDraw file.

!DATA
Format: 0 !DATA <filename>
Where:
<filename> is the filename of the following encoded data.

!:
Format: 0 !: <base64 string>
Where:
<base64 string> is a chunk of the base64 encoded data (A-Za-z0-9+/) with a maximum length of 80 characters (60 bytes of data). Each line except the last should use the same length and be a multitude of 4 characters. Padding the last line with '=' characters to complete its last group of 4 is optional.

NOFILE
Format: 0 NOFILE
There are no options or parameters.

Example

Code:

0 FILE main.ldr
1 7 0 0 0 1 0 0 0 1 0 0 0 1 819.dat
1 4 80 -8 70 1 0 0 0 1 0 0 0 1 house.ldr
1 4 -70 -8 20 0 0 -1 0 1 0 1 0 0 house.ldr
1 4 50 -8 -20 0 0 -1 0 1 0 1 0 0 house.ldr
1 4 0 -8 -30 1 0 0 0 1 0 0 0 1 house.ldr
1 4 -20 -8 70 1 0 0 0 1 0 0 0 1 house.ldr

0 FILE house.ldr
1 16 0 0 0 1 0 0 0 1 0 0 0 1 3023.dat
1 16 0 -24 0 1 0 0 0 1 0 0 0 1 3065.dat
1 16 0 -48 0 1 0 0 0 1 0 0 0 1 3065.dat
1 16 0 -72 0 0 0 -1 0 1 0 1 0 0 3044b.dat
1 4 0 -22 -10 1 0 0 0 0 -1 0 1 0 sticker.ldr

0 FILE sticker.ldr
0 UNOFFICIAL PART
0 BFC CERTIFY CCW
1 16   0 -0.25 0   20 0 0   0 0.25 0   0 0 30   box5.dat
0 !TEXMAP START PLANAR   -20 -0.25 30   20 -0.25 30   -20 -0.25 -30   sticker.png
4 16   -20 -0.25 30   -20 -0.25 -30   20 -0.25 -30   20 -0.25 30
0 !TEXMAP END

0 !DATA sticker.png
0 !: iVBORw0KGgoAAAANSUhEUgAAAFAAAAB4CAIAAADqjOKhAAAAAXNSR0IArs4c6QAAAARnQU1BAACx
0 !: jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAEUSURBVHhe7du9DcIwFABhk5WgQLSsQM0UjMEU
0 !: 1BQsQIsoYAt6NkAYxQV/JQ7WvfuKkFTR6UmOFJzR9bJLkXTlNwyD6QymM5ju5Tl8m67KGUt3XJcz
0 !: J/yY8HZ/6C8BFvNZPoaesMF0BtMZTGcwncF0BtMZTGcwncF0BtMZTGcwnf8t0bmLh85gOoPpDKYz
0 !: mM5gOoPpDKYzmM5gunDBf3tN+/zqNKt367cbOeGUTstxf1nJZHPOx68T/u3XB5/7/zMXLTqD6Qym
0 !: M5jOYDqD6QymM5jOYDqD6QymM5jOYDqD6QymM5jOYLpwwW3t8ajBXTxtTHgwLlp0BtMZTGcwncF0
0 !: BtMZTNfKZzyDiT3hCFy06IIFp3QH/CBMh66aBy4AAAAASUVORK5CYII=

-------------
Just a draft / initial setup.

Might need more examples (the above one being an advanced one).

note: I wonder if we should drop the ! in !DATA to get it consistent with the FILE meta.

[Travis Cobbs]

I'm ok with this. Some thoughts:

• We should state that geometry is not allowed in a !DATA section. Any geometry that does exist is an error and either should be ignored or will produce undefined results.
  
• We don't want to drop the ! for consistency. 0 FILE has been around for a long time, so it's not likely to show up in any files with another meaning. 0 DATA and 0 : could easily show up in existing files.
  
• We should probably have an example where the !DATA section isn't at the end of the MPD.
  

[Roland Melkert]

I'm ok with this.

What do the other LSB members think?

[Travis Cobbs]

What do the other LSB members think?

FYI, Chris is the only other member.

[Chris Dee]

FYI, Chris is the only other member.

I am happy with this proposal.

[Orion Pobursky]

I rescind any objections (explicit or otherwise) to this spec and would like to see it ratified.

[Roland Melkert]

I rescind any objections (explicit or otherwise) to this spec and would like to see it ratified.

I think everyone was happy with my sketch-up
https://forums.ldraw.org/thread-23683-po...l#pid35497

Only thing I would like to add, but could be done on the next revision, is clear rules about search order (for mpd as a whole)