Proposal: New !DATA meta-command for embedded textures - Printable Version +- LDraw.org Discussion Forums (https://forums.ldraw.org) +-- Forum: Administrative (https://forums.ldraw.org/forum-4.html) +--- Forum: Standards Board (https://forums.ldraw.org/forum-5.html) +--- Thread: Proposal: New !DATA meta-command for embedded textures (/thread-23683.html) |
Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2019-09-23 Note: this is in response to this thread. This is a proposal to add a new !DATA meta-command that can be used to embed texture data into model files (for use with unofficial parts in an OMR model). The proposed specification for the new meta-statement follows. Please discuss. Purpose This document describes the !DATA meta-statement, which contains a set of commands for embedding binary data into LDraw files. New meta-statement token !DATA This is the main data-embedding meta-command. Reuse of existing meta-statement token !: This will be used to specify a BASE-64 encoded data stream. This token was introduced with the !TEXMAP meta-statement, and will be reused here for three reasons: its reuse will not interfere with its use in !TEXMAP; it is useful for the same reasons that it was useful in !TEXMAP; and !DATA is only useful right now for embedding texture data for use with !TEXMAP, so the two meta-statements are related to each other. Syntax 0 !DATA START 0 !: <data1> 0 !: <data2> ... 0 !: <datan> 0 !DATA END Description This meta-statement allows for BASE-64 encoded data to be used inside a multi-part-dat (MPD) file. The decoded binary data replaces of the embedded MPD sub-file.
0 FILE main.ldr 0 Author: Roland Melkert 1 15 -40 -20 0 1 0 0 0 0 1 0 1 0 plane.dat 0 FILE plane.dat 0 plane texmap test 0 UNOFFICIAL PART 0 BFC NOCERTIFY 0 !TEXMAP START PLANAR 0 0 0 40 0 0 0 0 40 checker123yyz.png 4 16 0 0 0 40 0 0 40 0 40 0 0 40 0 !TEXMAP END 0 FILE checker123yyz.png 0 !DATA START 0 !:iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAABmJLR0QA/wD/AP+gvaeTAAAACXBI 0 !:WXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH3gcbDhUzUyNxeQAAArVJREFUeNrt3cENwzAMBMFjoM7d 0 !:p6vIX2lBeRECZzo4GFjIL9ZOdhpVeu3dOj9VZb/9bT4BxhIAEABAAAABAAQAEABAAAABAAQAEABA 0 !:AAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEADgX5Wk9UC6+/T22+8F 0 !:AAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgACAAgAIAAAAIACAAgAIAAAAIA 0 !:CAAgAIAAAAIACABwgeU+u/32z93vBQB+AQABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA 0 !:EABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAI4t99l79z9p3v/6/pP3ewGAXwBAAAAB 0 !:AAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAAABAAQA 0 !:EABAAIBjy3324fu/8f0H7/cCAL8AgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAA 0 !:gAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAxypJ64H06ffZ7bffCwAQAEAAAAEABAAQAEAA 0 !:AAEABAAQAEAAAAEABAAQAEAAAAEABAAQABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAA4ArL 0 !:fXb77Z+73wsA/AIAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAAAgAIACAAgAAA 0 !:AgAIACAAgAAAAgAIACAAgAAAAgAcqyStB9Ldp7fffi8AQAAAAQAEABAAQAAAAQAEABAAQAAAAQAE 0 !:ABAAQAAAAQAEABAAQABAAAABAAQAEABAAAABAAQAEABAAAABAAQAEABAAIAL/AB/0mJQmotO+gAA 0 !:AABJRU5ErkJggg== 0 !DATA END RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2019-09-25 So basically this is a encoded sub-subFile? If so why limited to one block per mpd subfile? I would propose to ether:.... EDIT: sorry based my comment on the wrong version. RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2019-09-25 (2019-09-23, 23:10)Travis Cobbs Wrote: Note: this is in response to this thread. I like this proposal, only thing I would like to add is to maybe use the DATA meta itself as an alternative to the FILE meta. That way you won't end up with empty models in non supported programs (they would just ignore the unknown 0 lines, as if they were part of the current FILE meta's content). Also you wont need the "0 !DATA END" line as normal FILE rules would apply (meaning the next FILE or !DATA meta will split subfiles) Code: 0 FILE main.ldr RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2019-09-25 (2019-09-25, 0:51)Roland Melkert Wrote: I like this proposal, only thing I would like to add is to maybe use the DATA meta itself as an alternative to the FILE meta. Would having a sub-file in an MPD with no geometry really hurt anything in programs that don't support !DATA? I can see plus and minus points both ways. Having !DATA behave just like FILE is arguably cleaner, since it doesn't involve replacing a file that is being loaded with completely different contents. But putting it inside a FILE is much less of a departure from established practice. Putting it inside a FILE allows for header information to go in. However, I'm not sure if this is a plus or a minus, since these are intended as a drop-in for actual texture files, and such files obviously cannot contain LDraw header information. My general leaning is toward having !DATA go inside a FILE (as indicated in the proposal). However, I'm not strongly in favor of that, so if you and Chris both feel that !DATA should be used instead of FILE, I can go with that. RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2019-09-25 (2019-09-25, 2:07)Travis Cobbs Wrote: Would having a sub-file in an MPD with no geometry really hurt anything in programs that don't support !DATA?Not during viewing. But most editors have a list of submodels somewhere. Navigating to them will present an empty model which you can then 'mess up' if not careful. And doing so would break the mpd for software which does understand the new meta. RE: Proposal: New !DATA meta-command for embedded textures - Willy Tschager - 2019-11-19 Could you please bring this to an end? Thx, w. RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2019-11-19 (2019-09-25, 19:13)Roland Melkert Wrote: Not during viewing. I don't understand your final point, but I'll bow to your superior wisdom with respect to editors. (Note, since it's often not obvious in posts, I am not being sarcastic.) Please write up a proposal (like mine above) for your proposed version of !DATA, and we can discuss it, and then vote on it. RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2020-01-15 (2019-11-19, 18:25)Travis Cobbs Wrote: I don't understand your final point, but I'll bow to your superior wisdom with respect to editors. (Note, since it's often not obvious in posts, I am not being sarcastic.) Please write up a proposal (like mine above) for your proposed version of !DATA, and we can discuss it, and then vote on it. Unless I'm misunderstanding, I think your are suggesting we go with my original proposal. Since you were the one to suggest using MPD submodels in the first place I want to make sure you haven't forgotten your original reasoning. RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-01-15 (2020-01-15, 2:09)Travis Cobbs Wrote: Unless I'm misunderstanding, I think your are suggesting we go with my original proposal. Since you were the one to suggest using MPD submodels in the first place I want to make sure you haven't forgotten your original reasoning. I was trying to make it an hybrid. With !DATA acting like FILE non supporting editors will ignore it all, and supporting ones could hide those subfiles and/or give give them special treatment while reusing the search mechanism already in place for the FILE meta. Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go? I can imagine adding more stuff to mpd's (e.g. LDCad scripts ) RE: Proposal: New !DATA meta-command for embedded textures - Orion Pobursky - 2020-01-15 (2020-01-15, 19:11)Roland Melkert Wrote: Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go? This actually worries me a bit. LDraw file could, potentially, be a security risk making me want to scrap the idea completely. RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2020-01-15 (2020-01-15, 19:11)Roland Melkert Wrote: I was trying to make it an hybrid. I don't understand what you mean here. My original proposal would be ignored by non supporting programs, since every line is a comment. If you want to treat it like a file entry in LDCad, that seems like an implementation detail, and not something that should be in the spec. Having said that, if we have !DATA be independent of FILE, then it should be valid in a non-MPD LDraw file. (2020-01-15, 19:11)Roland Melkert Wrote: Side note, should we include the mime type of the encoded data? "image/png" being the only one officially supported from the get go? I see your point, but given Orion's comment, perhaps we should not include the MIME type. RE: Proposal: New !DATA meta-command for embedded textures - Orion Pobursky - 2020-01-15 (2020-01-15, 23:39)Travis Cobbs Wrote: I see your point, but given Orion's comment, perhaps we should not include the MIME type. Yes. We should allow one format and one format only. PNG is probably the right choice. However, the more I think about this, the more I'm leaning more towards just distributing images alongside the model file and not embedding them. The original reason for this meta was to embed unofficial textures in OMR models and I'm starting to become unconvinced that we need a new meta to do something that is solved just by distributing an extra file. RE: Proposal: New !DATA meta-command for embedded textures - Philippe Hurbain - 2020-01-16 I still find the all inclusive single file much more elegant (and easier for the not so much computer savvy user). RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-01-16 (2020-01-15, 23:39)Travis Cobbs Wrote: I don't understand what you mean here. My original proposal would be ignored by non supporting programs, since every line is a comment. If you want to treat it like a file entry in LDCad, that seems like an implementation detail, and not something that should be in the spec. Sorry for the confusion. I liked that version of the proposal except its sub-subfile nature. I wanted to solve that by making !DATA a special version of FILE so they are 'normal' subfiles for programs whom support it. And there won't be a need for scoping rules etc as it can use the normal FILE search order methods already available in MPD supporting software. (2020-01-15, 23:39)Travis Cobbs Wrote: Having said that, if we have !DATA be independent of FILE, then it should be valid in a non-MPD LDraw file.I thought we wanted to drop the mpd/ldr distinction of files anyway? And because it's all comments like you said the file (using !DATA as an alt FILE) won't even register as mpd in non supporting software. RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-01-16 (2020-01-16, 6:46)Philippe Hurbain Wrote: I still find the all inclusive single file much more elegant (and easier for the not so much computer savvy user).I agree. Everything can be misused, even plain 1.0 LDraw files (buffer overflow bugs etc). Alternative would be to zip multiple file stuff and put it in a standardized '.ldz' file or something. RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2020-01-17 (2020-01-16, 22:26)Roland Melkert Wrote: Sorry for the confusion. I see where complications might show up. In LDView, once I load a file, it always wins when any other reference looks for a file of the same name. In order to support MPDs, I use a two-pass approach, where a placeholder for each sub-file is stored in memory on the first pass, so on the second pass any reference will find the placeholder before looking for the file on disk. So, what we need to do is specify that !DATA entries behave just like MPD FILE entries for binary data, but are not otherwise related. (2020-01-16, 22:26)Roland Melkert Wrote: I thought we wanted to drop the mpd/ldr distinction of files anyway? I wan't referring to the extension. Aren't MPD files supposed to begin with 0 FILE? RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-01-17 (2020-01-17, 3:02)Travis Cobbs Wrote: So, what we need to do is specify that !DATA entries behave just like MPD FILE entries for binary data, but are not otherwise related.Yes, but they are related though, I would even call the !DATA meta an mpd extension if we go this route. Because a !DATA will end the current FILE block if any. It could also be the first subfile in the mpd (unless we mandate it can't). (2020-01-17, 3:02)Travis Cobbs Wrote: Aren't MPD files supposed to begin with 0 FILE?They do. And this is another argument for !DATA being an MPD extension, because there needs to be at least one FILE block along side the !DATA one to have a meaningful model. Although you could have a ldr with just the !DATA block and reference that from another ldr (just like an mpd with only one FILE block), but that seems a bit wasteful RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-01-17 I took the liberty of editing the current MPD spec to include the !DATA one to see how it feels. ---------------------- Purpose MPD files or "Multi-Part Documents" are a way to combine several LDraw and encoded binary files into one consolidated source. This allows for ease in posting or emailing a model made up of many subparts. Structure A MPD file consists out of blocks of LDraw code separated by 0 FILE or 0 !DATA statements. Each block is considered a separate file and can be referenced by the other ones as needed. Blocks starting with the 0 FILE statement are normal type 0..5 line LDraw code. Blocks starting with the 0 !DATA statement contain binary data encoded using a multitude of 0 !: lines. Each of those lines contain a chunk of base64 encoded data which a parser must combine before decoding. No other LDraw statements may be used inside a !DATA block. The end of each block, or just the last block in the MPD, may be marked with a 0 NOFILE line. The 0 NOFILE command is only required if the file's contents are followed by non-LDraw content (such as the poster's signature lines). LDraw parsers must then ignore all content until a new 0 FILE or 0 !DATA is found. In order to support the inclusion of MPD files in message systems (like email), any text lines before the first 0 FILE or 0 !DATA statement will be discarded. It is considered to be an error for any LDraw code (other than comment lines) to appear before the first 0 FILE or 0 !DATA statement. The first block in the MPD is treated as the 'main model' -- all other files in the MPD will only be rendered if they are referenced by the main model, directly or indirectly. For this reason it is recommended (but not required) to always use a 0 FILE block as the first entry in the MPD. So far, there are no clear scoping or namespace rules on MPD files. If you put a file named stud.dat in your MPD file, don't be surprised to see your stud.dat file appear on the top of every single brick in your scene. MPD META Statements FILE Format: 0 FILE <filename> Where: <filename> is the name of the following LDraw file. !DATA Format: 0 !DATA <filename> Where: <filename> is the filename of the following encoded data. !: Format: 0 !: <base64 string> Where: <base64 string> is a chunk of the base64 encoded data (A-Za-z0-9+/) with a maximum length of 80 characters (60 bytes of data). Each line except the last should use the same length and be a multitude of 4 characters. Padding the last line with '=' characters to complete its last group of 4 is optional. NOFILE Format: 0 NOFILE There are no options or parameters. Example Code: 0 FILE main.ldr ------------- Just a draft / initial setup. Might need more examples (the above one being an advanced one). note: I wonder if we should drop the ! in !DATA to get it consistent with the FILE meta. RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2020-01-17 I'm ok with this. Some thoughts:
RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-02-14 (2020-01-17, 21:52)Travis Cobbs Wrote: I'm ok with this. What do the other LSB members think? RE: Proposal: New !DATA meta-command for embedded textures - Travis Cobbs - 2020-02-14 (2020-02-14, 22:05)Roland Melkert Wrote: What do the other LSB members think? FYI, Chris is the only other member. RE: Proposal: New !DATA meta-command for embedded textures - Chris Dee - 2020-02-15 (2020-02-14, 22:56)Travis Cobbs Wrote: FYI, Chris is the only other member. I am happy with this proposal. RE: Proposal: New !DATA meta-command for embedded textures - Orion Pobursky - 2020-05-20 I rescind any objections (explicit or otherwise) to this spec and would like to see it ratified. RE: Proposal: New !DATA meta-command for embedded textures - Roland Melkert - 2020-05-20 (2020-05-20, 20:21)Orion Pobursky Wrote: I rescind any objections (explicit or otherwise) to this spec and would like to see it ratified. I think everyone was happy with my sketch-up https://forums.ldraw.org/thread-23683-post-35497.html#pid35497 Only thing I would like to add, but could be done on the next revision, is clear rules about search order (for mpd as a whole) |