[Thoughts wanted] Parts.xml - xml improvement on Part.lst

[Tim Gould]

Hi All,

For a long time I've wanted there to be an .xml version of parts.lst containing not only the part and description, but also additional meta data. When Michael posted his fine looking LDFind today I had my final motivation to do something about it. Because a good Parts.xml would make searching so much easier.

As such LDMakeList 1.0 will make Parts.lst in the usual way, but will also make Parts.xml with a very simple format. I'd like some ideas of what should and shouldn't go into this format. Right now I'm wavering on 'License' for example.

So far each entry is as follows

Code:

<Part [Unofficial="1"] [Primitive="1"]>
<Number>XXXX.dat</Number>
<Description>Brick  7 x 19</Description>
[<License>GPL 4500.0</License>]
[<Category>Weird brick</Category>]
[<Keywords>this,brick,would,never,exist</Keywords>]
</Part>

where things in square brackets appear only when required.

I've attached an example.



Anyway, I welcome your thoughts on this.

[Tim Gould]

I should note that the other reason I like an .xml format is that it provides extra meta information to any LDraw program accessing the library. So e.g. LDFind could search straight from the .xml file, or a new editor could get meta information immediately from the .xml file.

[Michael Heidemann]

Thanks for mentioning my brand new tool LDFind.

The reason why I used xml for storing the data is, that the speed is much higher that I have done with my approach to store the necessary data.

I have also thought about using a complex structure to store the data, but I ended up with just a staight line in the xml file for one file.
By doing this I feel the speed is higher and if I browse with a text editor it is much easier to find the entries. Also the file is smaller

If we do such a file (my attempt is nearly the same) all data from the file should be stored in that xml file. So I think about this file like combining all header data into a single file with a reference to the place where the file can be found for the specific data.

[Tim Gould]

I hadn't actually run LDFind yet (was busy coding). Now that I have tried I have found that it doesn't run on my machine.

[Allen Smith]

My XML part catalog file also includes a category database.

Something similar to this:
<Catalog>
<Category>
<name="Arch"/>
<Part>2145.dat</Part>
<Part>2339.dat</Part>
</Category>
<!-- More categories -->
</Catalog>

Although such a thing could just as easily be generated from the list of parts, and probably quickly too.

Allen

[Tim Gould]

Yeah my plan was simply to generate a file with the important inform for the parts ie. a direct replacement for parts.lst. It can then be played with by any other code

As I'm not a Mac user, could you possible send me a copy of the .xml files you generate so I can check them out for consistency.

Tim

[Steffen]

I do not like this

Code:

<Part Primitive="1">

. A part is no primitive. I suggest to either use instead

Code:

<Primitive>

or

Code:

<File Primitive="1">
<File Part="1">

[Tim Gould]

Yes. Good idea. I will use <Part> or <Primitive>

[Tim Gould]

Version II

Code:

<{Part|Primitive} [Unofficial="1"]>
<Number>XXXX.dat</Number>
<Description>Brick  7 x 19</Description>
<LDrawOrg>Part UPDATE 2099-2</LDrawOrg>
[<License>GPL 4500.0</License>]
<Category>Weird brick</Category>
[<Keywords>this,brick,would,never,exist</Keywords>]
[<Help>LEGO have really started getting weird with their new moulds</Help>]
[<History>TG 2083</History>]
[<BFC>Certify CW</BFC>]
</{Part|Primitive}>

where things in square brackets appear only when required and {A|B} means A or B.

The License presently does not appear

I've attached an example.

[Michael Heidemann]

Why do you split parts primitives?
There is an entry what kind of file it is. To use it double is waisting space IMHO.
Also if you would define all entries a attributes then you can save also a lot of space.

<PartEntry Number="XXXX.dat" Description="Brick 7 x 19" LDrawOrg="Part UPDATE 2099-2" License="GPL 4500.0" Category="Weird brick" Keywords="this, brick, would, never, exist" Help="LEGO have really started getting weird with their new moulds" History="TG 2083" BFC="Certify CW">

Additionally I would add FullPath, Author and Comments attribute. If we also add an "IsOfficial" attribute that can be true or false we can add unofficial files to this database.

[Tim Gould]

Why do you split parts primitives?
There is an entry what kind of file it is. To use it double is waisting space IMHO.
Also if you would define all entries a attributes then you can save also a lot of space.

I'm of the opinion that more information is better than less And I think Steffen is right in saying they really are different types of entities.

<PartEntry Number="XXXX.dat" Description="Brick 7 x 19" LDrawOrg="Part UPDATE 2099-2" License="GPL 4500.0" Category="Weird brick" Keywords="this, brick, would, never, exist" Help="LEGO have really started getting weird with their new moulds" History="TG 2083" BFC="Certify CW">

Personally I prefer to use tags for most data, and attributes only for quirks (like Unofficial). It makes it more human readable.

Additionally I would add FullPath, Author and Comments attribute. If we also add an "IsOfficial" attribute that can be true or false we can add unofficial files to this database.

I'm iffy on Fullpath as it hardwires the file to a given computer. However perhaps sub-path (below %LDRAW) might be good though. Will think about it.

Good idea about Author.

I don't see what would go in Comment?

IsOfficial is already there in the LDrawOrg element, and in the absence of the Unofficial attribute.

[Tim Gould]

PS. Not that I want to base this on personal preference. My ideal goal would be to have a relatively simple official standard Parts.xml format decided by the LSC. But for now we might as well make a start

Tim

[Michael Heidemann]

A start has been done already.
If you have LDFind on your system you will have already such a file in the path:
%user%\AppData\Michael Heidemann\LDFind\1.0.0.0\LDrawLibraryPartHeaderData.xml

I am open for changes to this file if they make sense.

FullPath makes much sense, as if you have the file with the same name (one is official and one is unofficial) its very easy to catch the right file. And yes that is system depending. If it is not system depending, why do we ship mklist to the user??

The comments would be very helpful because we force the user to use it for critical information (what is missing at needs work parts)!

[Tim Gould]

A start has been done already.
If you have LDFind on your system you will have already such a file in the path:
%user%\AppData\Michael Heidemann\LDFind\1.0.0.0\LDrawLibraryPartHeaderData.xml

I still can't get it working. I sent you an email with the useless crap windows spit out. Could you email me a copy of your xml file.

I am open for changes to this file if they make sense.

FullPath makes much sense, as if you have the file with the same name (one is official and one is unofficial) its very easy to catch the right file. And yes that is system depending. If it is not system depending, why do we ship mklist to the user??

Good point. I'll add it

The comments would be very helpful because we force the user to use it for critical information (what is missing at needs work parts)!

I think you misunderstand me. I'm not sure what comments you mean. I do send !HELP but do you mean all type 0 lines before the first type 1-5?

Tim

[Michael Heidemann]

Attached the zipped xml file I use for LDFind.
These are the current official files.

I sent a new version by email to you. Please try. If that still does not work for you please tell me your system data.
You need NET 4.0, not a low end grafik card and Windows XP SP3 or later.

I hope it works now for you.

[Tim Gould]

I see. You use:

Code:

<PartEntry Partnumber="C:\Program Files (x86)\LDraw\parts\u9234c02.dat" Description="Train Track 12V Tapered Point Right Electric (Branching)" Author="Steffen [Steffen]" Filetype="Shortcut" IsOfficial="True" Keywords="" Category="Train" FilenameWithPath="C:\Program Files (x86)\LDraw\parts\u9234c02.dat" Filename="u9234c02.dat" />

Would you be averse to splitting most of those off to sub-tags. Looking at it I think a logical split would be:

Code:

<PartEntry Filetype="Shortcut" IsOfficial="True"> <!-- These describe the attributes of the entry -->
   <!-- Below is the data for the entry -->
  <Filename>u9234c02.dat</Filename>
  <Description>Train Track 12V Tapered Point Right Electric (Branching)</Description>
  <Author>Steffen [Steffen]</Author>
  <Partnumber>C:\Program Files (x86)\LDraw\parts\u9234c02.dat</Partnumber>
  <FilenameWithPath>C:\Program Files (x86)\LDraw\parts\u9234c02.dat</FilenameWithPath>  <!-- Why duplicated? -->
  <Category>Train</Category>
  [<Keywords></Keywords>]
</PartEntry>

As for another point:

Why <LDRAW-Library> and not <LDraw-Library>?

Tim

[Tim Gould]

Version III

I've taken into account Mike's file format, but changed it to improve (to me at least) human readability. I'd be really interested to hear thoughts about my style vs Mike's for readability.

Code:

<PartEntry    Filetype="Shortcut"    IsOfficial="True"    [IsAlias="True"]
    [IsPhysicalColour = "True"]    > <!-- These describe the attributes of the entry -->
   <!-- Below is the data for the entry -->
  <Filename>u9234c02.dat</Filename>
  <Description>Train Track 12V Tapered Point Right Electric (Branching)</Description>
  <Author>Steffen [Steffen]</Author>
  <FilenameWithPath>C:\Program Files (x86)\LDraw\parts\u9234c02.dat</FilenameWithPath>
  <Category>Train</Category>
  [<Keywords></Keywords>]
  [<Help></Help>]
  [<History>2012-03-30 [PTadmin] Official Update 2012-01</History>]
  [<BFC>CERTIFY CW</BFC>]
  [<Comments></Comments>]
</PartEntry>

Here the optional components only show if they have data. Similarly IsAlias and IsPhysicalColour will only show when true.

For my copy of the library the parts.xml file is 3.5Mb or one MP3 song.

Tim

[Travis Cobbs]

I feel that <Number> should be either <Name> (without .dat in the value) or <Filename> (with .dat in the value). I don't think it's appropriate to have non-numeric data in a field named "Number".

Note: LDView doesn't user parts.lst, and I don't see any reason why it would use this file either, but I'm throwing in my 2¢ anyway.

[Tim Gould]

PS. According to the Perl library XML::Simple, both attributes and sub-tags are equivalent. I'm not sure this is proper behaviour though. If it is, then Mike's file format and mine are equivalent representations of the same data.

Tim

[Tim Gould]

Already changed in V3.

What I would ideally like to see is the same basic format used for XML representations of the parts, with additional details added as necessary. Right now Mike and I seem to be using the same (or equivalent) representations. Once a de facto standard is in place we could look at officialising it, thus reducing the coding workload of future programs.

Tim

[Travis Cobbs]

Attributes and sub-tags are definitely not equivalent, although they are often used interchangeably.

[Tim Gould]

An xml standard allows one to write stuff like the below (it make parts.lst from parts.xml).

Code:

# use module
use XML::Simple;

# create object
$xml = new XML::Simple;

# read XML file
$data = $xml->XMLin("parts.xml");
@PEArray = @{$data->{"PartEntry"}};

foreach $Indx (0..$#PEArray) {
    print( sprintf("%-27s", $PEArray[$Indx]->{"Filename"})
       .$PEArray[$Indx]->{"Description"}."\n" );
}

[Tim Gould]

Functionally the only difference I can see is that attributes cannot be used to create sublists or sub-sub-elements. Semantically they're definitely different.

[Michael Heidemann]

Code:

Would you be averse to splitting most of those off to sub-tags.

I have choosen the current version because of filesize and reading speed. So at present I am not convinced to change that.

Code:

Why <LDRAW-Library> and not <LDraw-Library>?

Never thought about that I just needed a name. I am fine with both, as you normally do not have a look at this file.

[Allen Smith]

My file (sample attached*) uses Apple's Property List DTD, which makes it a one-liner to read in with Apple APIs.

It also has some structure artifacts from a more poorly-designed earlier version. I didn't want to break backwards compatibility.

Allen

* are we abusing the forum by faking the extension on all these files we're attaching?

[Michael Heidemann]

I just asked google for the question attributes vs elements.
One of the answers: http://stackoverflow.com/questions/28184...r-elements

The search i did: https://www.google.de/search?q=xml+attri...=firefox-a

As a result I think we should go with attributes.

[Tim Gould]

OK. Since I'm not expanding the keywords list out (which would require sub-tags) I'll use attributes.

Tim

[Tim Gould]

Cheers. So yours is more of a generic file rather than LDraw specific

PS. The forum isn't presently designed for .xml files because we haven't used them. But given the discussion I think they're appropriate.

[Michael Heidemann]

I suggest also to add the file date. With that information you only need to load and analyse those files that has changed, is not anymore in the library (should not happen) or is new in the directory. So update will be much quicker. I am going to add this in one of the next releases of LDFind.
Thanks for this idea goes to Alex Taylor

[Tim Gould]

I need to work out how to extract that information first I'll add it onto my Todo list.

Tim

[Steffen]

Hmmm, I am just coming back here after some time and would like to add some more thoughts
which just lately came to my mind. Let me share them here, they are just rough thoughts:
I am not sure if it really is a good idea to introduce a completely new syntax (here: XML) which carries
the same information which is already present in the header of LDRAW files.
Remember how much effort we need to keep the LDRAW header syntax clean, simple and consistent.
Remember the meta statements like 0 !HISTORY etc. in the files?
Would we officially launch this new XML format, then this will create a parallel universe.
Tool implementors will have to implement not 1, but 2 parsers: one for DAT/LDR/MPD files,
and one for the new XML format.
Wouldn't it be simpler to use the _existing_ LDRAW syntax also for the index,
and simply separate parts by something very simple, like a line of dashes? Something like

Code:

0 My Cool Part 1
0 !LDRAW_ORG Unofficial_Part
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt
------------------------------------------------
0 My Cool Part 2
0 !LDRAW_ORG Unofficial_Part
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt
------------------------------------------------
0 My Cool Part 3
0 !LDRAW_ORG Unofficial_Part
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt
------------------------------------------------
0 My Cool Part 4
0 !LDRAW_ORG Unofficial_Part
0 !LICENSE Redistributable under CCAL version 2.0 : see CAreadme.txt

[Tim Gould]

Would we officially launch this new XML format, then this will create a parallel universe.
Tool implementors will have to implement not 1, but 2 parsers: one for DAT/LDR/MPD files,
and one for the new XML format.

We do not need to write XML readers. They already exist. I have a <25 line file to read parts.xml, sort it at three different levels (category, description, number at the moment) and spit out parts.lst.

Wouldn't it be simpler to use the _existing_ LDRAW syntax also for the index,
and simply separate parts by something very simple, like a line of dashes? Something like

No. Because reading .dat files is a pain in the ass. This is the main reason I chose XML file format, and I presume it's the same for Mike and Allen.

[Steffen]

I've some 2 cents to add to this format suggestion:

(A)

I think that in the history the date needs to be better separated, i.e.
<history date="2012-01-01">blablablablabla</history>
or
<history><date>2012-01-01</date>blablablablabla</history>

(B)

I think the XML format should be minimized to be better human-readable. To me, it currently uses
too long strings. Shortening/simplification suggestions:
<PartEntry> simplify to <File> or <Entry>
Filetype="Shortcut" simplify to Type="Shortcut"
IsOfficial="True" simplify to Official="True"

I think further that <Description> is confusing and invites the user to put arbitrary text here
which better would be located in <Comments>. Thus I suggest to replace <Description> by
<Caption> or <Title> or <Name>

The syntax<BFC>CERTIFY CW</BFC> appears somewhat weird to me, a mix of LDRAW syntax with XML.
Why not use
<BFC Certified="True" Clockwise="True"/>
or, shorter,
<BFC Cert="True" CW="True"/>
here? Why carry over LDRAW syntax into the XML?

[Tim Gould]

I've some 2 cents to add to this format suggestion:

(A)

I think that in the history the date needs to be better separated, i.e.
<history date="2012-01-01">blablablablabla</history>
or
<history><date>2012-01-01</date>blablablablabla</history>

I agree this is the ideal situation. I chose the current compromise to combine human readibility and machine readibility, but perhaps I erred too much to the human side. I'll think about this...

(B)

I think the XML format should be minimized to be better human-readable. To me, it currently uses
too long strings. Shortening/simplification suggestions:
<PartEntry> simplify to <File> or <Entry>
Filetype="Shortcut" simplify to Type="Shortcut"
IsOfficial="True" simplify to Official="True"

Aside from PartEntry (which communicates that it is a part type entry as opposed to some future alternative type entry) I agree. But need to get Mike on side for this as these are now used by both codes.

I think further that <Description> is confusing and invites the user to put arbitrary text here
which better would be located in <Comments>. Thus I suggest to replace <Description> by
<Caption> or <Title> or <Name>

Yes I agree. Again I'd be happy to change if Mike is on board.

The syntax<BFC>CERTIFY CW</BFC> appears somewhat weird to me, a mix of LDRAW syntax with XML.
Why not use
<BFC Certified="True" Clockwise="True"/>
or, shorter,
<BFC Cert="True" CW="True"/>
here? Why carry over LDRAW syntax into the XML?

Same reason as history and help. But I agree <BFC Certified="true" Winding="CW" /> is much better.

Tim

[Michael Heidemann]

I fully agree with Tim!

[Michael Heidemann]

My intend, as i created such a xml database was the pure wish of the beta tester of LDFind to have the header information from the choosen part quickly displayed. As this database is currently at place stored that belongs to LDFind please go ahead and choose the format that you prefer. The xml that LDFind creates are not meant to be used in an editor, as they are purely "only" the compressed information from the files in the library and the other pathes you define in LDFind. If you change something in this xml file it only is valid until the next "Recreate Library".
So human readability is no option for me at all. Increasing reading speed and reducing filesize is my goal and nothing else.
The data stored in that file should also be minimized to the really needed for the user and of course for the application.
Currently I do not see a benefit to show the BFC state of the file nor that an application need that information in the database. I might be wrong on this.

This will be the format on release, as all for LDFind necessary informations are in that structure.
<Ldraw-Library>
<FileEntry Description= Author= Filetype= IsOfficial= Keywords= Category= FilenameWithPath= FileDate= NameEntry= >
</Ldraw-Library>

Future implementations might use also another library (an official one). As long as the above noted informations are in that file I can work with that.

[Roland Melkert]

No. Because reading .dat files is a pain in the ass. This is the main reason I chose XML file format, and I presume it's the same for Mike and Allen.

You could say the same about xml files (them being a pain), because any serious LDraw application needs to parse dat/ldr/mpd files at some point so a using it's core library reading the header stuff should also take only a handful of lines.

I'm just saying, not taking anybody's side or something.

[Roland Melkert]

OK. Since I'm not expanding the keywords list out (which would require sub-tags) I'll use attributes.

Why not separate? Isn't the whole point of this xml file so 3rd party software doesn't need to parse/disect stuff?

My 2cts:

Code:

<Part>
<Keywords>
  <Item>Classic space</Item>
  <Item>Space police</Item>
</Keywords>

<History>
  <Item date="yyymmdd" author="piet">Some kind of historic rant.</Item>
  <Item date="yyymmdd" author="sjaak">Another historic rant.</Item>
</History>
</Part>

As for elm vs attr discussion, I think the core rule is: An attribute needs to say something about an element's data.

[Tim Gould]

This is true. But XML is a more robust file format. It's easy to extend, well defined, and has a lot of work put into libraries and the likes. Which means changes to the file format are very easy to make.

Tim

[Tim Gould]

Hi Mike,

The advantage of having both the same is that I should be able to copy my parts.xml file over LDFind's xml file and use that instead. I might have, for example, removed all printed parts from parts.lst/parts.xml to avoid clutter and want to search through only parts.xml.

That's why I'm trying to match your file format.

Tim

[Tim Gould]

Why not separate? Isn't the whole point of this xml file so 3rd party software doesn't need to parse/disect stuff?

My 2cts:

Yes. And I will.

[Michael Heidemann]

Ok, I understand. But it is not done (now) with only changing the one file we are talking about in the moment.

I am searching first in the file "LdrawLibrarySearchWords.xml" and for showing the details I use "LDrawLibraryPartHeaderData.xml".

Doing that in another way would mean a major rewrite.

So for now your way will not work. Sorry.

I changed this format on release a little bit regarding the content, as all for LDFind necessary informations are in that structure.
<Ldraw-Library>
<FileEntry Description= Author= Filetype= IsOfficial= Keywords= Category= FilenameWithPath= FileDate= NameEntry= >
</Ldraw-Library>

FilenameWithPath now contains for official library parts an entry like follows: "%LDRAWDIR%\parts\3005.dat"

[Tim Gould]

Even if it can't do it now, by standardising we offer the option for the future With needlessly different formats it is much harder.

Some points:
* It's trivial but "L_D_raw-Library" would be better as that is the correct capitalisation of the 'brand'.
* What does NameEntry contain? Just "3005.dat" I guess
* What is the format of FileDate?

[Michael Heidemann]

Some points:
* It's trivial but "L_D_raw-Library" would be better as that is the correct capitalisation of the 'brand'.
"Ldraw-Library" is currently in but changing to "LDraw-Library" is easy and will be used in the next version. Sourcecode already changed.

* What does NameEntry contain? Just "3005.dat" I guess
If it is the file 3005.dat in the directory parts then yes. If it is a subpart s\filename.dat.

* What is the format of FileDate?
It is a string like all other values. I used the System.IO.File.GetCreationTime and System.IO.FileInfo.CreationTime
But currently I do not use that. It is for future use.

You can now download version 1.1.0.0!