LDMakeList V2.0

[Tim Gould]

http://code.google.com/p/ldmakelist/downloads/list

Please find LDMakeList 2.0, a replacement for makelist with added features and full support for >64 character description names. This updated version of LDMakeList contains a number of new options (-? will show them) and writes parts.xml in a format compatible with LDFind (although no support for parts.xml exists).

It can be run identically to the old "makelist" simply as LDMakeList -d or LDMakeList -n

If you do not wish to write Parts.xml please use the -x switch.

Please let me know if the compiled version works on your computer.

Tim

[Tim Gould]

For those of you with perl on your PC, the below code allows you to make your own custom sort by using parts.xml. It will write parts.lst to the screen so must be directed to parts.lst. Simply change $Level1, $Level2 and $Level3 to alter your sorting style.

Run as: perl SortParts.pl > parts.lst

Code:

use XML::Simple;

# Sort by category, then filename, then description
$Level1="Category";
$Level2="Description";
$Level3="NameEntry";

sub CmpParts {
    # Sort at first level
    $c=$a->{$Level1} cmp $b->{$Level1};
    if ($c==0) {
    $c=$a->{$Level2} cmp $b->{$Level2};
    if ($c==0) {
        $c=$a->{$Level3} cmp $b->{$Level3};
    }
    }
    return $c;
}

$xml=new XML::Simple;
$data=$xml->XMLin("parts.xml");
@PEArray = @{$data->{"FileEntry"}};

@PEArraySorted=sort CmpParts @PEArray;

foreach $P (@PEArraySorted) {
    print( sprintf("%-27s", $P->{"NameEntry"}).$P->{"Description"}."\n" );
}

[Max Martin Richter]

I got an error while starting:
"The program can not be started, because libgcc_s_dw2-1.dll is not available on the computer. Please reinstall the program, to solve the problem"

My System: Win7 Pro 64bit

/Max

[Tim Gould]

Grumble... right. So it's using a .dll secretly.

The updated version (just download again) should hopefully remedy the problem.

Tim

[Max Martin Richter]

Well, now I have another Problem. When I try to download the file, it seems, that there is something wrong on google. I get a file with 226 kb. Google says, that it must be about 1.5 Mb.
I think, it is the old file, because I get the same dll error as before.
(And no, it's nothing wrong with my cache in the browser, because I tested the download from a second machine with the same result :-( )

/Max

[Tim Gould]

Maybe it takes a while to settle the change...

Can you drop me an email: tgould.lego(AT@)gmail.com

[Travis Cobbs]

Just FYI, the new download runs for me, and appears to be the right size (1.6MB uncompressed).

[Tim Gould]

Thanks. I was being a bit naughty in the way I did things and apparently GoogleCode didn't like it (clearly it's a bug, but not one you should come across). Now I'm being better behaved.

Tim

[Willy Tschager]

Tim,

I'd like to substitute MKList in the AIOI with LDMakeList but this would require two things:

* That Chris does the same with the Official Parts Library
* The prog comes with a some sort of license or you have to sign the permission

w.

[Tim Gould]

Hi Willy,

I'd also wait for feedback on its success on 32bit and pre-Windows 7. So far I only know it works on two Win 7 64bit PCs.

The code is GPL licensed so distributing it should be no problem at all. You just need to know where the sourcecode is (http://code.google.com/p/ldmakelist/downloads/list).

Tim

[Philippe Hurbain]

Hi Tim,
Seems to be working fine on my Windows XP. Two little issues nonetheless:
- In usage, typo on unofficial: "-u or -U : include _U_noficcial part directory"
- hires primitives don't work in MLCad, file name in the list needs to be prefixed with 48\

[Tim Gould]

Thanks Philo. And it's especially great to have confirmation about Win XP. I've a few successful reports about Win 7 64bit.

Those changes have been made for 2.01 release. If anyone wants a partched version immediately they should drop me an email.

Tim

[Michael Heidemann]

Just tried this tool.
1) works good on win8 64bit
2) It is not mentioned where the files are created.
3) The mentioned help file is not in the zip file
4) Why duplication of the keywords in the xml file? (waste of diskspace)

[Tim Gould]

Just tried this tool.
1) works good on win8 64bit
2) It is not mentioned where the files are created.
3) The mentioned help file is not in the zip file
4) Why duplication of the keywords in the xml file? (waste of diskspace)

1) Awesome
2) .\=Same place as mklist. I'll add this.
3) I didn't notice there was meant to be a help file ! I've removed that message for now.
4) To match your usage, and 'standard practise'. Dropping the "keywords" attribute would only save 100kb (3%) in my parts.xml. The whole library is 100Mb.

[Michael Heidemann]

IMHO does not make sense.

I have the following thoughts.

1) Currently LDFind works only based on the folders you mention in the option dialog. So I have to take care that all matched in the resultlist of LDFind will give a file back.

2) For the future I can think of asking the user wheather he likes to use the build in library manager or only the files mentioned in the parts.xml (if i can find such a file). So if in the LDRAWDIR is a file called parts.xml the user is asked: Do you like to proceed with your current settings from parts.xml or define your own settings?.
Something in that way.
If parts.xml is the source, then i only have to build the searchword database and lets go.
Going this way you can choose the layout of the parts.xml just like you want to have it. I can read it.

Just my thought for now.

[Tim Gould]

2) For the future I can think of asking the user wheather he likes to use the build in library manager or only the files mentioned in the parts.xml (if i can find such a file). So if in the LDRAWDIR is a file called parts.xml the user is asked: Do you like to proceed with your current settings from parts.xml or define your own settings?.
Something in that way.
If parts.xml is the source, then i only have to build the searchword database and lets go.
Going this way you can choose the layout of the parts.xml just like you want to have it. I can read it.

Just my thought for now.

OK I think it makes sense to keep as much the same as possible, but definitely prefer separating the keywords in the .xml file (IMHO Steffen was right). So I'll drop the keywords attribute and use <keyword> sub tags, perhaps inside <keywords> tags.

[Michael Heidemann]

I already managed to implement the above said.

Suggestion:
For the part.xml I think it make sense for the following:
<FileEntry .....>
<Author Realname="" Username="PTadmin">
<Keywords>
<Keyword>word1</Keyword>
<Keyword>word2</Keyword>
</Keywords>
<History Date="2004-11-06" Author="PTadmin" >Official Update 2004-04</History>
<FileEntry>

This means the Author and Keywords attributes for FileEntry will go into elements.
I would eliminate the "[]" brackets from the username as they are not part of the username.
Also the "{}" brackets if they occur.

As we have now a version out we have to care also about versions of this.
We should introduce a new element at start
<Version Number="2" />
If version is not present, then it is version 1

I'll go now and implement the version question and if it is not 1 then part.xml can not be used currently.

Edit:
Just finished LDFind adjustment for versions.
If the version tag is present and the Number is another than 1 then LDFind will not parse the parts.xml file.

[Tim Gould]

All good ideas. I won't make these changes until 2.1 though. For 2.01 I'll simply keep patching mistakes as people find them.

And I'll be sure to send you an example parts.xml before going public so we can liase on final format.

Tim

[Michael Heidemann]

Code:

And I'll be sure to send you an example parts.xml before going public so we can liase on final format.

That makes definitely sense

[Michael Heidemann]

I just found a bug in this version:
your entry:

Code:

<FileEntry Filetype="Part"  IsOfficial="True"
NameEntry="30462.dat"
Description="Animal Dinosaur Body Legs (Needs Work)"
Author="Stan Isachenko [angmarec]"
FilenameWithPath="%LDRAWDIR%\PARTS\30462.dat"
Category="Animal"
Keywords="Adventurers, Harry Potter"
>
<Keyword>Adventurers</Keyword><Keyword>Harry Potter</Keyword>
<BFC Certify="True" Winding="CCW" />
<History Date="2011-11-06" Author="{LEGO" >Digital Designer} Original part shape</History>
<History Date="2011-11-07" Author="[angmarec]" >File preparation for LDraw Parts Tracker</History>
<History Date="2012-08-09" Author="[PTadmin]" >Official Update 2012-02</History>
</FileEntry>

My concern at present is for the history lines. As I already mentioned earlier I prefer to exclude the delimiter "{}[]".
Also it seems that you did not catch the {} delimiter.
Below the correct data. IMHO.

Code:

<FileEntry Description="Animal Dinosaur Body Legs (Needs Work)" Author="Stan Isachenko [angmarec]" Filetype="Part" IsOfficial="True" Keywords="Adventurers, Harry Potter" Category="Animal" FilenameWithPath="%LDRAWDIR%\parts\30462.dat" FileDate="09.08.2012 11:38:34" NameEntry="30462.dat">
        <BFC Certify="True" Winding="CCW" />
        <History Date="2011-11-06" Author="LEGO Digital Designer">Original part shape</History>
        <History Date="2011-11-07" Author="angmarec">File preparation for LDraw Parts Tracker</History>
        <History Date="2012-08-09" Author="PTadmin">Official Update 2012-02</History>
    </FileEntry>

[Michael Heidemann]

Now I remember the second reason why I like attibutes vs elements.
Writing is easy, but reading the file correctly is much more effort!

Edit:
Once you have understand it is not that difficult

[Tim Gould]

Although I didn't mention it, this was a known bug. I figured noone would be supporting <history> until the next release where I planned to fix it. I really should have said that once you started adding support.

It now works correctly:

Code:

<FileEntry Filetype="Part"  IsOfficial="True"
NameEntry="95342.dat"
Description="Animal Chicken"
Author="Magnus Forsberg [MagFors]"
FilenameWithPath="%LDRAWDIR%\PARTS\95342.dat"
Category="Animal"
>
<BFC Certify="True" Winding="CCW" />
<History Date="2011-09-08" Author="LEGO Digital Designer" External="True" >Original part shape</History>
<History Date="2011-09-10" Author="MagFors" >File preparation for LDraw Parts Tracker</History>
<History Date="2011-12-29" Author="PTadmin" >Official Update 2011-02</History>
</FileEntry>

Tim

[Michael Heidemann]

Ah, that's great news.
Currently I am in the last stage of a new version that supports parts.xml.
If parts.xml will be found in the ldraw base folder LDFind asks on startup whether you want to use that file or not.
If you use parts.xml then you are limited in LDFind. you can not add other folders to search for files, as parts.xml is the base for all searches then.

[Roland Melkert]

FilenameWithPath="%LDRAWDIR%\PARTS\95342.dat"

Bit of a nitpick, but 'path' is already including the filename. forexample:

"c:\this is\some very\cool file.txt"

path: "c:\this is\some very\cool file.txt"
filename: "cool file.txt"
folder: "c:\this is\some very" or "c:\this is\some very\" depending on os I think, i'm not really sure about that.

On a side note: are those values always using '\' or will that be os depended? It might be a good idea to document that ether way (also using the trailing \ yes/no).

Just my 2cts

[Tim Gould]

Hi Roland,

Firstly, Mike will have to answer about why he chose that name, but to me it definitely makes sense to start from %LDRAWDIR%.

Presently the type of slash is system dependent (in principle LDMakeList is OS independent, you just need to change a #define to reverse the slash). But as I recall any valid LDraw code should be able to read either \ or / as a separator for parts. The filename rules for parts are pretty strict to avoid problems. So it shouldn't matter which type is used

So the rule is: parts.xml will have the slash of the OS it was created on, and it will always feature trailing slashes to avoid problems with badly defined %LDRAWDIR%.

Tim

[Tim Gould]

Awesome. Let me know before you release it so I can upload v2.01 beforehand. Then you can recommend people update to get the maximum benefit from parts.xml.

Tim

[Travis Cobbs]

Looking at the sample Michael posted from LDMakeList, I see that the keywords show up both as attributes (unparsed) and sub-tags (parsed). Was this intentional? Also, if it was intentional, what happens to the attribute value if there are multiple !KEYWORDS lines in the part file? Do they get concatenated together, with ", " as the "glue"? (I ask this because my shell script to generate an HTML parts list initially had problems with multiple !KEYWORDS lines in part files.)

[Tim Gould]

Yes, I use commas as the glue when reading multiple lines (which I use unparsed for the 'keywords' attribute) and then split on all commas for the parsed 'keyword' sub-tags. I've not noticed it causing any bugs, but perhaps closer inspection would turn some up. One plus is that I can very easily write a script to generate a file of all the keywords to check which I'll do ASAP.

Tim

[Michael Heidemann]

Just discovered another bug?
I do not recognize a Category call "~Electric".

Code:

<FileEntry Filetype="Subpart"  IsOfficial="True"
NameEntry="932.dat"
Description="~Electric Mindstorms NXT Cable Segment"
Author="Philippe Hurbain [Philo]"
FilenameWithPath="%LDRAWDIR%\PARTS\932.dat"
Category="~Electric"
>
<BFC Certify="True" Winding="CCW" />
<History Date="2012-03-30" Author="[PTadmin]" >Official Update 2012-01</History>
</FileEntry>

Edit: And the next one
BFC Certify without a value. Only true or false should be there as value IMHO.

Code:

<FileEntry Filetype="Part"  IsOfficial="True"
NameEntry="3960p02.dat"
Description="Dish  4 x  4 Inverted with Black Spiral Pattern"
Author="Franklin W. Cain [fwcain]"
FilenameWithPath="%LDRAWDIR%\PARTS\3960p02.dat"
Category="Dish"
Keywords="Time Twisters, whirlpool, hypno-disk, radar dish, round, space"
>
<Keyword>Time Twisters</Keyword><Keyword>whirlpool</Keyword><Keyword>hypno-disk</Keyword><Keyword>radar dish</Keyword><Keyword>round</Keyword><Keyword>space</Keyword>
<BFC Certify="" Winding="NOCERTIFY" />
<History Date="2002-08-18" Author="[PTadmin]" >Official Update 2002-04</History>
<History Date="2004-09-15" Author="[PTadmin]" >Official Update 2004-03</History>
<History Date="2007-06-30" Author="[PTadmin]" >Header formatted for Contributor Agreement</History>
<History Date="2008-07-01" Author="[PTadmin]" >Official Update 2008-01</History>
<History Date="2010-05-09" Author="[cwdee]" >Remove CATEGORY Round</History>
<History Date="2010-07-05" Author="[PTadmin]" >Official Update 2010-02</History>
</FileEntry>

[Michael Heidemann]

I still needs to parse the elements from parts.xml. So far i only read the attributes. As this all is already working I assume to finish my part within the next 5 days. As soon as i think i am ready, I'll send the last build to you for test purpose.

[Michael Heidemann]

That the keywords show up twice is bad.
Some people like the elements more than attributes.

My (LDFind) first (and current) implementation for keywords is as attribute.
I also just add the two lines with comma as separator. I had not problems with this over the years.
There is a function in .NET called string.split that I always use for such things. So just adding the <additional parameters> with a "," inbetween works great (0 !<META command> <additional parameters>).

[Michael Heidemann]

Hi Roland,
I have not yet seen any explanation like you suggest in a norming document. I see many words discribing the different portions of a full qualified path name. If I have choosen a misleading name please suggest a better name. If the better name is not better because it is not clear enough, I leave it like it is.

As I am coding on windows, i assume that LDFind gets problems if used with a parts.xml with "/" delimiter. But I will check that. Should not be too difficult to solve

As I did not configured my system with LDRAWDIR myself, but with the AIOI (currently set to "C:\Program Files (x86)\LDraw" on my machine), I assume that the LDRAWDIR is fixed _without_ trailing delimiter.

[Roland Melkert]

And the next one
BFC Certify without a value. Only true or false should be there as value IMHO.

Or leave the element out alltogether if there it's nocertify.

Also why don't you encapsulate the history and keyword elements? Nesting them would make reading those lists somewhat easier because you will know the maximum number of items beforehand.

[Roland Melkert]

I have not yet seen any explanation like you suggest in a norming document

Maybe this one ?

msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx

As I am coding on windows .....

I'm under the impression this file is to become a global configuration file? . So you might want to force a certain format inside the xml no matter the os it's located on. That way tools can assume a certain format when reading the file and don't have to jump through all kinds of loops in order to use it.

[Michael Heidemann]

That is not necessary.

[Michael Heidemann]

Thanks for the link. Yes, that is interesting stuff.
But I still ask for maybe better naming then

Code:

I'm under the impression this file is to become a global configuration file?

You might be right here, but that was not my intend as I started the xml files I use for LDFind. Therefore I just used the names I liked and can remember what they are doing.
These files have never been ment to be read by human but xml was very quick in reading and writing the values. Therefore I used that format. LDFind can be used with the parts.xml but it is not needed. If you decide to use parts.xml then parts.xml delivers the data and not the library itself (for searching).

[Tim Gould]

That the keywords show up twice is bad.
Some people like the elements more than attributes.

Hopefully this can change in the next full document version (where I will drop keywords= ) as we allow some drift between my format and your internal format. Reading the <keyword> list and converting it to a single keywords attrib should be extremely easy.

[Tim Gould]

I'm under the impression this file is to become a global configuration file? . So you might want to force a certain format inside the xml no matter the os it's located on. That way tools can assume a certain format when reading the file and don't have to jump through all kinds of loops in order to use it.

I see that as more a job for the LSC. For now I'm pretty sure the major tools read both types of slash. As, IMO, they should. I'm not averse to locking it down to one or the other, but I'd like bigger input than my decision

[Tim Gould]

Hi all,

Version 2.01 is now out. Download here. If you are using LDFind or parts.xml in some customised way I highly recommend getting this update.

Changes 2.01
- Now defaults to . if LDRAWDIR not specified
- Correctly reproduces filenames 48\XXXX.dat
- More verbose about where the files go .\parts.lst .\parts.xml
- No longer refers to non-existent help file
- Automaticaly fixes non UTF-8 characters for .xml file
- Fixed bug in authro proccessing for History entries
- Strips leading ~ in categories
- Adds a version number "1" and subnumber "01"
- Improved handling of BFC

Tim

[Michael Heidemann]

LDFind that can use this parts.xml file is at present only delivered to some of you guys.
You will need LDFind version 1.1.0.2 latest builds to use it!! Still not available for the big audience!

[Michael Heidemann]

Today I noticed that LDMakeList version 2.12 generates a parts.xml that can not be used with LDFind, because there is one line that prevents the parts.xml to be a well-formed xml file.

<OS-Properties Style="DOS" />

If you entcounter problems with the parts.xml generated from LDMakeList please look into the parts.xml and if you have such a line, please just delete it.

[Tim Gould]

Thanks for pointing that out, Mike. I messed up my .xml and output a bad file.

Version 2.13 (and future versions) do not have this problem. So download the latest version and you won't need to edit anything.

Tim

[Michael Heidemann]

Thanks for your very quick response and bug fix.

This should help the guys that encountered those problems.

cu
Mike

[Magnus Forsberg]

I still get a file containing that line of code,
<OS-Properties Style="DOS" />

If I check for version (ldmakelist -v), it says "Version 2.13", but the xml-file say: version number ="2"
Is that correct?

Could I make a search in "Unofficial/p" ? How?

[Tim Gould]

Hi Magnus,

Firstly, the XML file has a different versioning system (V2) to LDMakeList (V2.13) and LDFind (V1.3.2.1). Mike and I try to keep our output cross-compatible so versioning the XML output helps to ensure that things are working out.

The problem wasn't the line, but where it was. It is bad XML (hence the errors) to have multiple "root" nodes and my placing of the line caused that problem. It is now contained within the root node (<LDraw-Library>) and will thus be ignored by Mike's program.

The recent problem was entirely my fault and was caused by me making a bad change at my end. I expected it to be ignored by LDFind but never actually tested it.

Tim

[Michael Heidemann]

The current place where the line in question is located now is perfect.

This error has gone. I have checked it here on my end and got no further error. Thanks Tim for quick action.

[Magnus Forsberg]

Thanks for your answers.

How about my last question. How do I add the unofficial primitives to my list?

[Michael Heidemann]

I just had a look into the Readme.txt from LDMakeList.

LDMakeList
Use -? for help, -v for version
Valid options are:
-d or -D : sort by _D_escription (default)
-n or -N : sort by file_N_ame
-p or -P : sort by description (_P_atterns at end of file)
-s or -S : use _S_hort descriptions*
-m or -M : include pri_M_itives in the list
-u or -U : include _U_nofficial part directory
-a or -A : exclude _A_liases from the list
-r or -R : _R_emove duplicate entries (experimental)
-o or -O : Include _o_fficial parts only
-l[DIRNAME] or -L[DIRNAME] : Read from _L_Draw director DIRNAME
-h[C] or -H[C] : -H_ide descriptions starting with character(s) [C]
-i[C] or -I[C] : str_I_p leading character(s) [C] from descriptions
-x or -X : do not write parts._x_ml file

Seems that you need to combine -u and -m and do not use -o.

[Tim Gould]

Actually, I notice that that combination does not read /unofficial/p/ or /unofficial/p/48/

I've uploaded a 2.13beta that should fix this problem. Please let me know if you encounter any problems as this was a _very_ quick and largely untested change.

Tim