Tutorial: Adding unofficial LDraw parts to MLCad

[Willy Tschager]

Guys,

attached you'll find the first draft of my new tutorial: Adding unofficial LDraw parts to MLCad.

I would appreciate some feedback. Also your tips and tricks when working with unofficials are welcome.

w.

[Santeri Piippo]

I disagree with the entire notion of installing the complete unofficial parts "library". It's not a library, rather a collection of WIP parts on the parts tracker. The website page for the unofficial parts which you even took a screenshot of goes out of its way to mention first off that these parts are liable to change without warning and it's generally much better to get them one-by-one. I don't think we want to encourage the ultimately reckless behavior of getting all the unofficial parts. We'll just get users complaining that their models break because they used held parts...

[Chris Dee]

I think Willy has been pretty explicit in identifying most of the risks, but it would be nice to put that standard disclaimer at the very top of the document.

I have some comments:

1) The zip file is normally refreshed every 5 minutes, but if only there have been any changes. [Sometimes it also gets refreshed manually in between, if I've done a bunch of renames].

2) I may have missed it, but there should be a recommendation to never, ever put the Unofficial "library" ahead of the Official library in the search path. This can potentially break existing models, and although it's rare and hard to explain, it might be worth adding.

a) This can happen because sometimes we change already official files to move code in or out of their parents. Whilst at Parts Update time we ensure that fixes to all dependent files are released together, they won't all necessarily exist on the Parts Tracker at a particular point in time. The most common case is when an existing official part is subparted to allow for patterning. The symptoms depend on what has been changed and in what order the author adds the new subpart and its parent to the Parts Tracker. If the parent is added first there will be missing surfaces until the subpart is added.

b) If the interface between a subpart and its parent get changed then even adding the subpart first may cause problems if it takes a while to upload all it related parents.

3) Sometime when a bunch of related parts and subparts get added with bad names, the renaming and re-organising can span across several interations of the file refresh. In extreme cases I may need to do something else (like eat) part way through. In these cases, even new parts the unofficial area can appear broken for extended period of time.

4) Unzipping a zip archive into an existing directory is an additive process. I think you should recommend that users clean out or delete the unofficial directory each time they unzip a new download. Otherwise they risk accumulating files that have later been renamed and will never appear in the official library with that number.

5) All files issued in a Parts Update are programmatically checked to ensure that they have valid LDraw headers, including the License statement and are authored by someone who has affirmed the Contributor Agreement. I have not implemented that check at upload to time for fear of novice authors might lose interest in failing those checks on an otherwise perfect part.

6) I can't remember if I have ever checked that we have nothing left on the Parts Tracker from any non-CA authors. If I find them I usually hold for redevelopment, but that doesn't stop them being included in the zip file. We shouldn't be distributing those (even as one-by-one downloads, really).

[Willy Tschager]

You've got some very good points I'm gonna include.

w.

[Magnus Forsberg]

I think you need to adress the issue of duplicated files somehow.

The unofficial library also contains all the updated, new versions of official parts being reworked.
As far as I know the scan order in mlcad.ini defines what version you will see. An updated part in the unofficial folder, will only be seen in MLCad if you remove it from the ordinary parts folder too. How do LDView decide which file to show? LPub? LDMakeList?

I do it like that. I always move the official version to a third folder "old parts", in a retained folder structure.
The downside is that every release of a Parts Update means that I have to rearrange all my files. It's either that or dump all the unofficial parts in the ordinary Parts folder. I want to only see and work with the latest versions of all the parts.

Or is there an another way?

That's also why I never use the "parts update".exe. It doesn't only adds the new parts. It also adds all the official files.
I don't want to have the files that are not ready for release, reinstated in my official folder.

[Santeri Piippo]

6) I can't remember if I have ever checked that we have nothing left on the Parts Tracker from any non-CA authors. If I find them I usually hold for redevelopment, but that doesn't stop them being included in the zip file. We shouldn't be distributing those (even as one-by-one downloads, really).

There are no files with the explicit Non-CA license line in the unofficial parts archive ldrawunf.zip. There are, however, a total of 40 files without the explicit CA license header. I don't know about William J. Wood or Paul Izquierdo Rojas but the others look rather CA-affirmed to me...

Code:

[~/desktop/ldrawunf] > for f in $(find -type f); do if [ -z "$(grep CAreadme $f)" ]; then echo -e "$f: $(head -n1 $f)\n$(head -n3 $f |tail -n1)\n"; fi; done    
./parts/32562.dat: 0 ~Electric USB Camera Bottom
0 Author: William J. Wood

./parts/73311.dat: 0 _Bike  2 Wheel Motorcycle with Red Wheels (Complete) Lt Grey
0 Author: Chris Dee

./parts/3496.dat: 0 Tap  1 x  2 Base
0 Author: Tore Eriksson

./parts/x224.dat: 0 ~Technic Rubber Belt element R13
0 Author: Philippe Hurbain [Philo]

./parts/x51ps1.dat: 0 Technic Dome 6 x 6 x 5 2/3 No Eye Pattern
0 Author: William J. Wood

./parts/32561.dat: 0 ~Electric USB Camera Top
0 Author: William J. Wood [wjwood]

./parts/41660.dat: 0 Technic Bionicle 5 x 5 Shield with Circular Saw (MOCKUP)
0 Author: Michael Heidemann [mikeheide]

./parts/x447.dat: 0 ~Train Wheel Old with  4 Studs and Traction Teeth
0 Author: Niels Karsdorp

./parts/41663.dat: 0 Technic Bionicle 5 x 5 Shield with Dual Blasters (MOCKUP)
0 Author: Michael Heidemann [mikeheide]

./parts/x51ps2.dat: 0 Technic Dome 6 x 6 x 5 2/3 Eye Pattern
0 Author: William J. Wood

./parts/x223.dat: 0 Technic Rubber Belt (open file for usage guide)
0 Author: Philippe Hurbain [Philo]

./parts/3004p07.dat: 0 Brick  1 x  2 with Ice Cream Pattern
0 Author: Michael Heidemann

./parts/73538.dat: 0 _Bike  2 Wheel Motorcycle with Red Wheels (Complete) White
0 Author: Chris Dee

./parts/3278.dat: 0 ~Tap  1 x  2 Spout (Needs Work)
0 Author: Tore Eriksson

./parts/3496c02.dat: 0 Tap  1 x  2 with Chrome Spout
0 Author: Steffen [Steffen]

./parts/43460.dat: 0 ~Subpart of Electric RC Race Buggy Steering Unit (Left)
0 Author: Guy Vivan

./parts/x228.dat: 0 ~Technic Rubber Belt straight
0 Author: Philippe Hurbain [Philo]

./parts/3496c01.dat: 0 Tap  1 x  2
0 Author: Tore Eriksson

./parts/32561c01.dat: 0 Electric USB Camera with TransBlack Bottom and LtGrey Focus Ring
0 Author: William J. Wood

./parts/2982c02.dat: 0 Electric Light Sensor (Complete Assembly Shortcut with Cable)
0 Author: Kevin L. Clague

./parts/73327.dat: 0 _Bike  2 Wheel Motorcycle with Red Wheels (Complete) Blue
0 Author: Chris Dee

./parts/x225.dat: 0 ~Technic Rubber Belt element R30.5
0 Author: Philippe Hurbain [Philo]

./parts/x226.dat: 0 ~Technic Rubber Belt element R47
0 Author: Philippe Hurbain [Philo]

./parts/3004p70.dat: 0 Brick  1 x  2 with Fire Logo and White Diagonal Stripes Pattern
0 Author: Michael Heidemann [mikeheide]

./parts/x227.dat: 0 ~Technic Rubber Belt element R9
0 Author: Philippe Hurbain [Philo]

./parts/4719c03.dat: 0 ~Minifig Bicycle Minifig Rider (Shortcut)
0 Author: William Howard

./parts/5306c01.dat: 0 Electric Wire with End Connectors  2 x  2 (Complete)
0 Author: Kevin L. Clague

./parts/32552.dat: 0 Technic Bionicle Weapon Double-Edged Sword
0 Author: Robert Sexton

./parts/s/32552s01.dat: 0 ~Technic Bionicle Weapon Double-Edged Sword Handle
0 Author: Robert Sexton

./parts/s/32552s02.dat: 0 ~Technic Bionicle Weapon Double-Edged Sword Blade
0 Author: Robert Sexton

./parts/s/x51s01.dat: 0 ~Technic Dome 6 x 6 x 5 2/3 No Pattern
0 Author: William J. Wood

./parts/s/x51s02.dat: 0 ~Technic Dome 6 x 6 x 5 2/3 Side
0 Author: William J. Wood

./parts/73419.dat: 0 _Bike  2 Wheel Motorcycle with Red Wheels (Complete) Yellow
0 Author: Chris Dee

./parts/2977c02.dat: 0 Electric Rotation Sensor (Complete Assembly Shortcut with Cable)
0 Author: Kevin L. Clague

./parts/43459.dat: 0 ~Subpart of Electric RC Race Buggy Steering Unit (Right)
0 Author: Guy Vivan

./parts/32561c02.dat: 0 Electric USB Camera with TransBlue Bottom and Yellow Focus Ring
0 Author: William J. Wood

./parts/x325.dat: 0 ~Electric Camera USB Focus Ring
0 Author: William J. Wood

./parts/73429.dat: 0 _Bike  2 Wheel Motorcycle with Red Wheels (Complete) Red
0 Author: Chris Dee

./parts/30185c01.dat: 0 Window Bay  3 x  8 x  6 with Smoke Glass
0 Author: William Howard [WilliamH]

./parts/33230.dat: 0 Brick 16 x 16 1/4 Circle
0 Author: Paul Izquierdo Rojas

[Chris Dee]

That's also why I never use the "parts update".exe. It doesn't only adds the new parts. It also adds all the official files.
I don't want to have the files that are not ready for release, reinstated in my official folder.

I don't understand. If you mean by "parts update".exe the Windows installer for the complete library (e.g. LDraw1401.exe), that only installs parts that are ready for release, but you are correct that it re-installs the entire Official Library. It does NOT install files that are not ready for release.

[Philippe Hurbain]

Just for the record, for years I have used the brute force, simple minded method: when there is an update, I unzip complete.zip over my LDraw folder, then on top of it, unzip current ldrawunf.zip (that I sometimes refresh between updates). No complex configuration to do, and I have yet to see ill effects of this method!
Granted, the thing I do is to make mpd of valuable models including unofficial parts, using mpdwizard and a reference folder containing complete.zip.

[Chris Dee]

Thanks for producing this and saving me some work in identifying those files. I'll work through the list.

[Magnus Forsberg]

Yes.
I know that it doesn't install any parts that are not ready for release.
It re-installs the entire current official library, and as I tried to explain, I don't want that. Only the new and updated parts.

Maybe I should do it like Philo does. All files in one library.
But I sometimes want to do a comparative study of new and old files.....

We have LDFind and LDStructure to help us in finding files. I wish we had a LD Library Manager.

[Roland Melkert]

Maybe you could use svn or something to 'revert' those unwanted files and only keep the new ones. You need to run an svn server for that though and incrementally commit every library update from the first one or couple of last ones from a previous full one.

Would be cool if we had a svn.ldraw.org

[Chris Dee]

Yes.
I know that it doesn't install any parts that are not ready for release.
It re-installs the entire current official library, and as I tried to explain, I don't want that. Only the new and updated parts.

For each Parts Update there is a zip file that contains just the new and updated parts as described (for 2014-01) here. The file is named lcadYYNN.zip, and all past versions are still available. Are you saying that your would prefer a Windows install file rather than a zip archive?

[Steffen]

In fact, I am locally using git to keep the history of my LDRAW files (and models and scenes).

I have a git repository containing the initial parts release,
and then have applied each parts update one after another.
The repository reflects the whole lifecycle of a file.

When a new parts update comes out, I apply it and commit it as a "change" to all the affected *.dat and other files.

I do the same for unofficial files coming from the PT,
and my models, and my scenes composed of models.

I also would LOVE to see a revision controlled PT DAT files server
(we could even use github etc.), but the problem is that then the review etc. mechanisms will not be available,
and in fact a new PT would have to be written.
Question is if it would be worth the effort.
In an ideal setup, I would like to be able to submit part updates to the PT via a git push
instead of a parts upload via web interface.

- just my personal opinion

Steffen

[Magnus Forsberg]

I'm satisfied with a zip file. I like to keep it simple.

I understand the concept of a version handling software, but please keep in mind that not all of us are programmers. I'm not.
It's always interesting to read the talks. I sometimes learn things I didn't knew.

But I would like to ask my original question again.
How do software know which file to show, if it is present in both the official and unoffical folders?
Do they all ignore the unofficial version?

MLCad does something strange. It shows the official file, but with the description from the unofficial file.

[Chris Dee]

Headers now corrected in all those files.

[Willy Tschager]

Digging a little bit deeper I came across these posts:

http://news.lugnet.com/cad/dev/?n=10742

Additionally, LDView is hard-coded to look in the <LDrawDir>/Unofficial/Parts
and <LDrawDir>/Unofficial/p directories when it doesn't find a part in its
search path. It then attempts to download unofficial parts from the part
tracker into that directory if it can't find the part (assuming the user hasn't
disabled this feature).

It could be argued that the user should be able to configure where unofficial
parts get downloaded to, but I'm not sure I'll ever implement that. The way it
is now basically produces a de-facto standard for anyone using LDView, and
adding a configuration option for this would defeat that.

As it happens, BrickSmith on the Mac has these directories in its search path
also (which was purely coincidental; Allen and I didn't talk about it, and
neither of us was aware of the other's use of that directory at the time we did
our implementation).

Willy has a tutorial and sample MLCad.ini file that sets up MLCad to look in
these Unofficial directories.

--Travis

http://news.lugnet.com/cad/dev/?n=11026

[SCAN_ORDER]
1 = SHOW <LDRAWDIR>Parts
2 = HIDE <LDRAWDIR>Parts\s
3 = HIDE <LDRAWDIR>P
4 = HIDE <LDRAWDIR>P\48
5 = SHOW <LDRAWDIR>Unofficial\Parts
6 = HIDE <LDRAWDIR>Unofficial\Parts\s
7 = HIDE <LDRAWDIR>Unofficial\P
8 = HIDE <LDRAWDIR>Unofficial\P\48

The entries I bolded above are wrong/unnecessary. The HIDE will prevent them from showing up in the parts list, but they’ll still be found when used properly inside other files, since the 48/ or s/ is included in the referring part, and you don’t want them to be found incorrectly if the part is broken.

--Travis

A potential problem could rise with LDView downloading to the VirtualStore on Vista or higher. Looks like I have to address this some how. Thoughts?

w.

[Roland Melkert]

If you know the ldraw folderis going to be changed by other software, you should not put it in 'program files' but in some user location (e.g. d:\LDRaw) or inside c:\programData, but you might need to set general write access to it's folder (programData is read only by default to other users).

In short anything you put in program files is expected to be static after installation by recent windows versions. And programData is for (dynamic) stuff shared between multiple programs. And the users/<name> tree is for user specific stuff (config per user etc).

[Travis Cobbs]

I'm not sure what the AIOI uses right now, but in my opinion, the default location for the LDraw directory should be in %ProgramData% (C:\ProgramData by default on recent Windows versions). I believe that is the equivalent of the "All Users" location from earlier Windows version. If it goes there, the virtual store doesn't come into play, and yet it is still accessible to all users of the computer. If the AIOI has an "install for all users" option that can be disabled, then when it is disabled, I think the user's own directory is where the LDraw directory should go.

[Steffen]

there are 2 more potential suitable locations on Windows:

C:\Users\Public\Documents\LDRAW
which is the "Documents" folder shared between users

C:\Users\YOURUSERNAME\AppData\Local\LDRAW
which is the "persistency" data folder for the applications of user YOURUSERNAME


I think everybody can decide on his own, but we need to decide which default to use in the AIOI.

I personally still have my LDRAW files nowhere there, but at
D:\LEGO\LDRAW

[Steffen]

note these Windows environment variables as well:

ALLUSERSPROFILE=C:\ProgramData

LOCALAPPDATA=C:\Users\YOURUSERNAME\AppData\Local

ProgramData=C:\ProgramData

PUBLIC=C:\Users\Public

[Willy Tschager]

Many thanks for all your valuable input. I'll give "C:\ProgramData" a try. The AIOI sets the base-path for almost all progs as well as the LDRAWDIR system variable. I'll have to check for LEO and what happens to SR3D and its "Conns" and "Image" folders.

w.

[Willy Tschager]

I'm not sure what the AIOI uses right now, but in my opinion, the default location for the LDraw directory should be in %ProgramData% (C:\ProgramData by default on recent Windows versions). I believe that is the equivalent of the "All Users" location from earlier Windows version. If it goes there, the virtual store doesn't come into play, and yet it is still accessible to all users of the computer. If the AIOI has an "install for all users" option that can be disabled, then when it is disabled, I think the user's own directory is where the LDraw directory should go.

I made some test with c:\ProgramData but the Parts.lst file written by MLCad ended up in:

C:\Users\<User>\AppData\Local\VirtualStore\ProgramData\LDraw

furthermore LDMakeList.exe needs to "Run as Administrator" in order to write a Parts.lst. Both restrictions are not an option 'cos it doesn't solve the initial problem. I'll therefore go for:

C:\Users\Public\Documents\LDraw

w.

[Willy Tschager]

I have moved the Parts Library in the AIOI from the LDraw root to the Public Documents folder located in "C:\Users\Public\Documents”. I would appreciate if you could perform some tests on your machines before I upload the installer to the site. The AIOI 2014-01 version 2 can be found at:

http://www.holly-wood.it/tmp/LDraw_AIOI_...bit_v2.zip

The toughest part was setting the LDraw path for LDFind in C:\Users\Willy\AppData\Local\Michael Heidemann\LDFind\1.0.0.0\ldfindini.xml giving the fact that I'm (and always will remain) a lousy programmer. Also the spreading of locations where all the different programs store their information today didn't help on this side.

LDfind uses AppData\Local, LDCad and LDDP (though not part of the AIOI) are in AppData\Roaming, where is present also LDFind\1.1.0.0, MPDCenter stores its mpdcenterini.xml in "C:\Users\<user>\Documents", DATHeader seams preferring C:\ProgramData. Until now it seams that MLCad is at least happy with the <LDRAWDIR>.

Honestly right now I'm not in the mood reading through all the Win specs and would just be happy if you guys could agree on a common location for your data on Vista or higher.

Thanks, w.

[Willy Tschager]

Here is an updated version of the tutorial with the adopted suggestions. It would be great if you could flick through once again.

It would also be helpful if those of you who have an Enterprise version could sent me a screenshot in English\German\Italian of this explorer window:


  ml_unofficial-screen2-en-gb.png (Size: 69.41 KB / Downloads: 30)

The pic should be a .png in the size of 640x400 pixel and show the tree in the new "C:\Users\Public\Documents\" location (a white background when using Aero would be ideal). There is no need to worry about the thumbnails showing the content of the parts, p, ... folders as I will them copy in later.

w.

[Willy Tschager]

I have moved the Parts Library in the AIOI from the LDraw root to the Public Documents folder located in "C:\Users\Public\Documents”. I would appreciate if you could perform some tests on your machines before I upload the installer to the site. The AIOI 2014-01 version 2 can be found at:

http://www.holly-wood.it/tmp/LDraw_AIOI_...bit_v2.zip

No issues, great! I'm gonna upload.

w.

[Willy Tschager]

Since there are no further suggestions I'm gonna setting it up at my site.

Nonetheless I would still appreciate getting the screenshot to avoid heavy editing of the pic. I currently have.

w.

[Santeri Piippo]

I figured to do something like this and started to use a hg repo for my parts work. Except I decided to leave the ldraw core repo out and figured to publish it. https://bitbucket.org/crimsondusk/ldraw-parts

While I agree that a git repo for ldraw would be quite cumbersome since we still need the review process, an API for the parts tracker could allow for some interesting solutions. Heck, I could probably integrate an interface into ldforge for that.