On the packaging of ldraw-parts and ldraw-mklist

[Sylvain Corlay]

Hello World,

Thank you for the amazing work!

I have been working on the packaging of ldraw for the conda package manager (both ldraw-parts and ldraw-list) and I have come across a couple of pitfalls I wanted to report back - even though it is not completely a blocker for moving forward.

1. mklist

- The mklist package does not build out of the box. Everyone (FreeBSD, Debian, ArchLinux) appears to be applying the same patches to the makefile and sources so that it builds succesfully (and I had to do so as well). It may be worth including these patches upstream and bump the version of mklist which has not been updated in a while. If the package was hosted on GitHub or GitLab, it may be easier to submit patches back.

- The mklist source tarball is included in the parts zip file, including built artefacts for windows. Since it is a very simple program, I think that it is fine to provide a built executable on the website, although the parts library and mklist may be better separated.

- Quick note if anyone on this forum is involved in the debian packaging: the version number for the ldraw-mklist debian package appears to be the same as for parts library, while it should really be 1.6.

2. Persistent source tarballs for the parts library

It seems that with every update of the parts library, the new parts are uploaded in a separate zip file, while the complete.zip file is overwritten with the new version of the full library.
This may be an issue because old versions of complete.zip are not available anymore and we can't use a persistent URL for a given version. If you kept a complete-????.zip around for each release, we would be able to point to it persistently.

I am looking forward to hearing back, and super excited to work on the packaging of the lego CAD stack for conda!

Cheers,

[Orion Pobursky]

2. Persistent source tarballs for the parts library

It seems that with every update of the parts library, the new parts are uploaded in a separate zip file, while the complete.zip file is overwritten with the new version of the full library.
This may be an issue because old versions of complete.zip are not available anymore and we can't use a persistent URL for a given version. If you kept a complete-????.zip around for each release, we would be able to point to it persistently.

The current version of the library is always the one that should be used. We take great pains to maintain backward compatibility for all parts in the library. I believe we have only broken compatibility once, ever. Therefore, is there a "good reason" ™ to archive the complete.zip? I'm mainly concerned with server space in archiving these files. When (and this may be a while) we switch to a source version control system (probably git) this will be easier but for now it probably won't happen unless there's a clear need.

[Sylvain Corlay]

The current version of the library is always the one that should be used. We take great pains to maintain backward compatibility for all parts in the library. I believe we have only broken compatibility once, ever. Therefore, is there a "good reason" ™ to archive the complete.zip? I'm mainly concerned with server space in archiving these files. When (and this may be a while) we switch to a source version control system (probably git) this will be easier but for now it probably won't happen unless there's a clear need.

Thank you for your reply!

The motivation is reproducible builds. What we try to achieve is to be able to rebuild ldraw-parts-1902 at a later point in time...  With the current setup, if someone produces another build of the same package (bumping the build number but not the version) while 1903 has been released in the mean time, he will get the new version of the parts library. (Although I think it is fine if there is also a complete.zip file pointing to the latest version).

[Sylvain Corlay]

Thank you for your reply!

The motivation is reproducible builds. What we try to achieve is to be able to rebuild ldraw-parts-1902 at a later point in time...  With the current setup, if someone produces another build of the same package (bumping the build number but not the version) while 1903 has been released in the mean time, he will get the new version of the parts library. (Although I think it is fine if there is also a complete.zip file pointing to the latest version).

Quick update: it may not be worth it to keep super-old versions around, which are unlikely to be rebuilt, but having a backup for the 3-4 last tarballs may be worth it (starting e.g. with the current version).

[Sylvain Corlay]

Quick update on the matter:

ldraw-parts and ldraw-mklist have been packaged for the conda package manager. If conda is installed, you can get these packages by typing

Code:

conda install ldraw-mklist ldraw-parts -c conda-forge

The issue on reproducible builds (permanent source tarball url) has been pointed by some of the conda-forge maintainers but they accepted the package nevertheless. I am happy to help with these packaging issues should you be interested.

[Sylvain Corlay]

Hi All,

I am following up on this. Is there a plan to address these concerns with respect to package management?

Several people on the conda-forge community were worried by the points that I raised earlier and I can only assume that it is also the case for other package managers...

I am happy to help if there should you need help with this.

[Johannes Schauer]

1. mklist

- The mklist package does not build out of the box. Everyone (FreeBSD, Debian, ArchLinux) appears to be applying the same patches to the makefile and sources so that it builds succesfully (and I had to do so as well). It may be worth including these patches upstream and bump the version of mklist which has not been updated in a while. If the package was hosted on GitHub or GitLab, it may be easier to submit patches back.

For reference, here is the patch that I've been applying to the Debian package since 2014:

Code:

--- a/mklist.c
+++ b/mklist.c
@@ -44,6 +44,7 @@
#include "alloc.h"
#include "dir.h"
#include "conio.h"
+#include <libgen.h>

char   *ProgVer = "mklist  v1.6 20100706  (C) 1999-2010 Lars C. Hassing  [email protected]";

@@ -53,8 +54,8 @@ int    quiet = 0;

/*****************************************************************/
/* Filename length compatibility stuff */
/*****************************************************************/
-#ifndef _MAX_PATH
-#define _MAX_PATH 256
+#ifndef MAX_PATH
+#define MAX_PATH 256
#endif
char    shortfilepath[MAX_PATH];
char    shortfilename[MAX_PATH];
@@ -84,7 +85,7 @@ int     namelen = 25; /* LSC specs allow 25
int GetShortPathName(char *longpath, char * shortpath, int psize)
{
     strncpy(shortpath, longpath, psize);
-    return(strlen(shortpath);
+    return(strlen(shortpath));
}
#endif

Please consider applying that patch and shipping it in the official release zip because otherwise mklist doesn't compile as it contains syntax errors and other typos and all consumers like Debian, Conda, FreeBSD, Arch or Fedora need to apply the same patch.

- The mklist source tarball is included in the parts zip file, including built artefacts for windows. Since it is a very simple program, I think that it is fine to provide a built executable on the website, although the parts library and mklist may be better separated.

In fact, mklist.c is included three times. mklist-c.zip contains version 1.3, mklist1_4.zip contains version 1.4, mklist1_6.zip contains 1.6. Versions 1.3 and 1.4 could be dropped from the release and 1.6 needs to get fixed so that it compiles.

- Quick note if anyone on this forum is involved in the debian packaging: the version number for the ldraw-mklist debian package appears to be the same as for parts library, while it should really be 1.6.

I'm of the impression that the mklist utility doesn't really get the attention it deserves and I didn't feel like spending the extra effort to make the Debian package version be the same as the utility version if the authors of the utility have not provided an update since 2010 and still ship uncompilable code.

2. Persistent source tarballs for the parts library

It seems that with every update of the parts library, the new parts are uploaded in a separate zip file, while the complete.zip file is overwritten with the new version of the full library.
This may be an issue because old versions of complete.zip are not available anymore and we can't use a persistent URL for a given version. If you kept a complete-????.zip around for each release, we would be able to point to it persistently.

This seems to be fixed now with the 2301 release as lcad2301.zip seems to be bit-by-bit identical to complete.zip. I changed the Debian packaging to use lcad2301.zip as the source instead of using complete.zip.

If you are using for old releases for reproducible build purposes, you can find those on snapshot.debian.org:

http://snapshot.debian.org/package/ldraw-parts/

[Orion Pobursky]

This seems to be fixed now with the 2301 release as lcad2301.zip seems to be bit-by-bit identical to complete.zip. I changed the Debian packaging to use lcad2301.zip as the source instead of using complete.zip.

This is not correct. 2023-01 was a complete rerelease of the library to change to the new LICENSE line so complete.zip and lcad2301.zip are the same. Subsequent updates will not be this way with lcad<update>.zip only containing updated parts and complete.zip containing the entire library. 

We have, however, started to retain the old complete.zip as complete-<update>.zip. This started with 2301 so currently there's only complete.zip and complete-2206.zip. Starting with the next release, complete-<update>.zip will also exist for the current update as well as a mirror of complete.zip. I do have an archive of all the old complete.zip's so I can manually post these in the above format. I also plan on posting the checksum since it's already provided by the (not yet reimplemented) ptrelease script.

I'll look into and update mklist. It's really legacy code at this point as no modern LDraw programs use it.