Unofficial parts once again being redirected to https

[Travis Cobbs]

LDView's unofficial parts downloading is once again broken due to the http URLs being redirected to https. For example, I debugged an attempt to download the following URL in LDView:

Code:

http://www.ldraw.org/library/unofficial/parts/s/3817cpy4s01.dat

The response was a 301 "moved permanently" response with the following new URL:

Code:

https://www.ldraw.org/library/unofficial/parts/s/3817cpy4s01.dat

This happened not too long ago and got fixed, so hopefully it can be fixed again.

Note: supporting https in a cross-platform fashion is not remotely easy, and I'm pretty sure that doing so with a third-party library such as curl also requires constant application updates to continue to function over time (updating the root certificates that the application trusts).

[Orion Pobursky]

This is fixed. I would really rather you update LDView to use the proper url (library.ldraw.org) instead of us having to have a redirect from www. 

Additionally https is becoming the default way of the web. It's becoming increasingly difficult to support non-encrypted routes on an otherwise encrypted server.

[Travis Cobbs]

This is fixed. I would really rather you update LDView to use the proper url (library.ldraw.org) instead of us having to have a redirect from www. 

Additionally https is becoming the default way of the web. It's becoming increasingly difficult to support non-encrypted routes on an otherwise encrypted server.

Thanks. My development code updated the library update process to use library.ldraw.org but missed the unofficial parts check. I can update the unofficial parts check to use library.ldraw.org, but I'm not sure what URL to use. I tried the following for this part, but it redirected to an https 404 error.

Code:

http://library.ldraw.org/unofficial/parts/s/3817cpy4s01.dat

[Travis Cobbs]

Thanks. My development code updated the library update process to use library.ldraw.org but missed the unofficial parts check. I can update the unofficial parts check to use library.ldraw.org, but I'm not sure what URL to use. I tried the following for this part, but it redirected to an https 404 error.

Code:

http://library.ldraw.org/unofficial/parts/s/3817cpy4s01.dat

Never mind; I realized that I could just check the parts tracker for the updated URL. I will update LDView to use library.ldraw.org URLs for unofficial parts, and I will investigate https.

[Orion Pobursky]

I appreciate the effort towards https even if it ultimately doesn't pan out.

Did you ever find a fix for the chunked vs unchunked downloads. Unfortunately it doesn't look we'll ever be able to enable unchunked.

[Gerald Lasser]

This is fixed. I would really rather you update LDView to use the proper url (library.ldraw.org) instead of us having to have a redirect from www. 

Additionally https is becoming the default way of the web. It's becoming increasingly difficult to support non-encrypted routes on an otherwise encrypted server.

Or you sure? Cannot update with LDView...(V4.5)

in the wireshark I see answeres to the GET request, but nothing is in there

EDIT: I could see the 301 as well

[Orion Pobursky]

Or you sure? Cannot update with LDView...(V4.5)

in the wireshark I see answeres to the GET request, but nothing is in there

EDIT: I could see the 301 as well

What url are you using? I only opened part downloads since updates had the chunked vs. unchunked problem.

[Travis Cobbs]

What url are you using? I only opened part downloads since updates had the chunked vs. unchunked problem.

I think that the URLs I posted in this thread no longer work, but maybe the part got renamed?

Regarding chunked, I believe that LDView 4.5 (the current latest release) contains a fix for the gzip/chunked combination, and I think the current server is serving up files with both gzip and chunked set. (Actually, I believe that the files now go through Cloudflare, so Cloudflare is serving up the files with gzip and chunked.)

While working on https support, I did notice that the responses (in both http and https) no longer have a Last-Modified header. (They also lack an ETag header, for what it's worth, but LDView never used ETags.) The missing Last-Modified header has a negative impact on LDView, since I think it means that every single time LDView loads an unofficial part that it has already downloaded, it redownloads the file. (Note: when a model uses the same part multiple times, it will only load and redownload once each time the model is opened.) I think that while the code was written to not blow up without Last-Modified, the logic always assumed that such would be present. If Last-Modified is gone for good, I can redo the logic that determines whether or not to try downloading an updated version.

[Orion Pobursky]

If Last-Modified is gone for good, I can redo the logic that determines whether or not to try downloading an updated version.

It's not gone for good. I'm sure I can add it back in.

[Orion Pobursky]

Or you sure? Cannot update with LDView...(V4.5)

in the wireshark I see answeres to the GET request, but nothing is in there

EDIT: I could see the 301 as well

I checked and corrected a potential issue. Try again.

[Orion Pobursky]

It's not gone for good. I'm sure I can add it back in.

A Last-Modified header has now been added.

[Travis Cobbs]

A Last-Modified header has now been added.

Thanks. I'll test this evening.

[Travis Cobbs]

I checked and corrected a potential issue. Try again.

It seems to work. I downloaded the following unofficial part and stored it in my Unofficial/parts directory:

https://library.ldraw.org/tracker/34412

When I opened it in LDView 4.5, the missing subparts were automatically downloaded. (Checking the user defaults, I see that these also have a LastModified time, while the other unofficial parts LDView had downloaded recently didn't.)

[Gerald Lasser]

I checked and corrected a potential issue. Try again.

Thanks! it seems to work now again!

I was not specific before, I used to download parts and subparts not an entire update.

[Travis Cobbs]

Thanks. I'll test this evening.

It seems that though Last-Modified is being sent, passing it back to the server in the If-Modified-Since request header isn't working (This worked before you changed servers.) The new server is sending a 200 OK response with the file data instead of a 304 Not Modified response.

Concrete example: when I request unofficial part 2676.dat, the response header contains "Last-Modified: 2023-08-07 18:57:57". When I request the file again with "If-Modified-Since: 2023-08-07 18:57:57" in the request header, I get the whole file back, with the same Last-Modified date.

Possibly related: the old server had a different date/time format for its Last-Modified value, and the format that new server is using appears to not follow the http spec. The example given in the spec looks like this: "Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT". Since If-Modified-Since also technically requires dates to be in the same format, it could be that the bad date format in Last-Modified is causing If-Modified-Since not to work. Note that these two headers very intentionally use the same format so that a program only has to store the string value for reuse, and isn't required to parse the date in order for the mechanism to work.

[Travis Cobbs]

I appreciate the effort towards https even if it ultimately doesn't pan out.

Did you ever find a fix for the chunked vs unchunked downloads. Unfortunately it doesn't look we'll ever be able to enable unchunked.

I have updated my development code to use native https libraries for the Windows and macOS versions, and Peter has updated the Qt version to do the same. The MesaGL command line version does not support https, and probably won't ever. I just tested this with the latest parts update and (as I mostly expected), the updates script still contains http URLs for all of the files, even when the script itself is accessed from https. So, everything works fine, but the actual zip file downloads still happen over http. For reference, the exact URL that my LDView development code is using is:

Code:

https://library.ldraw.org/updates?output=TAB

Having the script automatically switch to https URLs would be a very bad idea unless it can do so only when the script itself is accessed via https. I'd be happy to add a second argument to my code if you update the script to support that. Something like:

Code:

https://library.ldraw.org/updates?output=TAB&scheme=https

[Orion Pobursky]

It seems that though Last-Modified is being sent, passing it back to the server in the If-Modified-Since request header isn't working (This worked before you changed servers.) The new server is sending a 200 OK response with the file data instead of a 304 Not Modified response.

Concrete example: when I request unofficial part 2676.dat, the response header contains "Last-Modified: 2023-08-07 18:57:57". When I request the file again with "If-Modified-Since: 2023-08-07 18:57:57" in the request header, I get the whole file back, with the same Last-Modified date.

Possibly related: the old server had a different date/time format for its Last-Modified value, and the format that new server is using appears to not follow the http spec. The example given in the spec looks like this: "Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT". Since If-Modified-Since also technically requires dates to be in the same format, it could be that the bad date format in Last-Modified is causing If-Modified-Since not to work. Note that these two headers very intentionally use the same format so that a program only has to store the string value for reuse, and isn't required to parse the date in order for the mechanism to work.

The "If-Modified-Since" response likely hasn't existed since we switched to the new PT software. I've, hopefully, added it back in and corrected the malformed date format.

[Travis Cobbs]

The "If-Modified-Since" response likely hasn't existed since we switched to the new PT software. I've, hopefully, added it back in and corrected the malformed date format.

Thanks. I'll test. Due to the way LDView behaves (waiting a day before checking to see if an unofficial part has updated), I probably won't know for sure if things are working as expected until tomorrow at the earliest.

[Travis Cobbs]

The "If-Modified-Since" response likely hasn't existed since we switched to the new PT software. I've, hopefully, added it back in and corrected the malformed date format.

I was able to test today, and If-Modified-Since appears to still not be working. You can test this yourself using the following command line:

Code:

curl -I --header 'If-Modified-Since: Mon, 07 Aug 2023 18:57:57 +0000' https://library.ldraw.org/library/unofficial/parts/2676.dat

The above instructs curl to just fetch the response headers. You will see that the response code is 200, and the "Last-Modified" header exactly matches the date the If-Modified-Since value that I used above. The goal is for the above request to produce a 304 Not Modified response.

Note that I tested a similar command to the above on Curl's web site, and it returns a 304 response code:

Code:

curl -I --header 'If-Modified-Since: Tue, 30 Jan 2024 11:05:02 GMT' https://curl.se

The above returned the following:

Code:

HTTP/1.1 304 Not Modified
Connection: keep-alive
Date: Wed, 31 Jan 2024 02:13:12 GMT
Via: 1.1 varnish
Cache-Control: max-age=60
ETag: W/"21ee-61027bbe66889"
Expires: Tue, 30 Jan 2024 11:06:02 GMT
Age: 26
X-Served-By: cache-lax-kwhp1940114-LAX
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1706667192.147787,VS0,VE1
Vary: Accept-Encoding
alt-svc: h3=":443";ma=86400,h3-29=":443";ma=86400,h3-27=":443";ma=86400

Since curl.se could update at any time, you may get a 200 response if you run the same command. But if you then copy and paste that result's Last-Modified date into the curl command, it should give you a 304 response.

[Orion Pobursky]

There's probably a bug in the code.  I'll check and troubleshoot tomorrow.

[Travis Cobbs]

There's probably a bug in the code.  I'll check and troubleshoot tomorrow.

OK, thanks. For what it's worth, both of the other sites that I checked headers on (curl.se and one other) use GMT to indicate the time zone in their Last-Modified response header, while ldraw.org used +0000. I have no idea if that impacts things or not, but it is something I noticed.