[Tool/Web] LDraw to Studio Exporter

[Christoph Reum]

Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:

• https://www.reum.it/ldraw-to-studio-expo...t=47373-f1 
  
• https://www.reum.it/ldraw-to-studio-expo...t=47373-f2
  

*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!

[Orion Pobursky]

Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.

[Christoph Reum]

Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.

I can share you my very dirty PHP code (I am just an system-engineer, no professional developer  ) - you may use it as a base to give an idea for your tools.

The php file expects the ldraw folder beneath ldraw/ directory

edit: removed code as not of interest here I reckon..

[Max Murtazin]

Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:

• https://www.reum.it/ldraw-to-studio-expo...t=47373-f1 
  
• https://www.reum.it/ldraw-to-studio-expo...t=47373-f2
  

*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!

Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part

[Gerald Lasser]

Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part

Indeed the connectivity needs to be added through the parts designer, a little bit of manual work needs to be left ;-)

The great benefit of this tool is, that a common mistake will not happen. The mistake of not importing (new) subfiles and (new) primitives not yet present in studio, that happens daily and is the most common issue when part are imported to studio through the parts designer.

And even if you are aware of the structure and the need to download subfiles, it can be daunting for larger parts, so why not make it convenient? I like the tool a lot! good potential! Thanks Christoph

[Christoph Reum]

I like the tool a lot! good potential! Thanks Christoph

Thanks - as already pointed out the script itself isn't that complicated (yet).

At EB I was asked if I also could make it work for the unofficial LDraw lib too - I hope I will find time to add this in the next week (I know I will need to also write me a little cron-script to refresh the unofficial Lib on a daily basis and to recreate the parts.lst file 😪).

[Gerald Lasser]

So I kind of did a Python crash course and tried to do a small crawler for the unofficial stuff.

It looks pretty good so far, doing what I intended it to do, given my Python experience of 0, I consider this a major goal :-)

you pass it a URL of a part in the unofficial lib and it will compile a list of subparts required to be downloaded so that the part "works" You can then (in future, once I manage the download and zipping) pass the package to e.g. Studio

So it looks into the the section of the website "Required (unofficial) subfiles" and recursively walks through them.

Code:

import requests
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin

class CrawledPart():
    def __init__ (self, Part, PartLink, DATLink):
        self.Part = Part
        self.PartLink = PartLink
        self.DATLink = DATLink

class PartFetcher():
    def fetch(self, partno):
       
        # u9247 (lots of data)
        # u9576 (no subfiles)
        url="https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/"
        liburl="https://www.ldraw.org/library/unofficial/"
       
        time.sleep(0.5)
        print ("Part to Fetch: " + partno )
        r= requests.get(partno)

        # cut off the parent files at the marker for the RELATED subfiles, we do not need them
        doc = BeautifulSoup(r.text.split ("Related")[0],"html.parser")
        # doc = BeautifulSoup(r.text,"html.parser")

        # class .list contains the list of the required sub-parts
        link = doc.select (".list")

        if len(link)> 0:
            for subpart in (link[0].select(".header")):
                Part = subpart.attrs["href"]
                PartLink = urljoin (url, subpart.attrs["href"])
                DATLink = urljoin (liburl, subpart.attrs["href"].split("=")[1])
                crawled = CrawledPart(Part, PartLink, DATLink)
                crawledparts.append (crawled)
                print ("Subpart: " + PartLink)
               
                subparts=PartFetcher()
                subparts.fetch(PartLink)
               
        return crawledparts

and the call

Code:

subparts = PartFetcher()
crawledparts=[]
parturl= "https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/71603.dat"
for item in subparts.fetch(parturl):
    print (item.DATLink)

What remains:
- Include the root parts link as well
- download now the single files
- pack them in a ZIP with the correct folder structure

[Max Murtazin]

Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web

[Gerald Lasser]

Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web

When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part

[Christoph Reum]

When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part

Haha nice, less efforts for me to rewrite my dirty code to have the unofficials included for search results too.

Unfortunately my python skill level is also around zero - otherwise I could offer more assistance. 😪

[Christoph Reum]

Added functionality to download unofficial files too. Files will be refreshed nightly at ~01 am UTC.

[Christoph Reum]

@Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!

[Fredrik Hareide]

@Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!

That would be me. And until Ldraw gets a better download with dependencies I would love to have a possibility to use your tool to download a part with all subparts

[Gerald Lasser]

@Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!

We need to check with Orion, he was busy updating the tracker, but that was on the to do as well , but I do not know how far up this is now.

[Orion Pobursky]

We need to check with Orion, he was busy updating the tracker, but that was on the to do as well , but I do not know how far up this is now.

It's still on the list and edging up slowly. I'm still busy putting out the fires I caused with the new UI update.

[Christoph Reum]

It's still on the list and edging up slowly. I'm still busy putting out the fires I caused with the new UI update.

No stress from my side - just wanted to get a glimpse on "is it worth on my side to recode for direct fetches of .dat files instead of daily fullzip updates...

What would you prefer from your hosting perspective? The 1st one I reckon

So may be a "hold my beer, I'll do a 2.0 version" (still in PHP and dirty code)..

Thanks for the update Orion and Gerald!

[Gerald Lasser]

@christoph, Orion added the feature to download a part with all dependancies:

see here: https://forums.ldraw.org/thread-28191-po...l#pid53646

[Orion Pobursky]

Note: the zip route will return blank if there isn't a zip file. I intend to implement a redirect for those parts that don't have a zip but I thought about this after I implemented so it's not in place yet.

[Christoph Reum]

Ah thanks for the update - I recently changed my VPS the last weekend and my tool seems to have some flaws in its implementation, currently leading to not working for all parts...

Is there an rough estimate of when you may have your feature ready for the public? Just for my considerations of "worthiness of fixing mine".

When you have finished I may finally disable my tool and put a link to your functionality in for 2-3 months... (or could implemente a permanent redirect in HTTP directly to ldraw.org).

Sounds promising - over the last year I had roundabout 3.something downloads per day - so I see a regular demand of users, when it will be implemented on-site I reckon the usage may go up.

Best regards, Christoph!

[Orion Pobursky]

It's already implemented with the caveat stated above. That caveat should be removed today as this is trivial fix to the code.

[Orion Pobursky]

It's already implemented with the caveat stated above. That caveat should be removed today as this is trivial fix to the code.

I've fixed it so that the zip url redirects to the dat url if the part has no unofficial subparts.

[Christoph Reum]

nice! I still can't see a zip download button for all parts yet..

Two examples:

https://library.ldraw.org/tracker/25245
https://library.ldraw.org/tracker/26188

So you have prebuilt the zips in a one timer? I was just creating them with PHP for every request from the .dats - but this approach may create additional computing load on your webservers, so I can understand you want to build static zips.


I have two little additions:

1. Users wont recognize that "no sub parts needed for the .dat" means they may just go with the .dat download - esp. if they once realized there is a dedicated zip button for other parts - my recommendation would be to possible create single .dat zips - just for creating a convenient experience for the (not hardcore LEGO or IT) users.
2. The Download of the zips will just be available from the part details page if I got it correctly? Seems a little hidden - idea there would be to add a direct "Download Zip" right to the regular .dat Download button in the list view of the parts.

[Orion Pobursky]

nice! I still can't see a zip download button for all parts yet..

Two examples:

https://library.ldraw.org/tracker/25245
https://library.ldraw.org/tracker/26188

The zip download button only appears for parts that have subparts. However, as you point out below, novice users might not realize that you only need the bare file so I think I'll change this to all parts.

So you have prebuilt the zips in a one timer? I was just creating them with PHP for every request from the .dats - but this approach may create additional computing load on your webservers, so I can understand you want to build static zips.

They're created and served on request. The route is throttled to 60 requests/min as an anti-bot measure.

I have two little additions:

1. Users wont recognize that "no sub parts needed for the .dat" means they may just go with the .dat download - esp. if they once realized there is a dedicated zip button for other parts - my recommendation would be to possible create single .dat zips - just for creating a convenient experience for the (not hardcore LEGO or IT) users.
2. The Download of the zips will just be available from the part details page if I got it correctly? Seems a little hidden - idea there would be to add a direct "Download Zip" right to the regular .dat Download button in the list view of the parts.

1. Addressed above.

2. Good idea. I'll add it that as well.

[Orion Pobursky]

Everything I wrote above is now implemented

[Christoph Reum]

Wonderful! Guess it will make the experience for the users more stringent in all cases!

Thanks for all your efforts! ♥

[Fredrik Hareide]

This is an excellent addition to the site. Thanks!