[Tool/Web] LDraw to Studio Exporter

[Gerald Lasser]

So I kind of did a Python crash course and tried to do a small crawler for the unofficial stuff.

It looks pretty good so far, doing what I intended it to do, given my Python experience of 0, I consider this a major goal :-)

you pass it a URL of a part in the unofficial lib and it will compile a list of subparts required to be downloaded so that the part "works" You can then (in future, once I manage the download and zipping) pass the package to e.g. Studio

So it looks into the the section of the website "Required (unofficial) subfiles" and recursively walks through them.

Code:

import requests
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin

class CrawledPart():
    def __init__ (self, Part, PartLink, DATLink):
        self.Part = Part
        self.PartLink = PartLink
        self.DATLink = DATLink

class PartFetcher():
    def fetch(self, partno):
       
        # u9247 (lots of data)
        # u9576 (no subfiles)
        url="https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/"
        liburl="https://www.ldraw.org/library/unofficial/"
       
        time.sleep(0.5)
        print ("Part to Fetch: " + partno )
        r= requests.get(partno)

        # cut off the parent files at the marker for the RELATED subfiles, we do not need them
        doc = BeautifulSoup(r.text.split ("Related")[0],"html.parser")
        # doc = BeautifulSoup(r.text,"html.parser")

        # class .list contains the list of the required sub-parts
        link = doc.select (".list")

        if len(link)> 0:
            for subpart in (link[0].select(".header")):
                Part = subpart.attrs["href"]
                PartLink = urljoin (url, subpart.attrs["href"])
                DATLink = urljoin (liburl, subpart.attrs["href"].split("=")[1])
                crawled = CrawledPart(Part, PartLink, DATLink)
                crawledparts.append (crawled)
                print ("Subpart: " + PartLink)
               
                subparts=PartFetcher()
                subparts.fetch(PartLink)
               
        return crawledparts

and the call

Code:

subparts = PartFetcher()
crawledparts=[]
parturl= "https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/71603.dat"
for item in subparts.fetch(parturl):
    print (item.DATLink)

What remains:
- Include the root parts link as well
- download now the single files
- pack them in a ZIP with the correct folder structure