[Tool/Web] LDraw to Studio Exporter


[Tool/Web] LDraw to Studio Exporter
#1
Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:


*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#2
Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#3
(2022-06-24, 14:54)Orion Pobursky Wrote: Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.

I can share you my very dirty PHP code (I am just an system-engineer, no professional developer Big Grin ) - you may use it as a base to give an idea for your tools.

The php file expects the ldraw folder beneath ldraw/ directory

edit: removed code as not of interest here I reckon..
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#4
(2022-06-24, 14:42)Christoph Reum Wrote: Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:


*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!

Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#5
(2022-06-24, 20:45)Max Murtazin Wrote: Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part

Indeed the connectivity needs to be added through the parts designer, a little bit of manual work needs to be left ;-)

The great benefit of this tool is, that a common mistake will not happen. The mistake of not importing (new) subfiles and (new) primitives not yet present in studio, that happens daily and is the most common issue when part are imported to studio through the parts designer.

And even if you are aware of the structure and the need to download subfiles, it can be daunting for larger parts, so why not make it convenient? I like the tool a lot! good potential! Thanks Christoph
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#6
(2022-06-24, 22:17)Gerald Lasser Wrote: I like the tool a lot! good potential! Thanks Christoph

Thanks - as already pointed out the script itself isn't that complicated (yet).

At EB I was asked if I also could make it work for the unofficial LDraw lib too - I hope I will find time to add this in the next week (I know I will need to also write me a little cron-script to refresh the unofficial Lib on a daily basis and to recreate the parts.lst file 😪).
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#7
So I kind of did a Python crash course and tried to do a small crawler for the unofficial stuff.

It looks pretty good so far, doing what I intended it to do, given my Python experience of 0, I consider this a major goal :-)

you pass it a URL of a part in the unofficial lib and it will compile a list of subparts required to be downloaded so that the part "works" You can then (in future, once I manage the download and zipping) pass the package to e.g. Studio

So it looks into the the section of the website "Required (unofficial) subfiles" and recursively walks through them.

Code:
import requests
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin

class CrawledPart():
    def __init__ (self, Part, PartLink, DATLink):
        self.Part = Part
        self.PartLink = PartLink
        self.DATLink = DATLink

class PartFetcher():
    def fetch(self, partno):
       
        # u9247 (lots of data)
        # u9576 (no subfiles)
        url="https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/"
        liburl="https://www.ldraw.org/library/unofficial/"
       
        time.sleep(0.5)
        print ("Part to Fetch: " + partno )
        r= requests.get(partno)

        # cut off the parent files at the marker for the RELATED subfiles, we do not need them
        doc = BeautifulSoup(r.text.split ("Related")[0],"html.parser")
        # doc = BeautifulSoup(r.text,"html.parser")

        # class .list contains the list of the required sub-parts
        link = doc.select (".list")

        if len(link)> 0:
            for subpart in (link[0].select(".header")):
                Part = subpart.attrs["href"]
                PartLink = urljoin (url, subpart.attrs["href"])
                DATLink = urljoin (liburl, subpart.attrs["href"].split("=")[1])
                crawled = CrawledPart(Part, PartLink, DATLink)
                crawledparts.append (crawled)
                print ("Subpart: " + PartLink)
               
                subparts=PartFetcher()
                subparts.fetch(PartLink)
               
        return crawledparts


and the call
Code:
subparts = PartFetcher()
crawledparts=[]
parturl= "https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/71603.dat"
for item in subparts.fetch(parturl):
    print (item.DATLink)

What remains:
- Include the root parts link as well
- download now the single files
- pack them in a ZIP with the correct folder structure
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#8
Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#9
(2022-07-13, 16:58)Max Murtazin Wrote: Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web

When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#10
Thumbs Up 
(2022-07-13, 22:03)Gerald Lasser Wrote: When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part

Haha nice, less efforts for me to rewrite my dirty code to have the unofficials included for search results too.

Unfortunately my python skill level is also around zero - otherwise I could offer more assistance. 😪
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#11
Added functionality to download unofficial files too. Files will be refreshed nightly at ~01 am UTC.
Reply
RE: [Tool/Web] LDraw to Studio Exporter
#12
@Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!
Reply
RE: [Tool/Web] LDraw to Studio Exporter
#13
(2024-03-05, 15:46)Christoph Reum Wrote: @Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!

That would be me. And until Ldraw gets a better download with dependencies I would love to have a possibility to use your tool to download a part with all subparts
Reply
RE: [Tool/Web] LDraw to Studio Exporter
#14
(2024-03-05, 15:46)Christoph Reum Wrote: @Gerald Are there any updates on having the "single part" download functionality provided via ldraw.org site itself?

Asking because I got a request from a submitter of parts who asked about my tools update-scheduling (its daily scheduled). He mentioned after submitting he would directly test do use my tool for conveniently downloading is submission for testing purposes.

If I remember correctly, LDraw is already providing a REST api to retrieve all the single files of a part - so question which arised was: Is it worth for me reworking it for not going with a local copy of official and unofficial?

It may just be a edge case - but having all downloads correct by the time being downloaded is a nice goal to reach anyways in my thinking (and also saving your site from one daily pull of the full zips...).

Best regards, Christoph!

We need to check with Orion, he was busy updating the tracker, but that was on the to do as well , but I do not know how far up this is now.
Reply
RE: [Tool/Web] LDraw to Studio Exporter
#15
(2024-03-07, 12:54)Gerald Lasser Wrote: We need to check with Orion, he was busy updating the tracker, but that was on the to do as well , but I do not know how far up this is now.

It's still on the list and edging up slowly. I'm still busy putting out the fires I caused with the new UI update.
Reply
RE: [Tool/Web] LDraw to Studio Exporter
#16
(2024-03-07, 13:40)Orion Pobursky Wrote: It's still on the list and edging up slowly. I'm still busy putting out the fires I caused with the new UI update.

No stress from my side - just wanted to get a glimpse on "is it worth on my side to recode for direct fetches of .dat files instead of daily fullzip updates...

What would you prefer from your hosting perspective? The 1st one I reckon Big Grin

So may be a "hold my beer, I'll do a 2.0 version" (still in PHP and dirty code)..

Thanks for the update Orion and Gerald!
Reply
« Next Oldest | Next Newest »



Forum Jump:


Users browsing this thread: 1 Guest(s)