[Tool/Web] Get specific part by id out or LDraw with all needed files included


[Tool/Web] Get specific part by id out or LDraw with all needed files included
#1
Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:


*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#2
Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#3
(2022-06-24, 14:54)Orion Pobursky Wrote: Awesome, thanks for this tool. I plan to have this functionality in the update to the PT software but since this is still quite a ways off, this tool will fill that gap quite nicely.

Also, Studio uses LDraw. While Studio isn't typically the first choice amongst the regulars here, Studio discussions are welcome.

I can share you my very dirty PHP code (I am just an system-engineer, no professional developer Big Grin ) - you may use it as a base to give an idea for your tools.

The php file expects the ldraw folder beneath ldraw/ directory

PHP Code:
<?php
//error_reporting(E_ALL);

//https://www.ldraw.org/library/updates/complete.zip


function preg_grep_keys($pattern$input$flags 0) {
    return array_intersect_key($inputarray_flip(preg_grep($patternarray_keys($input), $flags)));
}

function 
getSubPartAssemblies($part) {
    global 
$zip;
    
    if(
is_file(getcwd().'/ldraw/parts/'.$part.'.dat')) {
        
$ldrawPath getcwd().'/ldraw/parts/';
        
$ldrawPath2 'CustomParts/parts/';
    }
    else {
        
$ldrawPath getcwd().'/ldraw/p/';
        
$ldrawPath2 'CustomParts/p/';
    }
    
$zip->addFile($ldrawPath.$part.'.dat'$ldrawPath2.$part.'.dat');
    
    
$tmpDatContentArray file($ldrawPath.$part.'.dat');
    foreach(
$tmpDatContentArray as $line) {
        
$line str_replace("\n","",$line);
        
preg_match_all('/.*[\s](.*\.dat)/'$line$subAssemblyArrayPREG_SET_ORDER0);
        if (
count($subAssemblyArray) > 0) {
            foreach(
$subAssemblyArray as $subAssemblyLine) {
                
$subPart str_replace("\\","/",$subAssemblyLine[1]);
                
$subPart str_replace(".dat","",$subPart);
                if (
$subPart != $part) {
                    
getSubPartAssemblies($subPart);
                }
            }
        }
    }
}

function 
searchPart($part) {
    
preg_match("/^([\d]{1,}).*?$/",$part,$strippedPartArrayPREG_OFFSET_CAPTURE);
    
$strippedPart $strippedPartArray[1][0];
    
$partsListContent file_get_contents(getcwd().'/ldraw/parts.lst');
    
preg_match_all('/^(.*?)[\s]+(.*)$/m'$partsListContent$partsListArrayPREG_SET_ORDER0);
    foreach(
$partsListArray as $tmpPart) {
        
$partsArray[$tmpPart[1]] = $tmpPart[2];
    }
    
$findings preg_grep_keys("/^".$strippedPart.".*/"$partsArray);
    return 
$findings;
}


if (
$_GET["part"]) {
    
    
$mainPart str_replace('.dat','',$_GET["part"]);
    
$zip = new ZipArchive;
    if(
is_file(getcwd().'/tmp/'.$mainPart.'.zip')) {
        
unlink(getcwd().'/tmp/'.$mainPart.'.zip');
    }
    
$zip->open(getcwd().'/tmp/'.$mainPart.'.zip'ZipArchive::CREATE);
    
$zip->addFile(getcwd().'/ldraw/CAlicense.txt''CAlicense.txt');
    
$zip->addFile(getcwd().'/ldraw/CAreadme.txt''CAreadme.txt');
    
$zip->addFile(getcwd().'/ldraw/Readme.txt''Readme.txt');
    
$zip->addFile(getcwd().'/ldraw/parts/'.$mainPart.'.dat''CustomParts/parts/'.$mainPart.'.dat');
    
    
getSubPartAssemblies($mainPart);
    
    
$zip->close();
    
    
header('Content-type: application/zip');
    
header('Content-Transfer-Encoding: Binary');
    
header('Content-Disposition: attachment; filename='.$mainPart.".zip");
    
header("Content-Length: " filesize(getcwd().'/tmp/'.$mainPart.".zip"));
    readfile(getcwd().'/tmp/'.$mainPart.".zip");

}
else {
    echo 
"<!DOCTYPE html>";
    echo 
"<html>";
    echo 
"<head>";
    echo 
"<meta charset=\"utf-8\">";
    echo 
"<meta name=\"viewport\" content=\"width=device-width\"/>";
    echo 
"<style>";
    echo 
"@media (prefers-color-scheme: dark) {";
    echo 
"    body {";
    echo 
"        background-color: black;";
    echo 
"        color: white;";
    echo 
"    }";
    echo 
"}";
    echo 
"body {";
    echo 
"    background-color: white;";
    echo 
"    color: black;";
    echo 
"    overflow: auto;";
    echo 
"}";
    echo 
"</style>";
    echo 
"</head>";
    echo 
"<body>";
    if (!
$_POST) {
        echo 
"<h1>LDraw to Studio Exporter</h1>";
        echo 
"<div align=left style=\"float:center; clear: both;border-style: dashed; margin-bottom: 2em; padding: 1em;\">";
        echo 
"<p>This tool is designed to import parts which are available in LDraw, but not in Studio. From the LDraw part number, it zips all relevant sub elements (there can be many!) into a part pack.</p>";
        echo 
"<h3>How to use this downloader</h3>";
        echo 
"<div align=center style=\"background-color:#FF7777\">Disclaimer: Some Bricklink part numbers won't work, please search in LDraw library below!</div>";
        echo 
"<ol>";
        echo 
"<li>Visit the LDraw Homepage's part lookup: <a href=\"https://www.ldraw.org/parts/official-part-lookup.html\" target=\"_blank\">https://www.ldraw.org/parts/official-part-lookup.html</a></li>";
        echo 
"<li>Search for your part of choice & copy the number</li>";
        echo 
"<li>Enter this number into the below searchfield </li>";
        echo 
"<li>A list of all fitting parts from the catalogue will show up - clicking the entries will create a zip package with just the needed parts</li>";
        echo 
"<li>To use the packs, unzip them and place their contents in your \"<i><b>C:\Users\{your username}\AppData\Local\Stud.io\CustomParts</b></i>\" (Windows).<br />The parts will then be available in Custom Parts palette of Studio.</li>";
        echo 
"</ol>";
        echo 
"</div>";
        echo 
"<form action=".$_SERVER['PHP_SELF']." method=\"POST\">";
        echo 
"<div align=center style=\"float:center; clear: both; font-size:1.5em;\">";
        echo 
"Part Number: <input style=\"font-size:1.5em;\" type=\"text\" name=\"part\" /> <input type=\"submit\" style=\"font-size:1.5em;\"/>";
        echo 
"</div>";
    }
    else {
        
$partText searchPart($_POST['part']);
            echo 
"<div style=\"float:center; clear: both; font-size:1.5em;\">";
            echo 
"<p align=center><a href=\"".$_SERVER['PHP_SELF']."\">Back to search</a></p>";
            echo 
"<ul>";
            foreach(
$partText as $tmpPartName => $tmpPartDesc) {
                echo 
"<li><a href=\"".$_SERVER['PHP_SELF']."?part=".str_replace(".dat","",$tmpPartName)."\" target=\"_new\">".str_replace(".dat","",$tmpPartName).".zip - ".$tmpPartDesc."</a></li>";
            }
            echo 
"<ul>";
        echo 
"</div>";
    }
    
    echo 
"<div align=center style=\"float:center; clear: both; margin-top: 4em; font-size:0.7em;\">";
    echo 
"<div align=left style=\"float:center; clear: both;border-style: dashed; margin-bottom: 2em; padding: 1em;\">";
    echo 
"<h4>Legal Information</h4>";
    echo 
"<p><b>LDraw&trade; Parts Library</b> is licenced under <a href=\"https://creativecommons.org/licenses/by/4.0/\"><b>Creative Commons Attribution Licence 4.0 (CC BY 4.0)</b></a></p>";
    echo 
"<p><b>LDraw&trade;</b> is a completely unofficial, community run free CAD system which represents official parts produced by the LEGO company.</p>";
    echo 
"<p><b>LDraw&trade;</b> is a trademark owned and licensed by the Estate of James Jessiman</p>";
    echo 
"<p><b>LDraw&trade;</b> website can be visited at <a href=\"https://www.ldraw.org/\">https://www.ldraw.org/</a></p>";
    echo 
"<p><b>Studio</b> software is copyrighted to LEGO BrickLink, Inc.</p>";
    echo 
"<p><b>LEGO&reg;</b> and <b>Bricklink&reg;</b> are a registered trademark of the LEGO Group, which does not sponsor, endorse, or authorize this site. Visit the official Lego website at <a href=\"http://www.lego.com/\">http://www.lego.com</a></p>";
    echo 
"</div>";
    echo 
"Tool written by <a href=\"https://www.reum.it/\" target=\"_blank\">Christoph Reum</a>";
    echo 
"</div>";
    echo 
"</form>";
    echo 
"</body>";
    echo 
"<html>";
}
?>
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#4
(2022-06-24, 14:42)Christoph Reum Wrote: Hello together,

first post here - so first of all: Thanks for the great efforts, people put into LDraw library - this is just amazing!

To contribute a little back, I wrote myself a web-based tool which gives one the posibility to search and download specific parts as one single zip-file out of the LDraw library which just includes all sub assemblies which are needed for the part of choice.

The zip-files' folder structure is prepared to be included into Studios' customParts folder (I know, we are in the LDraw forums here - so nevertheless, the tool might still be useful for some people in non-Studio-scenarios).

URL: https://www.reum.it/ldraw-to-studio-exporter/

Examples as direct-download Links:


*The tool is including all license / legal-related files of LDraw into all created zip-files.

Hope it helps some here, best regards!

Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#5
(2022-06-24, 20:45)Max Murtazin Wrote: Nice thing to have, but it's not really best thing. To properly port file to Studio, you need a connectivity file besides model itself, or it would be not easy to use the part

Indeed the connectivity needs to be added through the parts designer, a little bit of manual work needs to be left ;-)

The great benefit of this tool is, that a common mistake will not happen. The mistake of not importing (new) subfiles and (new) primitives not yet present in studio, that happens daily and is the most common issue when part are imported to studio through the parts designer.

And even if you are aware of the structure and the need to download subfiles, it can be daunting for larger parts, so why not make it convenient? I like the tool a lot! good potential! Thanks Christoph
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#6
(2022-06-24, 22:17)Gerald Lasser Wrote: I like the tool a lot! good potential! Thanks Christoph

Thanks - as already pointed out the script itself isn't that complicated (yet).

At EB I was asked if I also could make it work for the unofficial LDraw lib too - I hope I will find time to add this in the next week (I know I will need to also write me a little cron-script to refresh the unofficial Lib on a daily basis and to recreate the parts.lst file 😪).
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#7
So I kind of did a Python crash course and tried to do a small crawler for the unofficial stuff.

It looks pretty good so far, doing what I intended it to do, given my Python experience of 0, I consider this a major goal :-)

you pass it a URL of a part in the unofficial lib and it will compile a list of subparts required to be downloaded so that the part "works" You can then (in future, once I manage the download and zipping) pass the package to e.g. Studio

So it looks into the the section of the website "Required (unofficial) subfiles" and recursively walks through them.

Code:
import requests
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin

class CrawledPart():
    def __init__ (self, Part, PartLink, DATLink):
        self.Part = Part
        self.PartLink = PartLink
        self.DATLink = DATLink

class PartFetcher():
    def fetch(self, partno):
       
        # u9247 (lots of data)
        # u9576 (no subfiles)
        url="https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/"
        liburl="https://www.ldraw.org/library/unofficial/"
       
        time.sleep(0.5)
        print ("Part to Fetch: " + partno )
        r= requests.get(partno)

        # cut off the parent files at the marker for the RELATED subfiles, we do not need them
        doc = BeautifulSoup(r.text.split ("Related")[0],"html.parser")
        # doc = BeautifulSoup(r.text,"html.parser")

        # class .list contains the list of the required sub-parts
        link = doc.select (".list")

        if len(link)> 0:
            for subpart in (link[0].select(".header")):
                Part = subpart.attrs["href"]
                PartLink = urljoin (url, subpart.attrs["href"])
                DATLink = urljoin (liburl, subpart.attrs["href"].split("=")[1])
                crawled = CrawledPart(Part, PartLink, DATLink)
                crawledparts.append (crawled)
                print ("Subpart: " + PartLink)
               
                subparts=PartFetcher()
                subparts.fetch(PartLink)
               
        return crawledparts


and the call
Code:
subparts = PartFetcher()
crawledparts=[]
parturl= "https://www.ldraw.org/cgi-bin/ptdetail.cgi?f=parts/71603.dat"
for item in subparts.fetch(parturl):
    print (item.DATLink)

What remains:
- Include the root parts link as well
- download now the single files
- pack them in a ZIP with the correct folder structure
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#8
Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#9
(2022-07-13, 16:58)Max Murtazin Wrote: Wouldn't it br better to write this in JS tho? I think it would integrate way better with the web

When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#10
Thumbs Up 
(2022-07-13, 22:03)Gerald Lasser Wrote: When this is done on the server itself, there's no need to crawl at all, all relations are in the DB.

While typing this, the tree-view just came into my mind, which is the most straight forward way to crawl this information, so just call the page and look for the parent part...
Convenient would be to fetch the tree-view just for a part

Haha nice, less efforts for me to rewrite my dirty code to have the unofficials included for search results too.

Unfortunately my python skill level is also around zero - otherwise I could offer more assistance. 😪
Reply
RE: [Tool/Web] Get specific part by id out or LDraw with all needed files included
#11
Added functionality to download unofficial files too. Files will be refreshed nightly at ~01 am UTC.
Reply
« Next Oldest | Next Newest »



Forum Jump:


Users browsing this thread: 1 Guest(s)