LDraw OMR website

[Merlijn Wissink]

Hello everyone,

First off, I wasn't exactly sure in which subforum to post this, so I just placed it in general. I hope that's no problem.

Anyway, as the title already says, I wanted to talk about a website for the LDraw OMR sets. I'm the 'indexer'/'maintainer' of the huge list of user-submitted LDraw models at EuroBricks. I also was the one that started submitting LDraw files from EuroBricks onto Brickset. Due to reasons (which I won't include here now) the Brickset uploading stopped a while ago (for now at least).

And in between all of that and more, is the 'LDraw OMR'. A specification that a lot of people like, but the repository itself doesn't really exist. That's been bothering me since I started using LDraw years ago: everyone keeps talking about this 'Offical Model Repository', but it's nowhere to be found (easily at least). I'd like to change that. I was thinking about a little website which collects all those OMR files and makes it easy for the user to search for the file they'd like to download.

In fact, this morning I already made a prototype website. A little while ago I started experimenting with Python and a web-framework called Django (in my little experience, it's an amazing framework btw) and I saw this as a great oppurtunity to experiment and learn some more.

Well, prototype, it's actually already a fully working website which is just missing a bunch of features and content (and isn't hosted anywhere). As I said before, it's made in python using a framework called Django and it uses a sqllite database. It's fully functional; it has an admin-interface to add files and sets to the database. The only major feature that's missing is a search and listing of all available files. But those things are very easy to add. I just haven't done it yet.

You can view some images of what I have so far, here.

So, I was wondering what you guys think about this? Is this a good idea? Would it be better to be a standalone website or a subdomain at ldraw.org? Also, I have completely 0 experience with hosting anything, let alone a python application... Anyone who has more experience with that?

And, lastly: if this text is kind of vague or has no structure at all: I'm sorry, my writing skills were always very bad :|

Have a nice day!

[Philippe Hurbain]

Looks like a good idea! (but maybe I'm biased because you choose one of my models for the example...) This should indeed offer more visibility to the OMR that is lacking a lot. I guess it should be hosted at ldraw.org?

[Merlijn Wissink]

Thanks! I just used your model, because it was an OMR model and it was one of the first ones I came across when searching for a file to test with.

As said before, I have no experience at all with hosting. Would it even be possible (with the current servers/setup) to run a python website (at for example omr.ldraw.org)?

[Orion Pobursky]

Yes. I can't give you access to the main LDraw server but I can setup a sub domain and give you and account on my server where the forums and wiki are hosted.

[Merlijn Wissink]

Well, that would be nice. Still, I don't really know how to deploy a website and things like that, but having a server is of course a big step forward .

So, should I continue to finish (a first version of) this website? Does anyone maybe have any feature-ideas that would be nice to include?

[Max Martin Richter]

I like that idea a lot.
As we have several models with b-set and c-sets, I would vote for a structure that will bring the main model and b model within the same site.
Something like:

Code:

Main site
*Model number
**Model a
**Model b

would be the best in my eyes.

Furthermore the models should get tags for it's theme and year.

And another thing: How do we do the quality control for such an mpd-file?

/Max

[Merlijn Wissink]

There is already a basic structure in my prototype for handling multiple models from 1 set. You can see in the first image that the table includes a column "Submodel" which can be used for naming for example a B model. However, I think I'm going to change the database a little to be able to mark the 'Main' model of a set, the one that's on the front of the box. And, a extra textbox to name the other models from the same set. That way it's easier to filter and I think it's also less vague.

There are also already theme and year 'tags' included. I was not planning to make a detailed set database, there are already other websites for that (and that's also why a set page has a link to Brickset for more info). However, a little set information for easy filtering on the website itself is nice, that's why I included set number, name, theme and year.

Regarding quality-control: at the moment, I wasn't really planning to let users upload files themselves. Because, as you say, it's difficult to check for quality that way. At first, I think it's best to have a few admins that upload files that users submitted here in the forum. That might just work fine even after the first version, I mean, it's not a huge amount of sets. Once you've added the current OMR sets, it might be a few a week right?

-------

This morning, I also made some progress. I've started working on a simple filtering/search. I've also included a 'numbertrail' (for example 42044-1 instead of 42044) to handle different sets with the same set number. And, I've made some misc. changes. It's not a difficult website by far (I'm just relatively new to Django), so I think I might have a 'complete' version next week or maybe even this week

[Max Martin Richter]

You wrote exactly what I had in my mind. :-)
We don't need another web catalogue for all sets, it should be just a database to get the mpd-files of several models. And the only reason for adding some tags should be an easier user search.
I'm not sure how far Chris prepared something for the OMR. I would really like to see some thoughts of Chris, before we really start with this project...

/Max

[Merlijn Wissink]

Chris was (also) making something for the OMR? Or are you just wonderif if he was also making something for the OMR?
Anyway, anyone who could point Chris in this direction?

[Merlijn Wissink]

For your information: I have a pretty much complete and working website now. It just needs some polishing here and there, but other than that it just needs some hosting

[Michael Heidemann]

Do you make some tests on the files that are supplied?

Are you able to use MPDCenter as commandline tool?

cu
mike

[Merlijn Wissink]

If you mean that the website automaticly checks if the submitted file is OMR compliant: no, it doesn't. I'm not exactly sure how to implement that either. But, how it is now, is that there are a few people (although, at the moment it's only me of course) who can login to the admin-interface and add files submitted through the forum or something like that. There isn't a way for users themselves to add files. I could do that of course, but then again you need someone who checks if the files are correct.

What exactly did you have in mind?

[Orion Pobursky]

Ok. I can get the space setup this week.

[Michael Heidemann]

Yes, I have in mind that anybody (who is registered) is able to upload OMR conform files.
But for that case there needs to be a script (or application) that just do some checks and if that is not what we are looking for, we keep the upload, but marked as not OMR conform. If it has passed the checks with no faults, it is marked as OMR conform and after that a human being needs to check the file visually. After that check the file is marked as complete ok.

The checks that can be made are already integrated in MPDCenter. So I thought it would be easier for you to use that tool instead of coding all from the start.

[Merlijn Wissink]

Here are some more screenshots of how it is now.

[Magnus Forsberg]

I had a look at the pictures and would like to voice some opinions.

1.)
".. find LDraw files of official models released by LEGO" should be changed to not include the word "official". It is in my mind misleading.
".. find LDraw files of sets released by LEGO" would be better.
None of the OMR files are today "official". No file should become "official" without a review process.
Please keep in mind that the files beeing OMR compliant doesn't make them correct.

2.)
The word "submodel" is not correct in this context. Please don't use it.
To me there is also a big difference between "submodel" and "sub model".
A "submodel" is a smaller section of a bigger model.
A "sub model" is a complete model included in a set of models in a set. Like one of the cars in Flo's V8 Cafe.
"B-model" or "Alternate model" is better.

3.)
I don't like the combined names of the Author/username. I think it should be exactly the same as the Author line in the files.
Real name, followed by the username in hard brackets.

4.)
I think you should have some sort of status indicator in this repository.
Unofficial models may contain unofficial parts and assembly errors, but must be OMR compliant.
Offical models are reviewed and doesn't contain neither unofficial parts nor assembly errors.

[Willy Tschager]

The Thumbnail of the set should IMHO be a some sort of standardized render - (automatically generated by the system) or you'll end up with official pics, mixed with POVrays, Blenders, ...

w.

[Magnus Forsberg]

Why is it called OMR?
Isn't this actually a Unofficial model repository of official LEGO sets?

[Merlijn Wissink]

1) I think there's a bit of miscommunication. Yes, you are right a about the fact that 'official' might be misleading.

"Please keep in mind that the files beeing OMR compliant doesn't make them correct."

But this is not completely true. There is actually some kind of 'review proces'. As the website is now, only admins can log in and add/edit the LDraw files. Users themselves cannot upload files themselves. I think we should decide if we want to allow that or not (or maybe at a later stage). I personally think the admin-only approach is fine. Having users upload files themselves can attract trolls or just very bad LDraw files. Even if we would check if the file is OMR compliant doesn't say anything about the actual content of the file. It can be a mess of parts. And since it's not a huge workload to add maybe a few sets per week, I think a few admins can handle that (I can even do it on my own if needed). But, it seems the opinions are divided.

2) You're completely right. When making the page I couldn't come up with the right word , so I just used submodel. But, indeed, it should be alternate.

3) That's a nice suggestion and easy to change.

4) Well, this is actually a bit the same as at number 1: do we allow users submitting files or not? If yes, you should indeed have a indication if the model is checked or not. But, again, the opinions about user-content seem to be divided.

[Merlijn Wissink]

The thumbnail? You mean the image of the set?
I don't see why that should be generated. It isn't an image of the LDraw file but and image of the LEGO set. There are no images of the LDraw files anywhere.

[Merlijn Wissink]

I actually started to doubt that too yesterday. It suddenly dawned on me that 'Offical' in OMR can mean 2 things:
- It can mean that the specification is about 'Offical LEGO sets'.
- It can mean that the repository is 'Official' as in managed by LDraw.org itself.

I'm not sure which one it actually is, but I think it's the first one. In that case calling it OMR is of course not correct. Any suggestions?

[Max Martin Richter]

I would prefer a generated image from the ldraw file as well. So the user would have an uniform preview of the file.

/Max

[Max Martin Richter]

Mike started his own OMR a couple of years ago, but somehow this project died(?).
Anyway this was seen as a catalogue for official LEGO sets.

/Max

[Michael Heidemann]

Thanks for mention my personal OMR.

I had done it in the way we are now talking about. Instead of a cms system I created my own pages by an application I wrote.

But It was still too much work for only one guy. So it is not dead, but also not really alive .

[Magnus Forsberg]

How about "LDraw COOL Sets", the LDraw catalog of official LEGO sets?
Too cheesy? Juvenile?

[Merlijn Wissink]

Haha!
Sounds fun, but I think it'll create a lot of confusion when you see that name for the first time

[Willy Tschager]

Sure there are. Most models in the forums: http://forums.ldraw.org/showthread.php?t...4#pid18214 or at eurobricks http://www.eurobricks.com/forum/index.ph...opic=48285 come with a rendering of the content.

Here is a very simple way to render them effortlessly.

w.

[Merlijn Wissink]

Oh, like that. Well, I don't see any reason to not include it, but I'm not too sure how to make do that automaticly. I'll take a look into it.

Instead of images, we can also use a browser-based viewer, like this one. But, I don't know how well that works...

[Stephen]

Why not help reduce the work needed by an Admin and share the responsibility out more - require that a user submitting an .mpd also include at the very least a suitable screenshot according to some minimum resolution specification or other rules.

[Chris Dee]

If you mean this "Chris", then I have not done anything tangible to provide an OMR website. There is quite a lot I have on my 'to do' list to add functionality to the Parts Tracker, and there is a Parts Requestor in 'development limbo'. After a major downtime to move house, I am hoping that I will have more time for LDraw development this winter.

I do think an OMR should have an equivalent submission, review and certification process to parts, however an 'official(1) model tracker' wouldn't need to be tightly linked to an 'official(1) model display' website.

We have a well integrated user management system at present with the Forum being the primary register of LDraw users. This has a group membership functionality, and the Parts Tracker checks appropriate group membership for parts submission and review authentication. I suggest that we would want OMR submitters to be members of the LDraw community (i.e have a forum account), and that we only grant membership of an 'OMR submit' group on receipt of confirmation of shareability rights (equivalent to the LDraw Parts Contributor Agreement). Forum admins have the ability to add users to groups. Although we may not stay with Phorum forever, any replacement would need to have equivalent functionaility, and I would prefer that OMR management is integrated into the same authentication framework.

(1) official in the sense of an official LEGO model

[Merlijn Wissink]

Ah, I see. You have about the same thing in mind as a lot of other people here; that users can submit their own files. As I explained in one of my other posts here, I highly doubt it that's worth the effort.

Aside from the fact if it's possible to integrate the current authentication in the framework I'm using at the moment (python in combination with Django framework) or not, do we really need "an equivalent submission, review and certification process to parts". I mean, the current OMR (or what exists of it) is also managed by 1 person and isn't really checked except for OMR compliance.

And in the end, the files are not as 'important' as the parts right? I mean, people make the LDraw files of sets for fun and people download them for fun. It doesn't matter that much if the file isn't certified by x amount of people and checked for a long list of errors. If it is correct and not a complete mess, it's fine for 99% of the time the file is used. For parts that is much more important. In my opinion this OMR-site should be seen as some kind of 'extra service' to users.

But I'm open to discuss other possibilities

[Philippe Hurbain]

I agree with Merlijn on this one. Requiring OMR conformity is already quite frightening for the average user, a certification process would even more detract potential contributors.

[Roland Melkert]

I actually started to doubt that too yesterday. It suddenly dawned on me that 'Offical' in OMR can mean 2 things:
- It can mean that the specification is about 'Offical LEGO sets'.
- It can mean that the repository is 'Official' as in managed by LDraw.org itself.

I'm not sure which one it actually is, but I think it's the first one. In that case calling it OMR is of course not correct. Any suggestions?

I think it was supposed to be the second one (a long time ago). But this might be a somewhat restricted approach to use.

Maybe you should just drop the 'O' and call it a generic model repository and just add a tag/flag to indicate if it's OMR. This way people can also do donate MOC's etc.

Just my 2cts

[Merlijn Wissink]

Just in case anyone is interested: I've updated the website again. Although it's questionable if the website will ever come online, it's just a nice learning experience for me

It's difficult to show the new things when the website is just locally on my PC, but one of the major things I've added is a file-detail page aside from the set-detail page. A set page shows some basic information about the set and all files in the database for that set. A file page shows a specific file and its information. At the moment there's also a blank area where either a generated image of the file should come or a browser-based viewer (I've tried Brigl, but I couldn't get it to work). The file-detail page also shows the file header. Although I think the header doesn't have much use, it does look nice

[Chris Dee]

And in the end, the files are not as 'important' as the parts right? I mean, people make the LDraw files of sets for fun and people download them for fun.

I think people model parts mainly for fun too, and whilst part quality might be more important to the commuinity than model quality, I would like to see the LDraw 'brand' maintain its reputation for high quality deliverables.

It doesn't matter that much if the file isn't certified by x amount of people and checked for a long list of errors.

I didn't suggest a value for 'x', just that a certification process is my preference. Although for parts x=3, for models it might only need to be 1 (anyone, or the OMR admin).

If it is correct and not a complete mess, it's fine for 99% of the time the file is used.

That's the whole point of a certification process. Despite their best efforts, authors make mistakes so it makes sense [to me] to have some peer review.

I do think authentication is important to ensure that authors are happy for the LDraw organisation to share their work.

[Merlijn Wissink]

As it is now, there's already some kind of (basic) review before a model gets added, since only an admin can add files. If that admin does its job well, there's a minimal 'risk' of having files with errors on the website. Of course it's not a full review process on the website itself, but it's a review nonetheless.

Of course, it would be possible to allow authorized LDraw users to upload their files themselves and let them check by an admin before they get showed to the public. It's the same as the paragraph above, except that a user can upload the files themselves instead of sending them (or adding them onto the forum or something like that). And, as I said before, I wonder if it's worth the work to add that little bit of extra functionality (aside from the fact if I would even be able to, since I'm quite new to Python and Django).

How does the current authentication system work? Is it completely custom or does it use any of standard/framework/protocol?

[Chris Dee]

How does the current authentication system work? Is it completely custom or does it use any of standard/framework/protocol?

It uses a custom Perl module to query the user properties in the Phorum database.

[Magnus Forsberg]

Correct me if I'm wrong, but the current situation is that we do not have a model repository present at LDraw.org.
If this only turns out as a single place, to better find all the present files, stored somewere in the forum or at Eurobricks, I'm OK with it.

All I personally ask for is that we stay away from the word "official" in this context.
Create a page that makes it easy to find files, but inform the user about the status of the files.

I really want a review process (by more than one person), but I also want a better way to find all the models allready made.
Lets start with a collecting, searchable page, but lets also think ahead of the next step, a review process.

[Willy Tschager]

+1

w.

[Orion Pobursky]

You'll get the site online. It just that it's the holiday season and I have a lot of other stuff going on. I'll have webspace setup before the new year for you try out.

[Merlijn Wissink]

Oh, there's no hurry.
I just meant that there is a discussion about the (working of) site itself, not really the hosting My phrasing was a bit weird.

[Steffen]

yes, and to me the requirements for an OMR-compliant model are by far too restrictive to make it fun for me to contribute, sadly.
for me, it would be sufficient, if the file contained the model properly built, and its filename and title make somewhat sense.
this minimum set of req's would be enough for me.
of course, anyone wanting to go further formally inside the model, of course still may, but for me, the effort would be too high because I cannot see the gain.

[Steffen]

I fully agree.
May I suggest a new idea?
There already exist several uploaded models all over the world.
My idea is that our OMR repository is not a model repository, but instead simply a database pointing to models stored
wherever else. This _could_ include a server by us own, but does not have to.
The DB could point to models uploaded wherever, be it brickshelf, eurobricks, ldraw.org, or wherever else.
Inside the DB, each model is represented by a URL where it is physically stored, plus a md5 checksum of that file.
For every such model, attributes can be stored like
- the model is built properly (and this has been verified by X persons)
- all stickers or patterns are properly present
- a list of known issues like missing or substituted parts
- from which set that model was built
- year when that set appeared from TLG
and so on

The frontend of our database then would allow to search for models like this:
"give me all train sets between 1980 and 1985 containing at least 20 parts which got reviewed by at least 2 people to be correct"

This suggestion is a fundamentally different approach than creating a "PT" for models.
It allows better integration of all existing models, without having them to be copied to our servers.
This avoids the problem to ask all the existing model authors for their permission.
At the same time it avoids tedious re-creation of non omr-compliant models,
and has the charm that plenty of models are available from the start (most of them un-reviewed yet, of course)

play well
Steffen

PS, I agree with that we should drop the term "official".
I suggest titling that new thing a LEGO model-searching database.
If you want, it will in the end be a "peeron2", but with correctly built models, not only part lists.
The database could additionally match the uploaded models against existing part lists.
In the end, the new DB will be a much more powerful DB than peeron was, and this way also solve the problem of the slowly dying peeron.

[Orion Pobursky]

Ok. The webspace is setup and a (maybe temporarily named) subdomain of omr.ldraw.org is created. From here you'll have to send me the source code via email (orion at ldraw dot org) or upload it yourself via sftp (if you know how to do this). Either way you'll have to email me for login credentials.

[Philippe Hurbain]

The only thing I consider as very important in the OMR specification is the latest added bit: inclusion of unofficial parts!

[Merlijn Wissink]

Well, what I have made so far is exactly what you describe except for the fact that the files are locally stored and not linked.

I do like the idea of linked files to create a bigger collection, but we have no control at all about the files hosted at other locations. A lot of files are hosted at Brickshelf, how long does Brickshelf still exist? A lot of files are hosted at Dropbox, what if a user deletes or moves the files in dropbox? etc. etc.

So, I highly doubt if it would be useful to include links instead of adding the files themselves. It could become very annoying when you'd like to download a file a year from now and the link doesn't work anymore.

[Philippe Hurbain]

Totally agree: while links to external files might be interesting, I think that the core of the OMR should be locally stored.

[Stephen]

Although this thread had some discussion of "tagging" it related more to official information, such as year and theme.

It might be nice to also include freeform text tagging, so users can search for useful common terms like "building", "vehicle", "aircraft", "4-wide", and so on. Wouldn't want the tagging to get too out of hand (like the IMDB tags!) but perhaps up to 50 or so tags that are commonly used throughout the LEGO community.

It may seem like overkill now, but once there are 10000 models in there...

[Merlijn Wissink]

Well, it's a nice idea for searching lego-sets, but like I said before we don't really need to reinvent the wheel. When looking for a set, you can better use Brickset (which has those tags) or Rebrickable and once you found a set for which you'd like an LDraw file, you can go the OMR website to download that model. It would be a bit waste of time to make yet another database of Lego sets. That's my view of it at least.

Of course both Brickset and Rebrickable have an API for set info and maybe we can do something with that in the future, but let's get the thing online first Any progress Orion?

[Merlijn Wissink]

It seems we're life!
Thanks Orion!

At the moment, there's still a little problem with the (albeit small and simple) admin interface, but the site functions fine for 'normal' users.

There are only 6 files at the moment. I still can't download the zip file with current OMR files, the ldraw.org site still throws an error. Does anyone have an additional download link?

Let me know what you think about it (either good or bad). Also, if you think you can write something better for the homepage/about page, feel free. Writing text is certainly not one of my best skills...

Btw, I thought I changed all OMR to MR (Model Repository), but the huge heading at the front-page still says OMR. Oops...