p/low fast draft parts

[Ben Supnik]

Hi Y'all,

I was dusting off my notes on LOD when I noticed that octa-studs have already been moved to /8 in the official library, something that was discussed here.

http://forums.ldraw.org/showthread.php?t...45#pid9745

This gives us three levels of detail by searching /p/8 and /p/48 for primitives.

I would like to propose a fourth level of detail, designed to be the lowest quality rendering possible. Such rendering would have:
- No studs at all.
- No part interiors.

The LOD would be targeted for real-time viewing of stupidly-huge models, e.g. all of Datsville at once, etc. The goal is to preserve some vague shape and color of the model while bringing down vertex count as much as possible.

The goal would _not_ be to replace every single part with a simpler version, but rather to replace a subset of parts that are very commonly used (e.g. all of the standard dimension bricks, plates, etc.), possibly using a human-directed script.

In my past experimenting with this technique I found that a small number of 'draft' quality bricks could have a pretty huge impact on the weight of a model.

Anyway, if anyone thinks this is a terrible idea, or has strong feelings on how such a sub-set should be built, please let me know. I'd like to start putting a part set together.

Cheers
Ben

[Michael Horvath]

Yes, we already use something like this for Datsville. See here:

http://home.swipnet.se/simlego/ldraw/ldb...dboxer.htm

None of it is official unfortunately.

[Ben Supnik]

Right - huge models like Datsville are the ones I am targeting!

BTW how long does the automatic LDBoxer take to process all of Datsville?

cheers
Ben

[Michael Horvath]

Not very long IIRC.

I also made a less intelligent version of Tore's software that replaces all bricks with boxes regardless of whether the parts' tops and bottoms are exposed or not. You can find it here:

https://code.google.com/p/datsville/sour...%2Fmyboxer

[Philippe Hurbain]

I don't think it's a bad idea, but some of your proposals (eg. remove inside) target parts, while the p/8 method concerns only primitives.

[Ben Supnik]

Hi Philippe,

That's correct! Implicit in the proposal is that an LOD set could target for 'overriding' _any_ of:
- A part
- A sub-part
- A primitive.

In the original LOD discussion, Chris Dee commented in the context of octa-studs that he was "not (yet) convinced" about the need for this much substitution flexibility.

I think there is a strong case for part-overriding for such a low level of detail - my initial attempt at this indicated that we could get lower vertex counts at the part level tan by changing primitives by a pretty significant margin.

(From a theoretical standpoint, overriding parts and not primitives is needed when the LOD reduction in quality requires -merging- primitives into a single, simpler shape.)

I haven't specifically hit a sub-part case, but it seems odd to me to set up an overriding scheme that can override the primitive and part level and not the sub-part level.

[Roland Melkert]

I like your lod ideas, but I'm afraid it won't work a good as one would think because the library is only extensively using subparts etc in the more modern (later) modeled parts.

At least this is my experience while adding snap info to the library the last couple of weeks. So to make things really well alot of parts need to be edited.

[Ben Supnik]

Hi Roland,

We'll have to see what the real results are with some test files.

But again, my goal is not to produce a complete replacement of the library at LOD - my goal is to only replace "common" parts. My speculation is that on most models, a small percentage of the library results in most of the geometric output.

In other words, the payoff for replacing "Brick 1 x2 " will be high. The payoff for replacing "Minifig Head Nautolan, Black Eyes & Mouth, Brown Tentacle Bands" will be pretty minimal. :-)

Perhaps I can bug you for some of your large model collection to do some testing once I have a test library built up.

EDIT: I found my original post with stats...

http://forums.ldraw.org/showthread.php?t...63#pid9663

The large vertex cut-down for a relatively small set of parts changed was what gave me hope. While this was a -targeted- optimization (e.g. I modded parts based on the stats of my model) the parts I ended up modding were the usual suspects: standard, unpatterned bricks for the most part.

cheers
Ben

[Michael Horvath]

For a test case, compare "datsville" vs. "datsville_boxed", here:

https://code.google.com/p/datsville/sour...vn%2Ftrunk

[Michael Horvath]

In addition to the low-fi parts I'd like to see the L3GO parts made "official" and updated. My 2 cents.

[Ben Supnik]

Hi Y'all,

I have some numbers from hacking today. For a test file I grabbed Datsville r369, which has 127,691 parts. In BrickSmith I did a release build and turned off all culling tricks (so I was measuring the real cost of the whole model) as well as v-sync (so I was viewing the sustained throughput of the GPU with no fps loss due to waiting for vsync to come around). My box is an old Mac Pro Xeon, so it's more or less a core 2 at 2.8 ghz with a Radeon HD 4870.

I ran the model with sustained mouse rotation to measure total throughput of the GPU (e.g. how fast you'd have animation with scrolling) with 5 versions of the part library, designed to simulate "lower LOD sets":
- Original - the part library unmodified.
- Octastuds - the part library with the p/8 directory copied into p
- Studless - Octastuds, but then with the entire contents of stud.dat, stud2*.dat, and stud6*.dat blanked out.
- Boxed - Octastuds, plus many parts in the brick, technic brick, tile, and plate categories replaced with a single box5 sub-directive with the same size as the original part.
- Boxed no lines - same as above, but with all line directives programmatically removed by Bricksmith.

Here are the #s
ORIG: 241280493 vertices, 1.71 fps
OCTA: 149097204 (38%), 2.72 fps
STUDLESS: 89809425 (62%) 4.28 fps
BOXED: 58574196 (75%) 6.65 fps
BOXED, NO LINES: 41683162 (82%) 9.04 fps

A few notes:
- the boxed version uses 75% CPU, boxed no lines uses 100% of CPU - in other words, only by cutting the vertex count down by 75-82% did the GPU unloaded enough for CPU speed to matter.

- while the original is pretty unusable, the boxed no line version, at 9 fps on older hw is pretty tolerable.

- real numbers would have been better - the model is so zoomed out that bricksmith can opportunistically not draw tiny parts if allowed. But clearly there's a big fps win here, almost entirely proportional to vertex count.

Finally, the actual visual output isn't that different between the full, boxed, and boxed no line versions - they're all sort of a low-rez highly aliased mess...as is expected from just drawing a huge model at low zoom. So losing 82% of the vertices doesn't actually cause big visual problems.

My next step is to produce a much more careful "studless, boxed" variant of the library - something that could be part of the library someday.

Pics:
Original full datsville
https://www.dropbox.com/s/i26c1ey5cv51no...20full.png
Boxed parts
https://www.dropbox.com/s/187knlrv8b2373...0lines.png
Boxed parts, lines stripped.
https://www.dropbox.com/s/j0g0p4jpg912pc...0lines.png

Cheers
Ben

[Steffen]

Your suggestion in fact is to introduce a p/0 folder.
And subsequently, you suggest to introduce a
parts/0
folder for parts, analogously.

I like the p/0 folder idea. It matches the existing p/8, p/48 folders very well.

However, I have my doubts about the parts/0 suggestion, because that would duplicate lots of part implementations, and part authors need to be aware to change both files synchronously. I probably better would like a tool side solution for this problem, i.e.,
a tool discarding all invisible geometry at render time.
I suggest this because in a scene you would need to mix normal parts and parts/0 parts to
get a visually nice image. Each time you modify your scene you need to switch. For example,
if you build 3 bricks on top of another, the 2 bottom ones will be from parts/0,
and the topmost from parts. When you add another on top, you will need to change the previously topmost from parts to parts/0. And so on.
A programmatic approach would be much more versatile here imho.

So I support your p/0 suggestion, but do not like the parts/0 idea.

[Ben Supnik]

Hi Steffen,

On one hand, I have to admit that my parts/0 experiment was created entirely with automated tools and _mild_ human supervision.

On the other hand, there is some non-automatic knowledge going on. In particular, I chose to "box" only parts where the visual error for doing so was fairly low (e.g. bricks but not crazy technic parts).

So where should this knowledge live? I can think of a few places:

- Having parts in a parts/0 directory risks the parts being orphaned if a part is renamed. A part's geometry being radically changed should be a non-issue -- if the part is so totally borked, how did it get certified?

- We could have a META in the 'original' part, e.g 0 !BOXED. This would let tools box the right parts, and would be immune to library renames. This is similar to my original idea (LOD _in_ the part itself) but then we have to update a few hundred parts just to get started -- I seem to get huge push-back for this.

- A list of boxable parts could be kept around as part of the library for the purpose of generating parts/0 from scratch every time the library is released. I think this isn't actually a crazy idea - the number of library releases is small, and the process could be completely automated.

- I could keep a separate parts list of 'boxable' parts in Bricksmith - this is sort of my worst-case fallback idea, because it can be done with no LSC interaction, but it's not a great solution, because that list has to be kept up-to-date with a library that it's not part of.

Anyway, I'm open to people's ideas here...the perf win of a boxed or /0 library is there, if we can find a way to build it.

cheers
ben

[Roland Melkert]

I might have an alternative suggestion.

We could set up a small sub library of generic shapes (basically an extension of \p) and create a list (ini/xml) to map parts to (a collection of overlapping) basic general shapes (prisms, box, sphere, etc).

That list will basically direct how to setup very simple bounding box like replacement shapes for vertex heavy parts. You would only need extra stuff in the \p or (new) folder for that

Just my 2cts

[Steffen]

We could also think of introducing a new LDRAW command.

There are 2 possible ideas:

Idea 1:

Add a new command which encapsulates the portions of a part which should be left away when doing a ROUGH drawing.
Example:

Code:

0 !BEGINDETAIL
1 16 ...
1 16 ...
1 16 ...
1 16 ...
1 16 ...
0 !ENDDETAIL

However, this does not work for parts which consist of a geometry where leaving away geometry will
not work, but instead where a substitution of geometry by simpler primitives is needed. This leads to

Idea 2:

Same as yours, but not keeping the "roughifization" info in a separate file, but instead in the part file itself. Example:

Code:

0 !ROUGH 1 16 0 48 0 56 0 0 0 -4 0 0 0 16 box.dat

The lines 0 !ROUGH would contain the rough version of the part.
The disadvantage of this would be that it would be very difficult to use older tools like MLCad for seeing
the rough version of the part. For that purpose, your original suggestion of having a parts\0 folder
is by far the better solution instead of doing it in-file.

So I retract both these 2 ideas. I just wanted to mention them here for completeness.

[Ben Supnik]

LOL - Steffen, this is very similar to my first idea - that LOD be within a file. So...I like this a lot. :-) But the rest of the LDraw community seemed unhappy with it and preferred the file path ideas.

In theory, such a meta allows for the /8 subdirectory style of working ,e.g. in a file called foo.dat we could have

0 !ROUGH 1 16 0 0 0 1 0 0 0 1 0 0 0 1 8/foo.dat

In other words, files could 'point' to their siblings instead of using directory pattern matching.

[Steffen]

it may, in fact, be the case that back then when discussing your 0 !ROUGH suggestion, it was me also
argumenting against it and instead preferring a "folder where rough versions are stored", which nowadays
probably would be a parts/0 folder or something. funny how some circles close sometimes.

[Michael Horvath]

I support the idea. However, if you want the part/primitive replacement to not be noticeable to the end user, then things become exponentially more difficult. For instance, when you use Tore Eriksson's LDBoxer program - which has multiple variants of each brick depending on whether their tops, bottoms, or both can't be seen by the user - you have to run the program each time you make changes to the model.

A 'dumb' system that doesn't take part visibility into account would be easier to implement, and is what I use in Datsville.

[edit]

I also keep this thread bookmarked in case something comes out of the discussion.

[Ben Supnik]

Hi Michael,

The goal of my LOD proposal is definitely something very simple where there IS a visible loss in quality with low LOD - the idea is to let programs intentionally trade quality for speed in a clear way, at appropriate times, e.g. when parts are smaller on screen, when the machine is struggling, or when parts are more likely to be hard to see.

One thing that still needs to be flushed out is what meta-data we will have for each LOD, so that the programs can make a good guess about how 'drastic' the LOD switch will be. (E.g. going to /8 isn't that much of a change, even at moderate viewing. My "boxed" parts make very little difference in very far views, but up close the mode becomes quite destroyed.)

Cheers
Ben