Parsable Meta-Data for High-detail vs. Low-detail parts and primitives

[Ben Supnik]

Hi Y'all,

A while ago someone pointed me at the low-detail versions of the stud primitives, which cut the polygon prism side count in half.

I just want to confirm that there is no machine-readable way to determine which files are higher or lower res versions of each other, either via a meta command or a separate index file.

I am interested in coding Bricksmith to dynamically swap between high and low res primitives to improve rendering performance without having to hard code a list of "known swaps".

cheers
Ben

[Chris Dee]

There is no meta-statement within the file, although meta-information is effectively embedded in the file name, but this functionality in confined to stud primitives. Each primitive (files in ldraw/p) that matches stud*.dat will have an equivalent stu2*.dat partner in the same folder.

This dates back to the original LDraw implementation and is probably not how we would do it now. In concept it is not that different to the hi-res primitives which are stored in ldraw/p/48 and (when present) can be substituted for normal-res files of the same name. I wouldn't have an issue moving the ldraw/p/stu2* files to ldraw/p/8/stud* except that MLCad has a hard coded expectation and whinges if a stu2* file is not found.

[Magnus Forsberg]

Have you tried doing that? I did, and it seems to work just fine.

I renamed and moved my P/stu2.dat to P/8/stud.dat and made a new P/stu2.dat as a move to-file, refering to P/8/stud.dat.
I also changed the description to "~Moved to Stud". No problem.
I even moved that new file to unofficial/p-folder. No problem
The only thing I can't do is to remove stud.dat from P-folder, and move it to unofficial/p-folder.
Then MLCad just stops working.

I don't know if this little test is of any relevans here, but I tried it and it seems to work.
Is it so that MLCad only need a file called stu2.dat in any P-folder to stay quiet, and the content of the file is not relevant?

[Chris Dee]

I imagine all MLCad needs is a stu2* file in ldraw/p for each stud* file in that folder. Even if we move the physical lo-res studs to ldraw/p/8/ it doesn't release us from needing to issue a stu2* every time we add new stud* variant.

[Travis Cobbs]

I'm not exactly sure if it's a good idea or not, but having a p/8 folder would allow for all the other circular primitves that aren't n/16 to be put there for software to use as desired. If p/8 were created, I think Magnus's suggestion of turning the stu2* primitives into moved to files pointing to p/8/stud* makes sense.

If you want to have some idea of what effect this would have on the look of models, you can mostly simulate it in LDView by going into LDView's preferences, enabling primitive substitution on the Primitives tab and then setting "Curve quality" to its minimum setting, and then checking the "Flat shading" check box in the Misc box of the Effects tab. (Note that "Flat shading" and "Smooth curves" are mutually exclusive; they're check boxes because they can both be disabled at once, but only one can be enabled at a time, so checking one unchecks the other.)

[Steffen]

Oh, that sounds like a Good Idea ™ to me.
Keeping the "stu2" MOVED-TO's for a while (until MLCad has been updated........) would be tolerable for me.

Especially, putting the low-res stud variants into a dedicated folder
opens the way for ANALOGOUSLY treating their HI-RES variants:

JC Tchang has created a bunch of VERY NICE, VERY HIGH-RES studs,
which of course are somewhat redundant to LDViews internal stud replacement,
but they allow to use ultra high res studs also in other tools. They are here:
http://jc-tchang.philohome.com/manuel/pa...ogo_tchang
His current solution is, for switching between the studs resolutions,
to invoke a batch which replaces stud.dat etc.
I would better like to have his files included with the library normally as primitives,
and as hi-res-replacements for the normal studs, analogously to their low-res replacements "stu2*".

This additionally would solve ANOTHER of our currently pending problems: see
http://www.ldraw.org/cgi-bin/ptdetail.cg.../3334h.dat
There, duplicates of our parts have been created, just to have better, hi-res underside studs.
This is a bad idea ™. Instead, we should analogously to the hi-res/low-res top studs,
offer hi-res/low-res underside studs, and applications should be enabled to switch between them
at runtime.

[Ben Supnik]

Hi Guys,

Maybe I'm off in outer space on this compared to the rest of you, but....

It seems really fragile to have part substitution done by file naming conventions and not by a meta directive or something else that is "data driven" (meaning information about the availability of parts at differing resolutions is part of the part library download).

In other words, moving the stu2 parts to a folder solves this one problem once for now, but if any more res-specific parts are added, everyone's applications will need to be updated. :-(

What I was originally going to suggest (given that such a thing apparently does not exist) was a proposed meta data command that could point from the canonical mid-res part (e.g. stud.dat) to its high and low-res variants.

Thus programs could determine at load time from the currently library what resolution variants are available. If authors wanted to create additional versions of the parts, no software mods would be needed. LDraw viewers could write the code to detect variants once and then leave it alone.

cheers
Ben

[Michael Heidemann]

Maybe I am wrong, but why do we need a new meta command?

If we are going to have two separate folder trees for high-res and low-res, then it would be easy to handle.

Every application reads the standard resolution. Now if you have an application that allow for low or high-res then the user has to inform the application that what kind of resolution should be loaded.

Benefit of this solution:
All current application work like at present (AFAIK)
The user chooses the resolution.

The only bad thing is that we need to define high and low-res standard. If we would like to have more resolution as the three current, then it would get complicate.

[Steffen]

we from the beginning should care about different levels of high-res.
for example, for jc's beautiful studs I mentioned,
he already offers "3 increasing levels of detail and beauty".

sometimes they will not increase, but simply be alternative.
like "logo modeled by lines" vs "logo modeled by projected texture" vs "logo modeled from real 3D objects"

[Chris Dee]

It seems really fragile to have part substitution done by file naming conventions and not by a meta directive or something else that is "data driven" (meaning information about the availability of parts at differing resolutions is part of the part library download).
Ben

Yes, it is a bad design, but it dates back the the very birth of LDraw, and has unfortunately been hard-coded into MLCad (at least).

[Michael Heidemann]

Yes, you are right, but I do not see any practical solution for such diversification of high res (or maybe later low-res) parts. If we go and put all that possibilities into the main part as a link how do you like to choose which one should be shown?
In my opinion we should leave these diversifications to the experienced user - and they know how to show what they like to see.

[Ben Supnik]

Hi Guys,

A few comments on file conventions vs. meta commands.

First: I think we can all agree that either a file naming convention or a meta-command could serve as a vehicle for primitive substitution, for the purpose of changing the level of detail of output. And while I am advocating meta commands over file naming conventions, I do believe that a directory scheme is better than hard-coding a stud -> stu2 in a file parser. :-)

Second: why I care. I believe that cutting the vertex count of large models is the major barrier to real-time preview of seriously-huge-ldraw-creations. Here are some numbers: the Datsville model that Allen sent me to do stress testing has 3960 pieces, and results in 63.6 _million_ vertices when drawn in full. This is an insane number of vertices -- it's almost 100x the number of pixels on screen, thus the vertex count is perhaps overkill by 100x and way outside the range of what GPUs are optimized for.

By substituting octa-studs for the normal ones, we cut the vertex count to 38.04 M studs a 40% improvement in vertex count that basically gives us a 40% improvement in framerate. That's huge! I ran the same scheme on a 64k part model that I built and found that it normally produces 122 million vertices (ouch) and cuts to 81 million with octa-studs - another big reduction.

So substituting meshes with lower detail is a big perf win, but even with octa-studs, we're still way vertex heavy. There has to be a way to draw an entire lego town into a 900 x 900 window without burning 38M vertices. :-)

So I think we have to be looking at additional ways to LOD substitute, e.g.

- more decimations of the stud, e.g. can we have simple quad-studs? These would look lousy but if a stud is covering 3 screen pixels, I think we'll be okay. :-) And we're cutting the stud geometry count in half again (and we saw already that studs do produce a huge amount of vertices).

- how about dropping studs entirely? We know that studs are small, so even if a 32x32 baseplate is big (10" on a side) and thus must be drawn at low zoom, the studs themselves turn to noise at some point.

I can think of other ways to cheat LOD...for example, we could drop the tubes inside bricks - this is a different kind of LOD reduction because it's not just "crude", it's actually "wrong" if viewed closely. But there's a lot of brick interior that is never seen in a huge model - when doing a real-time rotation of a large model at low zoom, we can live without these things. We could even drop the _interiors_ of bricks for a lowest-res, crudest-but-quite-fast rendering.

So my question is - if we are going to introduce a file naming scheme, how would we cope with
- a whole set of resolutions for studs.
- how would a program know what stud resolutions are available.
- if there are other decimation schemes to reduce LOD, how are they accessed?
- can the schemes be combined? E.g. can we have a brick with no studs _and_ no interiors?

My fear with the file naming convention is that when we go to expand it later, we're going to be boxed in by the limits of what can be done with file paths. And at 38 M vertices with octa-studs, I think we need to be able to go several steps further under _some_ rendering schemes.

(Consider also high-res rendering for the close-up....we'd like LOD to go both ways!)

Anyway, if we can really get that kind of flexibility out of file paths, I'm not against it. But I would like to see something that allows us to go further with LOD and doesn't require rewriting clients every time we push LOD farther.

Cheers
Ben

[Ben Supnik]

Yes, you are right, but I do not see any practical solution for such diversification of high res (or maybe later low-res) parts. If we go and put all that possibilities into the main part as a link how do you like to choose which one should be shown?
In my opinion we should leave these diversifications to the experienced user - and they know how to show what they like to see.

Hi Michael,

My hope is for an LOD scheme that is machine-readable so that the rendering software can manage LOD with minimal or no user input.

(BrickSmith provides a really friendly user experience - it allows users to create complex models without knowing about the guts of ldraw files. I'd like to preserve that user friendliness while letting users get the performance boost of LOD.)

Here are three ways to manage LOD that could be user friendly:
1. If the sets of resolutions are named for humans ("high, medium, low") then a user could pick a resolution set on a menu, e.g. a view setting. For this, the part library would need some kind of description of what resolutions are available, and how to load them. In this case, new resolutions ("ultra-high") could be added without software modification.

2. The renderer can detect when a part is small on screen and change rendering. Thus the lower resolutions would only be used when the part is so small on screen that the loss of detail is hard to see. (This is the time that we need low res most because at low zoom, lots of parts must be drawn at once.) In this case, the resolutions sets need indications of their intended consequences on detail.

3. The renderer can detect the actual frame-rate and switch to lower resolution when the model becomes 'heavy' enough that the current resolution is too slow. So for example, if spinning the model goes below 10 fps, go down one resolution set. In this case, the renderer must have some way of knowing the 'order of detail' of resolution choices, so that it can step up and down based on actual performance.

Anyway, it's on me to write up some kind of proposal for this, but my hope is to come up with something (meta command or file scheme) that would allow for LOD to be accessed by the program and not the user, to make the user experience better.

cheers
Ben

[Michael Heidemann]

Ok Ben, now I see your point of view - no. of vertices and fps.

I like your approach very much, but I see difficulties in defining the detail level for each level.

I would prefer a defined storage according file path, because by doing this we do not need to touch the current library in general.

[Ben Supnik]

Ok Ben, now I see your point of view - no. of vertices and fps.

I like your approach very much, but I see difficulties in defining the detail level for each level.

I would prefer a defined storage according file path, because by doing this we do not need to touch the current library in general.

Hi Michael,

I spent a little time today trying to come up with the features I would need to improve framerate in BrickSmith, and I think the scheme I came up with could work equally well with META commands or with a file search path scheme.

In _both_ cases, no change to the library is mandatory - adding lower or higher resolution variants to a part are always optional. I imagine us adding this data tactically for the most used parts or most used primitives.

In both cases, some modification must be made to the current stud.dat vs stu2.dat situation - either by a META command in stud.dat or by copying stu2.dat to a sub-directory.

So I think the work-load on the community would be quite similar.

(Allen asked me if a file search path would be adequate; originally I thought now but now I think it would work well enough.)

cheers
Ben

[Santeri Piippo]

I'll x-post what I posted on the PT about this matter:

I wonder would it make any sense to have some meta-commands to establish relations between the normal and logoed studs. With this I mean keep the stud-logo.dat file as it is and add something like this to stud.dat:

Code:

0 !SIBLING LOGOED_STUD stud-logo.dat

which would tell programs that the logoed counterpart of stud.dat is stud-logo.dat. This sort of syntax could open up other possibilities for other possible future relationships, for instance, hollowed underside studs for the modern 1 x N plates.

It could also help resolve this matter about the lo-res studs:

Code:

0 !SIBLING LO_RES_STUD stu2.dat

[Chris Dee]

I think an extension to the current directory based approach would work and be easiest to implement.

- All primitives must exist in ldraw/p
- Hi-res versions may exist in ldraw/p/48; tools running in hi-res mode should substitute the same named file from this folder, if it exists - effectively inserting ldraw/p/48 in the search path ahread of ldraw/p.
- Lo-res versions may exist in (the new folder) ldraw/p/8; tools running in lo-res mode should substitute the same named file from this folder, if it exists - effectively inserting ldraw/p/8 in the search path ahread of ldraw/p.
- The same could be done for ultra-hi-res or ultra-lo-res if there is a need.

No meta statement should be necessary to implement this. Adding a meta statement to the ldraw/p primitive would add unnecessary complexity to the library maintenance. Why should we have to update p/cyl5-16.dat when p/48/cyl5-16.dat is added to the library? Tools that need to know what hi-res or lo-res primitives exist can check the contents of the named folder, which should be less processing than reading the contents and parsing each file in ldraw/p.

All existing ldraw/p/stu2* file would need to be moved to ldraw/p/8/stud*, although we would have to leave leave the stu2* files as "~Moved to" files in ldraw/p.

[Chris Dee]

Isn't the logoed stud just a version of stud.dat in ldraw/p/48?

It's much easier if the equivalent primitives have the same name in a different folder (e.g. ldraw/p/stu28a.dat becomes ldraw/p/8/stud28a.dat. We can deal with the stu2 legacy with ~Moved to files until such time as MLCad no longer hard-codes the relationship.

[Santeri Piippo]

The scheme you posted would work for me as well and you're right about the cumbersomeness related to the 48-primitives. Oh well, just an idea anyway.

Anyway if the directories are named after the amount of divisions, a stud.dat in 48/ would imply a stud made out of 48-primitives. I'd suggest p/logostuds/stud.dat or something like that, then a program could substitute primitives with their counterparts in p/logostuds/ for stud logos, p/48/ for hi-res primitive substitution, p/8/ for fast-draw studs and so on. IMO we should keep this generic like this.

[Michael Heidemann]

All what you perfectly discribes was in my mind when talked about file path benefits .
Thank you very much for these detailed explanation.

But there is still the question for more or less detailed level. We talking generally about low - normal - high resolution. But if we need a low-low resolution or a high-high resolution, what can be done?

Now after reading your suggestion it seems to be easy
If the resolution of ldraw/p/8 is still too high, just search for a lower number in the path! (f.e. ldraw/p/4)

That's it.

[Tim Gould]

I like this. No new meta commands, it makes sense, and it's well defined. And you can make 48- equivalent parts for non-round primitives.

Tim

[Ben Supnik]

Why should we have to update p/cyl5-16.dat when p/48/cyl5-16.dat is added to the library?

Hi Chris,

I am sympathetic to the desire to minimize cross-file changes, although I think the extent of them would be contained (e.g. you could think of every version of cyl5-16.dat being a "family" of levels of detail, and edits would be contained within the family).

For a fully path based approach, what about specializing a part or sub-part? For simplifying studs having search paths on the /p/ directory works, but if I want to produce a _really_ high-speed 1x1 brick, I need to be able to replace 3005.dat with a new version with "stuff" cut out.

What do you think about parts/<lod>/ being a legal part search path, and or parts/s/<lod>/file.dat being a legal search path for sub-parts?

My concern is the search rules might get a little bit weird, but for "draft" quality rendering, I think we need LOD access to the parts, not just the primitives.

Cheers
Ben

[Ben Supnik]

Now after reading your suggestion it seems to be easy
If the resolution of ldraw/p/8 is still too high, just search for a lower number in the path! (f.e. ldraw/p/4)

I think it is important that the valid "LOD" paths (e.g. 8, 48, 4, etc.) we explicitly named in a config file or an official spec, rather than having client apps just go searching the directory structure. We have some folders in parts (s, textures) that are not LODs, and we may have future reasons to add sub-folders to the parts or primitives folder.

If we know that clients only treat sub-folders as LODs when they are told to, I think it will make future expansion of the library less difficult to plan.

Cheers
Ben

[Chris Dee]

We have some folders in parts (s, textures) that are not LODs, and we may have future reasons to add sub-folders to the parts or primitives folder.

If we know that clients only treat sub-folders as LODs when they are told to, I think it will make future expansion of the library less difficult to plan.

I agree, tools should be able to tell what a sub-folder represents. We could either add fixed name configuration file localised to each folder, so long as it does not have a .dat extension or (my preference) add a single file in the ldraw root folder. Does anyone know if any of the tools would whinge if they found a file with an extension that is not .dat in an ldraw folder?

I'm not (yet) convinced of the need for LOD folders for parts, but the inclusion of parts/s/ and parts/textures (when that is implemented in the official library) would allow that.

[Santeri Piippo]

Maybe we could use LDConfig.ldr for this purpose? To my knowledge it was meant as generic configuration and just ended up hosting only colors?

[Ben Supnik]

I agree, tools should be able to tell what a sub-folder represents. We could either add fixed name configuration file localised to each folder, so long as it does not have a .dat extension or (my preference) add a single file in the ldraw root folder. Does anyone know if any of the tools would whinge if they found a file with an extension that is not .dat in an ldraw folder?

I'm not (yet) convinced of the need for LOD folders for parts, but the inclusion of parts/s/ and parts/textures (when that is implemented in the official library) would allow that.

Hi Chris,

My thoughts were similar to Santeri's: a single list of LOD sub-folders in a top level file, similar to the color config file.

Re: the need for parts, the burden is on me to prove it. I have only run real numbers tests on octastuds so far (and it was the easiest test to do since the parts existed already).

Let me see what it would take for me to get real numbers for the difference between simply going "studless" and starting to remove pieces of part structure. (This is where we would need to sub-folder the parts themselves).

Two data points are pushing me toward this test:

1. My 60k part test is still stuck at 6.6 fps on not-amazing-but-not-unreasonable hardware, and it is hard limited by vertex count. So to get smooth scrolling/rotating we need at least another 50% reduction in vertex count. (And that would still be 15 fps scrolling - not the smooth 30 fps we'd like.)

2. 60k parts isn't a lot of parts. Alice Finch's hogwarts is apparently about 400,000 bricks - that is, 6x more parts than my biggest test case so far.

So not only is our current vertex perhaps 2.5x too big to be realtime, our test cases are 6x smaller than what real-world builders are making. My goal is to allow BrickSmith to edit this kind of huge model in real time... so I'm looking for a 15x reduction in vertex count. :-) I think for this we need to start cutting muscle and not just fat, metaphorically speaking - we need to be able to _temporarily_ drop significant geometry like brick interiors.*

I imagine having four levels of detail:
high - corresponds to using the /48 studs.
default - what we have now
low - corresponds to using the /8 studs.
draft - no studs, no tubes, maybe interiors missing.

An app could render at low, default or high depending on zoom level, but then when the user goes to spin the model, temporarily use draft rendering to keep the UI responsive.

Anyway, I'll post some numbers when I get them. :-)

Cheers
Ben

[Steffen]

I fully agree with Chris here.
It is not necessary IMHO to interrelate the files from one to another, this would require much too much frequent touching of files.
Using dedicated folders with purpose for that is exactly the way I would go here as well.

A little thing that I don't like yet is:
it is something different to substitute the normal primitives used in a part by 48ed ones
THAN
substituting studs with their hi-res counterparts.
Ah, just as I am writing this, tools of course can finely offer options
- substitute all primitives
- substitute just the studs
So, take it just as a reminder.

I also LIKE the idea of using ldconfig to describe the purpose of folders,
instead of putting some stray files everywhere.
We should keep things as centralized and simple as possible,
not distributing information too much across various places.

[Steffen]

replying to Chris: YES, +1

[Steffen]

+1

[Ben Supnik]

I'm not (yet) convinced of the need for LOD folders for parts, but the inclusion of parts/s/ and parts/textures (when that is implemented in the official library) would allow that.

Hi Chris,

I ran a test that makes me think there's pay dirt in being able to sub parts too, and not just primitives.

I produced a few quick & dirty level-of-detail sets by taking a text editor to my library:
Full - what we have now
Octastuds - using the stu2.dat files for all studs.
From here down I started really destroying things - to get a sense of what kind of perf we might get out of a 'crude draft' quality part set.
No line - removing all lines.
No studs - all studs, tubes and stud-like things gone
No interiors - interiors and edging removed from some parts.

For this last one I looked at a histogram of my test case. The 17 most frequent parts made up 25% of the model by part count. I removed the interiors and edging (or made some other simplification for 16 of them).

Here's the numbers:

Code:

set vertex count (millions)  fps
full            122.7       3.1
octastud        81.1        4.6
no lines        54.3        6.6
studless        25.9       12.0
no interiors    13.2       18.0

So...we can still get a 2x cut-down of vertex count beyond where we can do with primitive replacement by also allowing part replacement. And I suspect that the histogram-type distribution applies quite often in large models. In Dastsville (the version I have) the top 14 parts make 50% of the model.

I think I could, given a reasonably quiet weekend, produce no-interior versions of the entire set of basic bricks, tiles, and plates, and this collection would go a long way toward reducing vertex count for draft rendering.

Cheers
Ben

[Steffen]

all that might be true,
but it for sure will not be achieved by us _modeling_ all these variants.
we have plenty enough to do modeling the real stuff.
all these simplifications you mentioned can be done programmatically by software if/when it wants.

[Ben Supnik]

all that might be true,
but it for sure will not be achieved by us _modeling_ all these variants.
we have plenty enough to do modeling the real stuff.
all these simplifications you mentioned can be done programmatically by software if/when it wants.

Hi Steffen,

I am not sure what the best way to produce a draft-quality LOD part set would be, but whether it is automation, manual, or a mix (and I think a mix of the two is most likely) I think it still makes sense to store the _results_ of the LOD reduction in the part library and do the processing ahead of time, for a few reasons:

* If rendering programs have to compute the low LOD on the fly, then we get to write this code many times over (just as we have had to do with smooth shading - I think smooth shading has been coded maybe 4 times?). If we precompute the data, then if any one of us writes code to reduce LOD, we can all use it.

* Precomputing the LOD would let us use algorithms that are potentially higher quality but too slow to do on the fly in-app.

* Precomputing the LOD would let us use different techniques for different parts.

I think an 'alternate LOD' set will be a relatively small number of files per set - 50-100 files perhaps, not a complete clone of the library!

cheers
Ben

[Jude Parrill]

I think at this scale, you need to start looking at removing geometry programmatically. For example, if you have a brick that is part of a wall that is covered both on top and bottom, you can remove all studs and bottomside detail, and even close up, it will still look right.

After all, isn't this why we have BFC? If the camera is looking at the "back side" of some primitive/reference, why draw it? I think you're better off writing algorithms that detect whether or not something can be seen and if so, which side, and then drawing/not drawing based on that. You're absolutely right that you're going to end up with a lot of unneeded/unseen geometry in really any size file, and rendering it is pointless/a waste of resources. Even with just a single part, at any given time, only half (at most) of it's geometry is visible. Writing algorithms to cull this unseen geometry is (I believe) going to be key to solving your performance issues.

[Ben Supnik]

I think at this scale, you need to start looking at removing geometry programmatically. For example, if you have a brick that is part of a wall that is covered both on top and bottom, you can remove all studs and bottomside detail, and even close up, it will still look right.

What you are describing is similar to occlusion culling. But:
- I do not think this will solve the problem I am trying to solve. For the size of models and scale of drawing, often even the visible surfaces represent "too much stuff"...I have to accept lower quality output.
- Because Bricksmith (and other LDraw apps) are often editors and not viewers, they can't use occlusion algorithms that require heavy pre-processing of the data. BrickSmith needs to be responsive when a user moves a brick. Thus if it can't determine visibility in more-or-less real-time, visibility determination isn't an option. I definitely don't have enough CPU power to do occlusion culling on every single stud in a model.

After all, isn't this why we have BFC? If the camera is looking at the "back side" of some primitive/reference, why draw it?

LOD and BFC solve different but related problems.
- BFC reduces the number of pixels shaded by the graphics card when surfaces are not already submitted in front-to-back order, but it does not decrease the number of vertices submitted.
- LOD reduces the number of vertices submitted but does not change the number of pixels shaded.

For very large models at low zoom, BrickSmith is definitely _vertex_ bound - BFC won't make any difference.

I think you're better off writing algorithms that detect whether or not something can be seen and if so, which side, and then drawing/not drawing based on that. You're absolutely right that you're going to end up with a lot of unneeded/unseen geometry in really any size file, and rendering it is pointless/a waste of resources. Even with just a single part, at any given time, only half (at most) of it's geometry is visible. Writing algorithms to cull this unseen geometry is (I believe) going to be key to solving your performance issues.

Per above, I do not think this is feasable for an editor with real-time part editing/movement and real-time camera movement. :-( This is why I am coming to the LDraw community to discuss geometrically simplified model sets.

cheers
Ben

[Orion Pobursky]

I'd like to point out that some of the legwork for reduced LOD parts has been done in the past by Tore Eriksson with his LDBoxer program:
http://home.swipnet.se/simlego/ldraw/ldb...dboxer.htm

[Allen Smith]

We need a configuration file in order to make this fully machine-readable, so that software authors don't need to update their software if new detail levels are defined.

At first I thought simple numbered folders would be sufficient, but now I don't. The discussion about modeling the LEGO logo on studs convinced me of that. That's a whole new level above and beyond primitive sizes.

The config file could look something like this:

0 // the following are order-dependent, from least to highest resolution
0 !DETAIL-LEVEL folderNameForLeastDetail
0 !DETAIL-LEVEL folderNameForMoreDetail
0 !DETAIL-LEVEL ...
0 !DETAIL-LEVEL folderNameForMostDetail

That should allow an infinite sliding scale to accomodate the most insane simplications or complexifications. Folder name could be something like "128-3DStudLogo" if you wanted.

Allen

[Ben Supnik]

0 // the following are order-dependent, from least to highest resolution
0 !DETAIL-LEVEL folderNameForLeastDetail
0 !DETAIL-LEVEL folderNameForMoreDetail
0 !DETAIL-LEVEL ...
0 !DETAIL-LEVEL folderNameForMostDetail

Hi Allen,

This proposal provides the folder name and the relative LOD ordering, both of which are critical. I would also like to see:

- Some kind of human-readable name, so that the folder name isn't restricted by being pretty.
- Meta-data as to whether the LOD is "lossy" (meaning actual parts of the brick are missing). For example, removing studs, removing inner geometry, or any other "heavy-weight" cuts would make an LOD lossy. But using octagonal studs would not be lossy.
- Ideally, in my fantasy world with the unicorns and flying elephants, some kind of metric to indicate the physical error of each of the higher LODs...in other words, how much different is a 16-stud from a perfect circle, or a 48-stud. For the non-lossy cases, a program could then decide on the fly when the zoom factor makes such an LOD irrelevant.

Cheers
Ben

[Roland Melkert]

- Ideally, in my fantasy world with the unicorns and flying elephants, some kind of metric to indicate the physical error of each of the higher LODs...in other words, how much different is a 16-stud from a perfect circle, or a 48-stud. For the non-lossy cases, a program could then decide on the fly when the zoom factor makes such an LOD irrelevant.

I think this is pretty much the most important parameter. If the paths described in the config ldr use e.g. a percentage number that indicates howmuch 'detail' is been cut away from files at that location. Any renderder could then do simple (and low cost) frustum space z distance calculations resulting in a raw number of part pixels per screen pixel (let's say 10 for now, meaning one ldu is 10 pixels at 100% detail). And use that number to find the closest match low/high detail variant of the part.

For example you have an part that's 20LDU (we have to decide what dimension this is, highest, lowest or average)
And it would be rendered at 50pixels, you could e.g. calculate that it's using (20*10)/50 -> 25% of it's 'resolution', now if there are LOD sources of e.g. 75%, 50% and 25% it would look in the 25 one for a replacement, if none it would try the 50 and so on.

Just a rough idea / my 2cts

edit: changed some of the numbers to make more sense.

[Chris Dee]

I'm not particularly keen on there being implicit meaning in the sequence of metadata lines in the config file.

Would something like this work? Two positional parameters to describe the level of detail and the folder name, then free text to the end of the line for the description(which can include spaces)

Code:

0 // !PRIMITIVE-DETAIL <Polygon edge count> <folder name> <description>
0 !PRIMITIVE-DETAIL 0  p/0   Ultra-low resolution
0 !PRIMITIVE-DETAIL 8  p/8   Low resolution
0 !PRIMITIVE-DETAIL 16 p     Standard
0 !PRIMITIVE-DETAIL 48 p/48  High resolution

The polygon edge count would be used to sequence the entries and would describe how circular elements are coded, but wouldn't prevent other primitives existing in the same folder.

The description would be used by tools as a standard way of presenting the detail levels in selection lists.

How should 16 be defined as the default within the metadata? Or will applications need to assume that - it is the only folder that is guaranteed to include a version of each primitive?

This is the right time to incorporate support for the logoed stud, but I'm not sure how it would fit into this schema.

[Santeri Piippo]

I wonder did my post somehow slip unnoticed, but I suggested having a p/logostuds for this purpose. Also I'd still suggest using LDConfig.ldr for this, it seems to be the proper place for this IMO. Furthermore I think we should also adopt the tag-value syntax of the !COLOUR meta command so as to future-proof it should we need to improve it sometime. This allows having tags for stud logos and addresses Chris's concerns about p/'s status as default.

This would make my suggestion something like this:

Code:

0 !PRIMITIVE-DETAIL FOLDER p/0         SEGMENTS 0            DESCRIPTION "Ultra-low resolution"
0 !PRIMITIVE-DETAIL FOLDER p/8         SEGMENTS 8            DESCRIPTION "Low resolution"
0 !PRIMITIVE-DETAIL FOLDER p           SEGMENTS 16 DEFAULT   DESCRIPTION "Standard"
0 !PRIMITIVE-DETAIL FOLDER p/logostuds SEGMENTS 16 STUDLOGOS DESCRIPTION "Logoed studs"
0 !PRIMITIVE-DETAIL FOLDER p/48        SEGMENTS 48           DESCRIPTION "High resolution"

[Chris Dee]

I like that and the way you have incorporated support for logoed studs.

I'd like some opinion from developers who would need to parse this, particularly the optional second parameter to SEGMENTS.

I have two concerns about incorporating this metadata into the existing LDConfig.ldr file
- are existing applications robust enough to ignore !PRIMITIVE-DETAIL meta-statements, or have they assumed (incorrectly) that the file is just for colour palette configuration information?
- would this need to be duplicated in LDCfgalt.ldr which would not be good (meta)data management practise?

[Ben Supnik]

Hi Guys,

Chris, if you don't like having 'implicit' ranking via the order of LOD definitions in the config file, then I think we need an explicit ranking.

In particular, the stud polygon count isn't quite good enough. For example:

- 16-gon studs with/without logos. Clearly the ones with logos represent a higher rendering load, but the stud side count is 16 for both.
- No-studs vs no-studs-and-interiors. If we wanted both levels of detail, they would both have a stud side count of 0.

I really like Santeri's tag-value proposal - it lets us flexibly add more meta data to LODs, and not every client has to care about all meta data. So...

The polygon edge count would be used to sequence the entries and would describe how circular elements are coded, but wouldn't prevent other primitives existing in the same folder.

I would like to see polygon edge count be separate from sequence/LOD cost because the polygon edge count is ambiguous in some cases. With Santeri's proposal we can add another tag.

Cheers
Ben

[Ben Supnik]

Hi Santeri,

I like the use of tag-value syntax, but for the 'default' and 'studlogos' values the syntax breaks - we have tags without values.

Ideally I would like to be able to extend this syntax in the future (adding new keywords) without breaking existing clients, which implies that a parser can parse the data without knowing all of the tags.

This syntax might be sort of stupid looking, but what if we do something like:

Code:

0 !PRIMITIVE-DETAIL FOLDER p/8         SEGMENTS 8               DESCRIPTION "Low resolution"
0 !PRIMITIVE-DETAIL FOLDER p           SEGMENTS 16 DEFAULT    1 DESCRIPTION "Standard"
0 !PRIMITIVE-DETAIL FOLDER p/logostuds SEGMENTS 16 STUDLOGOS  1 DESCRIPTION "Logoed studs"

The default and studlogos tags would then have a value of 0 or 1, their default (if omitted) is 0, and clients assume that every pair of words (with quotes forming a single big word) is a key-value pair.

If people think parsing quoted words is annoying we could also do this:

Code:

0 !PRIMITIVE-DETAIL "Low resolution"
0 !PRIMITIVE-DETAIL-INFO FOLDER p/8        
0 !PRIMITIVE-DETAIL-INFO SEGMENTS 8              

0 !PRIMITIVE-DETAIL "Standard"
0 !PRIMITIVE-DETAIL-INFO FOLDER p          
0 !PRIMITIVE-DETAIL-INFO SEGMENTS 16
0 !PRIMITIVE-DETAIL-INFO DEFAULT

0 !PRIMITIVE-DETAIL "Logoed studs"
0 !PRIMITIVE-DETAIL-INFO FOLDER p/logostuds
0 !PRIMITIVE-DETAIL-INFO SEGMENTS 16
0 !PRIMITIVE-DETAIL-INFO STUDLOGOS

I don't have strong aesthetic opinions on the file format (particularly if Allen writes the parser ;-) but I would like to be able to add keywords later without having to worry about existing clients.

cheers
Ben

[Santeri Piippo]

I think we should be consistent with LDConfig.ldr's !COLOUR definitions even though if we wouldn't be able to add these meta-commands directly into it. With !COLOUR, tags don't necessarily need values, there's attributes like PEARLESCENT, METAL, CHROME and RUBBER. Thus, by analogy, this would have such attributes like STUDLOGOS and DEFAULT.

For the same reason - and for LDraw's overall policy of having one statement per line - I also think we should have these as one-liners as well. Going by this, a would-be-spec should, IMO, say very clearly that adding other attributes or even meta-commands should be possible and parsers should take that into account. I wonder does LDConfig.ldr have such a disclaimer anywhere?

[Ben Supnik]

I think we should be consistent with LDConfig.ldr's !COLOUR definitions even though if we wouldn't be able to add these meta-commands directly into it. With !COLOUR, tags don't necessarily need values, there's attributes like PEARLESCENT, METAL, CHROME and RUBBER. Thus, by analogy, this would have such attributes like STUDLOGOS and DEFAULT.

Hi Santeri,

Let me ask you this: if a client does not have a complete list of keywords (but does know whether, for any _known_ keyword, the keyword has a value associated), what is the correct algorithm for parsing the line/skipping unknown tags?

cheers
ben

[Santeri Piippo]

I just use a parser class and read tokens based on a list of known keywords. With syntax like this, unknowns should get ignored anyway, otherwise it kind of defeats the point of the tag-value syntax.

https://github.com/slatenails/ldforge/bl...config.cpp

[Ben Supnik]

I just use a parser class and read tokens based on a list of known keywords. With syntax like this, unknowns should get ignored anyway, otherwise it kind of defeats the point of the tag-value syntax.

Ah - the implication here is that a FUTURE tag strings (not yet known to us now) must never occur as a value string. This is a little bit unnerving since some of the tags (e.g. human readable name, folder name) could be anything.

So...this isn't my favorite proposal for that reason, but it can be made it work, if similarity to existing tech is highly desirable.

Cheers
Ben

[Allen Smith]

But now you have introduced order dependency and content limitations. Parsing syntax in this kind of format is an unnecessary headache, because it's inherently very fragile.

Most structured languages have solved this problem. Look to XML for inspiration, not LDraw. If you want key-value pairs, then you either need to force keys and values, or you need a symbol to indicate a key with no value, e.g. <tag/>. You also need an escape sequence for including the delimiter(s) themselves in values, e.g. <tag>&lt;</tag>. Frankly, it's a lot easier to write parsing code for something like that than the alternative.

Allen

[Roland Melkert]

You could use something similar to what I've setup for LDCad meta's, for example:

Code:

0 !LDCAD PATH_POINT [type=bezier] [posOri=0 0 0 1 0 0 0 0 1 0 -1 0] [prevCPDist=25] [nextCPDist=25]

It's kinda like a flattened ini file. This way order and future parameters are parser independent.

[Chris Dee]

My initial thoughts were to suggest XML rather than continuing to develop the LDraw style of keyword-value syntax, but I didn't think I would get much support here.

Well-formed XML is easy to parse (at least in the programming languages with which I am familiar), self-descriptive and extensible with care, but suffers a little in human-readability. To my mind it is ideally suited to configuration files, where the bloat of <...> tags is less of an issue than in large "data" files.