I need a fast and frugal renderer

[Tore Eriksson]

I wouldn't complain so much about every new step in the LDraw format evolution, if it wasn't that I'm 100% depending on good old L3Lab. There is absolutely no viewer or editor that comes remotely close to its speed and its resource economy. Sorry, Roland's new editor included.

But L3Lab doesn't support colours in "the forbidden zone" as I call it. This is a growing problem for me.
And L3Lab doesn't support space characters in file references either.
And almost every change in the LDraw file specs is done without the least consideration to programs like L3Lab.

I would of course compile a fast and frugal renderer myself if only I had the programming skills to do it, but that's impossible. I'm not even close to that.

Apparantly, it seems to me all focus is on nifty details and impressive effects when it comes to viewers made today. But if it was possible eleven years ago to make an LDraw viewer that still is able to produce fast renderings, wouldn't it be possible to make the same today? And support for LDConfig and the latest LDraw file specs shouldn't slow down things noticable, should it?

If somebody could produce such a viewer, I will (most likely) no longer protest against the other LDraw programs and utilities we leave behind as we "develop" (hmm?) the LDraw standard. But like I said, I still see no real alternative to L3Lab when making huge LDraw models.

/Tore

[Roland Melkert]

Had to look up the meaning of "frugal"

Maybe you should consider dividing models into (even) smaller portions.

Although I'm not quite understanding the L3Lab is faster statement, it's considerably slower than e.g. LDView or LDcad on my system. It's memory usage is impressive though (only 20MB for the 10179 model).

So I'm thinking you haven't got decent 3D hardware or an all around (very) old system.

If that's the case you could expect things to get worse because, most developers will assume you have (semi) recent hardware and hardware acceleration.

[Orion Pobursky]

If that's the case you could expect things to get worse because, most developers will assume you have (semi) recent hardware and hardware acceleration.

I second this thought. Even OSs nowadays are requiring discreet 3D graphics solutions.

[Tore Eriksson]

Roland Melkert Wrote:
-------------------------------------------------------
> So I'm thinking you haven't got decent 3D hardware
> or an all around (very) old system.
>

Well, it's not exactly vintage.
An automatic Computer Overview reports the following:


*********************************************************

SiSoftware Sandra

Computer
Model : OEM System Product Name
Workgroup : *****
Host Name : *****
User : *****

Processor
Model : Pentium® Dual-Core CPU E5300 @ 2.60GHz
Speed : 2.62GHz
Cores per Processor : 2 Unit(s)
Type : Dual-Core
Integrated Data Cache : 2x 32kB, Synchronous, Write-Thru, 8-way, 64 byte line size
L2 Cache : 2MB, ECC, Synchronous, ATC, 8-way, 64 byte line size, 2 threads sharing

Computer
Mainboard : ASUS P5QPL-AM
BIOS : AMI (OEM) 0314 07/22/2009
Bus(es) : ISA X-Bus PCI PCIe IMB USB i2c/SMBus
Multi-Processor (MP) Support : No
Multi-Processor Advanced PIC (APIC) : Yes
Total Memory : 4GB DIMM DDR2 DDR3

Chipset
Model : ASUS G41 DRAM Controller
Front Side Bus Speed : 4x 202MHz (808MHz)
Total Memory : 4GB DIMM DDR2 DDR3
Channels : 2
Memory Bus Speed : 2x 404MHz (808MHz)

Memory Module(s)
Memory Module : Corsair CM2X2048-6400C5 2GB DIMM DDR2 PC2-6400U DDR2-800 (5-5-5-18 3-22-6-3)
Memory Module : Corsair CM2X2048-6400C5 2GB DIMM DDR2 PC2-6400U DDR2-800 (5-5-5-18 3-22-6-3)

Video System
Video Adapter : NVIDIA GeForce 8400 GS (8 SM4.0 1.4GHz, 512MB DDR2 798MHz 64-bit, PCIe 2.00 x16)
Video Adapter : NVIDIA GeForce 8400 GS (8 SM4.0 1.4GHz, 512MB DDR2 798MHz 64-bit, PCIe 2.00 x16)

Graphics Processor
OpenCL GP Processor : NVIDIA GeForce 8400 GS (8SP 1C 1.4GHz, 512MB DDR2 798MHz 64-bit)
Compute Shader Processor : NVIDIA GeForce 8400 GS (8SP 1.4GHz, 512MB DDR2 798MHz 64-bit)
CUDA GP Processor : NVIDIA GeForce 8400 GS (8SP 1C 1.4GHz, 512MB DDR2 798MHz 64-bit)

Storage Devices
SAMSUNG HD753LJ (750.2GB, SATA300, 3.5", 7200rpm, 32MB Cache) : 699GB (C
TSSTcorp CDDVDW SH-S223B (3.7GB, SATA150, DVD+-RW, CD-RW, 2MB Cache) : 3GB (E

Logical Storage Devices
System Reserved : 100MB (NTFS)
Hard Disk (C : 699GB (NTFS) @ SAMSUNG HD753LJ (750.2GB, SATA300, 3.5", 7200rpm, 32MB Cache)
Bev_04B (E : 3GB (CDFS) @ TSSTcorp CDDVDW SH-S223B (3.7GB, SATA150, DVD+-RW, CD-RW, 2MB Cache)

Peripherals
LPC Hub Controller 1 : ASUS 82801GB/GR (ICH7) LPC Interface Controller
LPC Legacy Controller 1 : T0 B0-73
Audio Device : ASUS 82801G (ICH7) High Definition Audio
Audio Codec : Realtek Semi High Definition Audio
Serial Port(s) : 1
Parallel Port(s) : 1
Disk Controller : ASUS 82801G (ICH7) Ultra ATA Storage Controller
Disk Controller : ASUS 82801GB/GR/GH (ICH7) Serial ATA Storage Controller
Disk Controller : Silicon Image SiI 0680 UltraATA/133 Medley RAID Controller
USB Controller 1 : ASUS 82801G (ICH7) USB Universal Host Controller
USB Controller 2 : ASUS 82801G (ICH7) USB Universal Host Controller
USB Controller 3 : ASUS 82801G (ICH7) USB Universal Host Controller
USB Controller 4 : ASUS 82801G (ICH7) USB Universal Host Controller
USB Controller 5 : ASUS 82801G (ICH7) USB 2.0 Enhanced Host Controller
SMBus/i2c Controller 1 : Intel ICH SMBus

Printers and Faxes
Printer : HP Color LaserJet 2600n (600x600, USB, Colour)

Network Services
Network Adapter : Atheros AR8121/AR8113/AR8114 PCI-E Ethernet Controller (Ethernet, 100Mbps)

Power Management
Mains (AC) Line Status : On-Line

Operating System
Windows System : Microsoft Windows 7 Ultimate 6.01.7601 (Service Pack 1)
Platform Compliance : x64

Windows Experience Index
Current System : 3.6

*********************************************************

[Roland Melkert]

Hmm, these specs should result (in LDCad at least) in >100fps for most models (<1000 pieces).

Aren't you getting that?

[Tore Eriksson]

I'm terribly sorry, but I have no idea of how to interpret that to me very strange information on Rendertime and fps rate I get from LDCad.

I load a model, it takes maybe one second to render, but LDCad may claim 1 ms (~1000 fps) and when I hover over it with the mouse pointer, those values change all the time. That's extremely odd, because I haven't even rotated or changed it in any way at all, so what those perpetually changed values stand for, I have no clue at all. Sorry.

[Tore Eriksson]

I used another model, the red pickup truck from Datsville (68 parts), as a benchmark.
It took 0.015 seconds to load and render with L3Lab.
LDView does not provide that information AFAK (Niether soes LDCad), but the fps rate is about 75 when in spin.
Once a large model is loaded, LDView is much faster than L3Lab to rotate it, since L3Lab basically reloads the model from scratch. (Yes, it does cache parts by default, but I don't see any real difference when I uncheck that option)

Now, setting a model to keep spinning is great fun to play with, but has no practical use IMO. Once a file is loaded, parsed and rendered, LDView is impressively fast to make it spin. But the important thing for me is the time it loads the first time, or when the model file has been changed, not how fast it can spin. And that's not made in 1/75 second. And, like I said many times before, the larger the model, the more dramatic the difference in loading time.

The bottom line is, I still need a good replacement for L3Lab, absolutely max 50% slower when it comes to loading models. I know I can't demand it from the LCad community, but it can't be that difficult to make for anyone who can make such impressive top-notch viewers when it comes to all the other values, like realism, shadows, zooming, rotating, and all the different ways of presenting the images, can it? And I'm surprised that it seems like I'm all alone wishing for that(?) I can understand the fascination of photo realism even though I'm not into it. LEGO parts and models form so very unrealistic representations of the real world anyway. But there must be somebody else in this world more than me who likes to build really large and complex models and find it unacceptably tedious to work with the current set of software tools?

/Tore

[Steffen]

I used to use L3Lab as well for a long time, but then switched to LDView finally and am happy now.

The problem in my eyes currently seems to be that each tool has its advantages
in some special features, and it may fall behind in some other aspect, e.g. loading performance etc.

So what could be a possible solution of this? In my eyes, creating even another tool will not solve this problem.
That tool might be better in some aspect which the tool's author has in main focus, but again,
it will lack some feature, or will have other problems.
Of course it is tempting to create new tools, especially as a developer.
However, my suggestion would be to get in contact with the authors of the existing tools
and help them work on improving their tools. I think that the benefit in sum for the community
will be bigger than having yet-another-tool-with-pros-and-cons.

When I had contacted Travis in some BFC regards of the LDView tool some time ago,
I was simply amazed how fast he responded, and also the code changes appeared
even the next day. Of course, improving the loading speed of LDView will take longer
and will take some investigation and experiments, but doing that in the long range
IMHO will be a better solution than having 2 tools: one spinning fast, and one loading fast :-D :-D

[Roland Melkert]

The rendering time is the time the last (partial)redraw took to complete. This time (rounded to ms) is then used to give a raw fps, this value can fluctuate greatly depending on activity and performance.

But it's only a raw guidance, I could make it more precise in the next version by using floating point internally (so it doesn't go from 1000fps to 500fps to 333fps etc). Although the fact it's fluctuating this much is the result of it being fast (the slower, the more constant the fps should be). So I'm not sure why you still find it slow. It will also redraw on any change (e.g. different hot part).

But you shouldn't be to fixated on the fps, especially while rotating, placing parts etc is fluently.

[Roland Melkert]

Any renderer has to be optimized for the task it's used for.

LDView is optimized for quality viewing, so it takes all the model data and compiles it into a single object so the VGA hardware can do it thing efficient.

Editors can't do that cause you will be moving parts, so they need to optimize on part level in order to find balance between rendering speed and quality.

L3Pub seems to do no optimizations (or very little) resulting in fast loading times (no mesh preparation etc) but slow rendering (drawing single triangles etc).

[Travis Cobbs]

I agree that with you that your system seems powerful enough. I also agree that the initial load time for files isn't one of LDView's strong points. (It is on my list of things to try to improve.) One thing you can do now to improve load time in LDView (at the expense of rendering performance) is to set the "Memory usage" setting in LDView's General tab in Preferences to "Low". This will skip the final step in the model load. (For the OpenGL programmers out there, the final step is display list compile, which takes a little bit of time, and tends to use a lot of memory.)

Unfortunately, it doesn't seem to make a very big impact on load time. On my test computer, Orion's rendition of 10030 loads in about 5.8 seconds with memory usage set to "High", and 5.4 seconds with memory usage set to "Low". Frame rate once loaded has a bigger difference, with 42FPS when set on "High", and 27FPS when set on "Low". Please note that both load times were timed with my stopwatch that only has .2 second accuracy, and both were timed for a reload (F5), since the first load of the file has to load all the parts from disk, while subsequent loads get them from the cache. Since I didn't feel like rebooting my computer between test runs, that's the best timing info I have.

[Allen Smith]

Tore Eriksson Wrote:
-------------------------------------------------------

> The bottom line is, I still need a good
> replacement for L3Lab, absolutely max 50% slower
> when it comes to loading models.

Tore,

I know you're not using a Mac, but I figure you might appreciate some generally applicable background information.

Part of your frustration is that you conflate loading time and drawing time. They really are very different operations, it's just that you apparently don't notice it because of your workflow. In any modern 3D graphics API (having vastly more power than non-hardware-accelerated systems), the structure in which you submit the geometry data is critical to fast drawing. LDraw files are not naturally favorable to that structure and must be massaged quite a bit to produce something which draws reasonably fast. This process elongates loading time, but results in drawing performance improvements of over an order of magnitude. In any graphical editor, the model is redrawn over and over; drawing time is vastly more important than a few seconds of parsing and reformatting. Likewise, any viewer that wants to support drawing the same model more than once must optimize for drawing over loading.

Loading impact can be reduced by amortization and caching. In Bricksmith, the first load of Orion's 10030 takes 5.8 seconds on my three-year-old Mac. The second load (provided you don't quit the program in between) takes 0.06 seconds, simply because I only throw away the 10030 itself, not the individual parts inside it. Even better, using a hardware-accelerated graphical editor to build your model would incur no loading time whatsoever while adding and moving parts, and would only be bound by the drawing fps. In which case, faster drawing wins the day.

Allen

[Tim Gould]

Hi Allen,

One thing I've often wondered is whether or not a 'render friendly' format for the LDraw library wouldn't be useful. For example if you and Travis (as rulers of the two most common rendering tools) could come up with a standard format you could cache the parts library into the new format and add new parts as needed.

Or more precisely, when you run an LDraw viewer/editor the following happens
1) Check to see if LDrawParts.qik exists - if yes, go to 3)
2) Generate LDrawParts.qik with parts in format useful for fast 3d rendering
3) Load model. If it contains any part does not exist in LDrawParts.qik but exists in /ldraw/parts conver and add to LDrawParts.qik.
4) Use preprocessed parts in LDrawParts.qik to render

Of course stage 2) could be ignored and you simply add parts as they appear in models. Or 2) could be voluntary.

This way you wouldn't break the standard format but could have an in-between format. And if you and Travis could come up with a standard a user could even share the file.

Tim

[Travis Cobbs]

I've actually considered doing something like this but didn't do so for two reasons:

• It's a fairly involved feature, and requires careful thought. Cached data needs to contain a timestamp, and any parts that are updated in the library need to automatically trigger regeneration of the cached data.
  
• LDView's architecture makes it less useful than it otherwise might be (see below).
  

If you look at the status bar while loading a big model in LDView, you might notice that it has a number of stages: Loading, Calculating Size, Parsing, Compiling.

The Loading stage loads the data from the LDraw files into memory in structures that contain all the data from the files, but that's about it. (It also constructs a tree representing all the files referenced in the model, reusing files that are reused.) Very specifically the data structures at this point are not graphics-specific (and in fact get used directly for my POV exporting). I could create a binary cache of the data at this point, but that's only about 20% of the total load time, so it's not particularly useful.

The Calculating Size stage does basically what it says it does, and wouldn't be aided by caching of the parts.

The Parsing stage transforms the data from the original format into the format I need for my graphics engine, and also performs primitive substitution and curve smoothing. I could create a binary cache of the final processed data, but LDView relies on the original data from the Loading stage for various functions, so I'd need to either create a separate cache for that, or simply still load the original LDraw files for that step, or go through a major rearchitecture of LDView so it didn't rely on that data. All of those options are fairly major, so caching has been put on the back burner.

To make things even more complicated, the results of both the Loading stage and the Parsing stage are different based on what options are selected in LDView. As one example, if you ask for low-res studs, it loads the low-res stud files from disk instead of the normal ones. It doesn't load both, so the data I get is different depending on that setting. Similarly, if you turn on Primitive Substitution, you get different geometry out of the Parsing stage than you get if that feature is disabled. So any caching in LDView would have to take this into account, thus complicating things quite a bit more than they already were.

[Travis Cobbs]

I forgot the one setting that is likely to speed up loading the most: disable the "Smooth Curves" setting on the Effects tab of Preferences. On the same 10030 model on a different computer, unchecking this box brought the load time from 10 seconds down to 5 seconds (still with memory set to Low).

[Tim Gould]

Thanks for the explanation. This all makes sense.


Tim

[Don Heyse]

Is the load/calc/parse/compile time a problem because the viewer (l3lab or ldview) is being used in polling mode to display work done in a a separate program such as lddp or a text editor?

If so, would it be possible for ldview to keep most of that work from the previous poll and only recompile the actual bits that have changed? Or does it already do this? If ldview could keep the recompile time under a second then it'd make a really nice editor component.

[Travis Cobbs]

LDView reloads completely when in polling mode. This could be fixed, but would require a lot of work due to the fact that changing settings in LDView can also force it to either reload or reparse. Adding file-changed reload logic into the existing settings-triggered reload logic is a lot of work. Also, I the polling currently only pays attention to the top-level file, and while it would be nice for it to pay attention to all loaded files and only reload the ones that actually changed, that's also a lot of work.

[Don Heyse]

Ah well, one of these days I'll convince you to add an editing mode to ldview.

Is there no easy way (be it meta command, socket, named pipe, or whatever) that an outside program could tell ldview to add, delete, or move just one part in the main model file without starting over from scratch? That would cover the vast majority of the editing operations. If those three ops were fast then ldview would be fine for creating and editing ldraw files.

Have fun,

Don

[Travis Cobbs]

I've spent a lot less time on LDView in the past year than probably any year since its original release. So changing its focus like that just seems unlikely. It could be done, with probably relatively minor changes to the graphics engine, but making a good editor is a huge task. And with Roland now actively creating a new OpenGL-based cross-platform editor (LDCad), I don't really think there's much point.

Having said that, LDView is GPL, and the latest source is always on sourceforge.net. If someone else wanted to create an editor based on it (or modify it to be an editor), there's nothing stopping them. I'd even try to provide anyone trying to do so a crash course in LDView's overall architecture. It would probably take a fair bit of work to convince me to let the editor functionality become a part of the mainstream LDView, but I wouldn't rule out a separate editor app based on the same source tree for the underlying loading and rendering.

[Jean-Philippe Ouellet]

Yup, LDview the editor... that wouldn't be a misleading name at all