Welcome! Log In Create A New Profile

Advanced
Re: Compressed Library was Re: Compressed LDraw files
January 31, 2013 12:26PM
I looked at compression efficiency when we were selecting a compression format for X-Plane scenery data. When maxing out compression (not always a good idea as it increases encode time significantly) I found that from worst to best compression ratio the list went:

pkzip, bzip2, rar, and 7zip.

Rar is heavily license-entangled, so it's not really an option. We used 7zip because it had the best ratio, but for LDraw, I'd recommend pkzip because of the wide availability of simple, clean, portable compression/decompression utils. (7zip's quite a bit messier.)

I also found that compression ratios were very sensitive to source content. Our file format is a heavily bit-packed binary format, and it has optional RLE encoded in some cases. I found that with 7zip, the RLE encoding significantly worsened 7-zips compression ratio...the moral of the story was: let the experts do the compression.

So...I suspect that the compression ratio differences are coming from iteratively superior compression algorithms (no surprise the oldest tech does the worst) but YMMV based on file contents.

Honestly, the more I read this discussion, the less valuable compression suppot seems, relative to other possible development work. My LDraw file is 138 MB on disk. By comparison, LDD is using 562 MB on disk, itunes 253 MB, Keynote 562 MB, and FireFox 184 MB. My MOC airport is only 6.6 MB on disk for 22,000 bricks and months of my lif

I'm not saying that this is okay or that I'm happy that my web browser is approaching nearly 200 MB in size, but it seems to me that the data sizes the LDraw software suite deals with are quite tame. :-)

Anyway, sorry for the rant...my take-away point is that given multiple options, I would suggest we optimize for developer convenience and not storage efficiency, since that last few % improvement from pkzip to 7zip aren't going to turn into a huge user experience win, but I think we are limited on volunteer dev time.
SubjectAuthorViewsPosted
Compressed LDraw files Roland Melkert668January 29, 2013 12:33PM
Re: Compressed LDraw files Tim Gould324January 29, 2013 12:44PM
Re: Compressed LDraw files Travis Cobbs329January 29, 2013 12:49PM
Re: Compressed LDraw files Roland Melkert318January 29, 2013 02:01PM
Re: Compressed LDraw files Tim Gould318January 29, 2013 02:10PM
Re: Compressed LDraw files Ben Supnik376January 29, 2013 10:36PM
Re: Compressed LDraw files Roland Melkert369January 30, 2013 10:52AM
Re: Compressed LDraw files Ben Supnik388January 30, 2013 12:59PM
Re: Compressed LDraw files Allen Smith368January 30, 2013 09:17AM
Compressed Library was Re: Compressed LDraw files Travis Cobbs388January 30, 2013 09:40AM
Re: Compressed Library was Re: Compressed LDraw files Philippe Hurbain363January 30, 2013 09:49AM
Re: Compressed Library was Re: Compressed LDraw files Allen Smith333January 30, 2013 08:43PM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs345January 30, 2013 10:27PM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert395January 31, 2013 09:55AM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs369January 31, 2013 11:01AM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert357January 31, 2013 11:07AM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs364January 31, 2013 11:24AM
Re: Compressed Library was Re: Compressed LDraw files Ben Supnik376January 31, 2013 12:26PM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert372January 31, 2013 01:09PM
Re: Compressed Library was Re: Compressed LDraw files Steffen465February 02, 2013 05:34PM
Re: Compressed Library was Re: Compressed LDraw files Tim Gould385January 31, 2013 04:02PM
Re: Compressed LDraw files Roland Melkert378January 30, 2013 10:57AM
Re: Compressed LDraw files Ben Supnik373January 30, 2013 01:15PM
Re: Compressed LDraw files Roland Melkert363January 30, 2013 02:07PM
Re: Compressed LDraw files Allen Smith330January 30, 2013 08:39PM
Re: Compressed LDraw files Roland Melkert360January 31, 2013 01:06PM
Re: Compressed LDraw files Travis Cobbs371January 31, 2013 01:59PM
Re: Compressed LDraw files Travis Cobbs337January 29, 2013 12:46PM
Re: Compressed LDraw files Roland Melkert317January 29, 2013 01:57PM
Re: Compressed LDraw files Steffen307January 29, 2013 03:23PM
Re: Compressed LDraw files Travis Cobbs380January 29, 2013 04:17PM
Re: Compressed LDraw files Roland Melkert380January 30, 2013 10:49AM
Re: Compressed LDraw files Steffen365February 02, 2013 05:33PM



Sorry, only registered users may post in this forum.

Click here to login