Welcome! Log In Create A New Profile

Advanced
Re: Compressed Library was Re: Compressed LDraw files
January 31, 2013 12:26PM
I looked at compression efficiency when we were selecting a compression format for X-Plane scenery data. When maxing out compression (not always a good idea as it increases encode time significantly) I found that from worst to best compression ratio the list went:

pkzip, bzip2, rar, and 7zip.

Rar is heavily license-entangled, so it's not really an option. We used 7zip because it had the best ratio, but for LDraw, I'd recommend pkzip because of the wide availability of simple, clean, portable compression/decompression utils. (7zip's quite a bit messier.)

I also found that compression ratios were very sensitive to source content. Our file format is a heavily bit-packed binary format, and it has optional RLE encoded in some cases. I found that with 7zip, the RLE encoding significantly worsened 7-zips compression ratio...the moral of the story was: let the experts do the compression.

So...I suspect that the compression ratio differences are coming from iteratively superior compression algorithms (no surprise the oldest tech does the worst) but YMMV based on file contents.

Honestly, the more I read this discussion, the less valuable compression suppot seems, relative to other possible development work. My LDraw file is 138 MB on disk. By comparison, LDD is using 562 MB on disk, itunes 253 MB, Keynote 562 MB, and FireFox 184 MB. My MOC airport is only 6.6 MB on disk for 22,000 bricks and months of my lif

I'm not saying that this is okay or that I'm happy that my web browser is approaching nearly 200 MB in size, but it seems to me that the data sizes the LDraw software suite deals with are quite tame. :-)

Anyway, sorry for the rant...my take-away point is that given multiple options, I would suggest we optimize for developer convenience and not storage efficiency, since that last few % improvement from pkzip to 7zip aren't going to turn into a huge user experience win, but I think we are limited on volunteer dev time.
SubjectAuthorViewsPosted
Compressed LDraw files Roland Melkert416January 29, 2013 12:33PM
Re: Compressed LDraw files Tim Gould191January 29, 2013 12:44PM
Re: Compressed LDraw files Travis Cobbs200January 29, 2013 12:49PM
Re: Compressed LDraw files Roland Melkert200January 29, 2013 02:01PM
Re: Compressed LDraw files Tim Gould199January 29, 2013 02:10PM
Re: Compressed LDraw files Ben Supnik221January 29, 2013 10:36PM
Re: Compressed LDraw files Roland Melkert217January 30, 2013 10:52AM
Re: Compressed LDraw files Ben Supnik233January 30, 2013 12:59PM
Re: Compressed LDraw files Allen Smith213January 30, 2013 09:17AM
Compressed Library was Re: Compressed LDraw files Travis Cobbs230January 30, 2013 09:40AM
Re: Compressed Library was Re: Compressed LDraw files Philippe Hurbain211January 30, 2013 09:49AM
Re: Compressed Library was Re: Compressed LDraw files Allen Smith199January 30, 2013 08:43PM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs200January 30, 2013 10:27PM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert230January 31, 2013 09:55AM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs222January 31, 2013 11:01AM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert218January 31, 2013 11:07AM
Re: Compressed Library was Re: Compressed LDraw files Travis Cobbs222January 31, 2013 11:24AM
Re: Compressed Library was Re: Compressed LDraw files Ben Supnik215January 31, 2013 12:26PM
Re: Compressed Library was Re: Compressed LDraw files Roland Melkert225January 31, 2013 01:09PM
Re: Compressed Library was Re: Compressed LDraw files Steffen276February 02, 2013 05:34PM
Re: Compressed Library was Re: Compressed LDraw files Tim Gould241January 31, 2013 04:02PM
Re: Compressed LDraw files Roland Melkert235January 30, 2013 10:57AM
Re: Compressed LDraw files Ben Supnik223January 30, 2013 01:15PM
Re: Compressed LDraw files Roland Melkert222January 30, 2013 02:07PM
Re: Compressed LDraw files Allen Smith189January 30, 2013 08:39PM
Re: Compressed LDraw files Roland Melkert223January 31, 2013 01:06PM
Re: Compressed LDraw files Travis Cobbs227January 31, 2013 01:59PM
Re: Compressed LDraw files Travis Cobbs206January 29, 2013 12:46PM
Re: Compressed LDraw files Roland Melkert196January 29, 2013 01:57PM
Re: Compressed LDraw files Steffen205January 29, 2013 03:23PM
Re: Compressed LDraw files Travis Cobbs234January 29, 2013 04:17PM
Re: Compressed LDraw files Roland Melkert233January 30, 2013 10:49AM
Re: Compressed LDraw files Steffen200February 02, 2013 05:33PM



Sorry, only registered users may post in this forum.

Click here to login