LDraw.org Discussion Forums

Full Version: Auto correcting (some) defects in the official library
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6
Ok I finished the changes, but it generated a 1000 files Smile

This seems to be mostly because of case mismatches though, I could leave those out if those are the only corrections in a file.

It also made me wonder about all the fuss about the auto corrections during loading as it took a release version of LDCad about 90 seconds to load and correct all 8193 parts (recursively) in the parts folder of the 1401 lib.

This taking into account loading 8000 parts (taking just below 1GB of memory) will probably never happen in a normal drawing session.

Anyone still interested in processing those files by hand?
Quote:I'm not sure about "type 5 line with missing or duplicate control points (degrade to normal line)"
It really happens though, take for example this one:

[attachment=1392]
Could you give us some example files?
Before and after fix?
This might be easier to get the idea.

[attachment=1393]

[attachment=1394]

[attachment=1395]

One side note, when I feed the corrected library again there will still be a couple of problems but loading the entiry library only generated about 65 warnings/errors, these are things like quads with multiple duplicate points, triangle with dup pnt, line with dup pnt, Those kinds of problems are not fixed and therefore currently also not saved.

I could do something like adding a 0 !INVALID in front of them, although it's probably easier for the reviewers to just follow the notes in the log file.

relevant log portion:
Code:
00033 | 2014-11-16_23:11:15 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2507p01.dat | line 41 | CLIP OFF not supported, culling disabled for whole part.
00034 | 2014-11-16_23:11:19 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2677.dat | line 321 | Duplicate point detected, line will not be used.
00035 | 2014-11-16_23:11:19 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2677.dat | line 500 | Duplicate point detected, line will not be used.
00036 | 2014-11-16_23:11:40 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\u9064.dat | line 27 | CLIP OFF not supported, culling disabled for whole part.
00037 | 2014-11-16_23:11:41 | Warning  | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4116604.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00038 | 2014-11-16_23:11:50 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 210 | Multiple duplicate points detected, quad will not be used.
00039 | 2014-11-16_23:11:50 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 214 | Multiple duplicate points detected, quad will not be used.
00040 | 2014-11-16_23:11:50 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 219 | Multiple duplicate points detected, quad will not be used.
00041 | 2014-11-16_23:11:50 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 223 | Multiple duplicate points detected, quad will not be used.
00042 | 2014-11-16_23:11:55 | Warning  | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\54869.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00043 | 2014-11-16_23:11:57 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\57895p01.dat | line 17 | CLIP OFF not supported, culling disabled for whole part.
00044 | 2014-11-16_23:11:57 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\s\59228s03.dat | line 1303 | Duplicate point detected, line will not be used.
00045 | 2014-11-16_23:12:00 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6156.dat | line 101 | Duplicate point detected, triangle will not be used.
00046 | 2014-11-16_23:12:00 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6156.dat | line 117 | Duplicate point detected, line will not be used.
00047 | 2014-11-16_23:12:00 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6159.dat | line 70 | Multiple duplicate points detected, quad will not be used.
00048 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 688 | Duplicate point detected, line will not be used.
00049 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 689 | Duplicate point detected, line will not be used.
00050 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 690 | Duplicate point detected, line will not be used.
00051 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 691 | Duplicate point detected, line will not be used.
00052 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 692 | Duplicate point detected, line will not be used.
00053 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 693 | Duplicate point detected, line will not be used.
00054 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 694 | Duplicate point detected, line will not be used.
00055 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 695 | Duplicate point detected, line will not be used.
00056 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 696 | Duplicate point detected, line will not be used.
00057 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 697 | Duplicate point detected, line will not be used.
00058 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 698 | Duplicate point detected, line will not be used.
00059 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 699 | Duplicate point detected, line will not be used.
00060 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 701 | Multiple duplicate points detected, quad will not be used.
00061 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 702 | Multiple duplicate points detected, quad will not be used.
00062 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 703 | Multiple duplicate points detected, quad will not be used.
00063 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 704 | Multiple duplicate points detected, quad will not be used.
00064 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 705 | Multiple duplicate points detected, quad will not be used.
00065 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 706 | Multiple duplicate points detected, quad will not be used.
00066 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 707 | Multiple duplicate points detected, quad will not be used.
00067 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 708 | Multiple duplicate points detected, quad will not be used.
00068 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 709 | Multiple duplicate points detected, quad will not be used.
00069 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 710 | Multiple duplicate points detected, quad will not be used.
00070 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 711 | Multiple duplicate points detected, quad will not be used.
00071 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 712 | Multiple duplicate points detected, quad will not be used.
00072 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 713 | Multiple duplicate points detected, quad will not be used.
00073 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 714 | Multiple duplicate points detected, quad will not be used.
00074 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 715 | Multiple duplicate points detected, quad will not be used.
00075 | 2014-11-16_23:12:01 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 716 | Multiple duplicate points detected, quad will not be used.
00076 | 2014-11-16_23:12:02 | Warning  | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\62930.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00077 | 2014-11-16_23:12:04 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6551.dat | line 868 | Duplicate point detected, line will not be used.
00078 | 2014-11-16_23:12:04 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6551.dat | line 879 | Duplicate point detected, line will not be used.
00079 | 2014-11-16_23:12:05 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\70022.dat | line 78 | CLIP OFF not supported, culling disabled for whole part.
00080 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 243 | Duplicate point detected, line will not be used.
00081 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 245 | Duplicate point detected, line will not be used.
00082 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 259 | Duplicate point detected, line will not be used.
00083 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 261 | Duplicate point detected, line will not be used.
00084 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 275 | Duplicate point detected, line will not be used.
00085 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 277 | Duplicate point detected, line will not be used.
00086 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 291 | Duplicate point detected, line will not be used.
00087 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 293 | Duplicate point detected, line will not be used.
00088 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 307 | Duplicate point detected, line will not be used.
00089 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 309 | Duplicate point detected, line will not be used.
00090 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 323 | Duplicate point detected, line will not be used.
00091 | 2014-11-16_23:12:09 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 325 | Duplicate point detected, line will not be used.
00092 | 2014-11-16_23:12:12 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\973p60.dat | line 81 | Duplicate point detected, triangle will not be used.
00093 | 2014-11-16_23:12:12 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\973p60.dat | line 82 | Duplicate point detected, triangle will not be used.
00094 | 2014-11-16_23:12:13 | Error    | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\99148p01.dat | line 22 | CLIP OFF not supported, culling disabled for whole part.
00095 | 2014-11-16_23:12:14 | Warning  | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\979.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00096 | 2014-11-16_23:12:14 | Warning  | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\980.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
Quote:Anyone still interested in processing those files by hand?
By hand, no, but I would appreciate if all parts with "obvious" errors were automatically corrected!
- "This seems to be mostly because of case mismatches though," What do you mean, reference to eg. 4-4DISC.DAT instead of 4-4disc.dat? If so, that's an excellent exemple of thing that can be automatically corrected without further ado.
- "CLIP OFF not supported, culling disabled for whole part." NOCLIP is presently in BFC specification, does this means that we should change spec? This has been used for transparent patterned parts here, where we should see back face of pattern through part. Latest LDView beta does work fine now even without NOCLIP, but LDCad do not show patterns on back face (except for those that contain NOCLIP!).
To me, autofixing some hundreds files already looks like a great improvement.

Roland Melkert Wrote:Although I'm wondering do things like bow ties really have to go though the full review stage? Isn't it easier to just look at the differences text wise combined with a quick LDView checkup.

I agree. These are things routinely done at runtime, i'm pretty sure the software is tried and tested. And even in the worst case, it's not like we cando much damage, since all this parts are already broken to begin with.
+1 !!!
Go for an automatic cleanup as much as possible!
I agree 100% with this.
Instead of talking for days or weeks about fixing broken parts, let's just fix them. Smile
How should a corrected "bowtie quad" be controlled to have the correct winding???
Quote:NOCLIP is presently in BFC specification, does this means that we should change spec?
No, those errors are the result of the bfc implementation (clipping on/off at part level) I choose for LDCad it's not a line formatting error. The above log chunk are all >=warning messages while reading the entire (corrected) library, most of them are unrecoverable malformed lines though.

Quote:By hand, no, but I would appreciate if all parts with "obvious" errors were automatically corrected!
I did also a test leaving out the case mismatch only files which resulted in ~300 files, which might be a more manageable chunk.

I'll compile zip files with the clanged files (one with the cap change only files, and one with the rest) later tonight so people can take a look them selves.
Pages: 1 2 3 4 5 6