+- LDraw.org Discussion Forums (https://forums.ldraw.org)
+-- Forum: General (https://forums.ldraw.org/forum-12.html)
+--- Forum: Official File Specifications/Standards (https://forums.ldraw.org/forum-32.html)
+--- Thread: Auto correcting (some) defects in the official library (/thread-14597.html)
Pages:
1
2
Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-15
Like mentioned in some of the other threads, the current official library suffers from a number of defects which can be corrected automatically but can be somewhat of a pain to new software trying to use the library.
It also has been suggested many times most of these errors can be fixed using an automated process.
So I did a quick hack in the LDCad 1.4 code to make it possible to save corrected .dat files back to disk, this resulted in about 116 corrected .dat files.
Only downside I noticed is the loss of some indenting (only for the corrected lines, as LDCad preserves unmodified line content).
I also noticed some minor mixed dos/unix line endings which are indirectly auto corrected for any file that needed rewriting.
This test includes only 'hour glass quad' corrections, but with some additional tweaks I can also auto correct:
case mismatches
non flat triangles
quad with duplicate point (degrade to triangle)
all zero row or col ref matrix
type 5 line with missing or duplicate control points (degrade to normal line)
Should I continue my tweaks, and compile a patch containing all changed files so someone else can double check and merge them into the part tracker?
If so which corrections are actually wanted, and or which additional correction would be needed/wanted.
ps: this topic might be better off in the part authoring forum, not really sure though, admins feel free to move it.
Re: Auto correcting (some) defects in the official library - Chris Dee - 2014-11-15
Yes this would be great. Fixes to official parts need to be sent to [email protected]. They need to go through the Parts Tracker, where the changes can be reviewed.
Of your suggested additional tweaks, I'm not sure about "type 5 line with missing or duplicate control points (degrade to normal line)"
Re: Auto correcting (some) defects in the official library - Philippe Hurbain - 2014-11-16
Quote:non flat trianglesI guess you mean quads? ;D
-> gives 2 triangle + 1 condline?
Re: Auto correcting (some) defects in the official library - Michael Heidemann - 2014-11-16
I should probably have a look into DATHeader body autocorrection procedure. I am sure there are some more of this stuff.
Please remember these are only the body corrections!
1) RGB value to uppercase corrected.
2) ~Moved to file references corrected.
3) 0 WRITE corrected.
4) 0 ROTATION deleted.
5) 0 BFC CERTIFY INVERTNEXT corrected.
6) 0 BFC INVERTNEXT not followed by linetype 1 corrected.
7) Unnecessary '0 BFC INVERTNEXT' corrected.
8) Flat subfile scaled in flat direction corrected
9) 0 COLOR deleted.
10) Double lines are deleted.
11) Bow-tie quads corrected. Winding?
12) Warped quads splitted into triangles.
13) Concave quads splitted into triangles.
14) Use of only '0' for comments corrected.
15) Matrix all zero – corrected.
16) Identical vertices - tried to correct. Please check again.
17) Incorrect color for linetype - corrected
If you need more details for the points, please just ask. I think most of them are self explaining.
Re: Auto correcting (some) defects in the official library - Philippe Hurbain - 2014-11-16
18) Color code in LDConfig (but true color are valid)
Re: Auto correcting (some) defects in the official library - Magnus Forsberg - 2014-11-16
Maybe without understanding all the details here, IMO we don't need another tool to fix all these issues.
We need more human hands and eyes that are willing to start working on all the problem files.
Everybody seams to think these issues is going to be fixed programatically. I doubt it.
Tools are a good help, but if no one starts working with them, the errors will always be there.
What errors could LDCad fix, except those that DatHeader already fixes?
Re: Auto correcting (some) defects in the official library - Michael Heidemann - 2014-11-16
"We need more human hands and eyes that are willing to start working on all the problem files." - That's why i never coded DATHeader for automatic process. There is always a human eye needed
.
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-16
Quote:What errors could LDCad fix, except those that DatHeader already fixes?Looking at the list Michael gave none
Did not realize datheader also fixed mesh problems (given it's name).Only difference is the batch approach as I can load the entire official LDraw library and write all corrected files back to disk afterwards in one go.
Quote:IMO we don't need another tool to fix all these issues.I'm not creating a new tool for this, LDCad does these fixes during loading anyway I just tweaked some things to let it save all those modified files.
But you are both right it's only useful if anyone is willing to double check the resulting changed files (probably about 200 or so).
Although I'm wondering do things like bow ties really have to go though the full review stage? Isn't it easier to just look at the differences text wise combined with a quick LDView checkup.
It's just to get rid of some of the more annoying errors not to get those parts up to modern standards etc (as most of them also have bigger authoring problems like missing invalid/bfc less then perfect conditional lines etc).
Anyway I'll finish my tweaks and leave the result for anyone willing (or not) to pick it up.
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-16
Philippe Hurbain Wrote:I guess you mean quads? ;DYes, although it doesn't add the conline at the moment as it's only a fallback.
-> gives 2 triangle + 1 condline?
What's a normal minimal angle for splitting a quad anyway, I noticed for example 30378 has very weird quads (bow tie and non flat). Jet it seems LDView keeps them as quads anyway.
nose.png (Size: 9.97 KB / Downloads: 8)
This (newer improved) version of my quad fixer uses ~18 deg (max 0.05 dotprod diff), but I'm wondering if it should be even lower.
Re: Auto correcting (some) defects in the official library - Philippe Hurbain - 2014-11-16
Yes it should. According to spec, 3° is the maximum non-coplanarity.
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-16
Ok I finished the changes, but it generated a 1000 files
This seems to be mostly because of case mismatches though, I could leave those out if those are the only corrections in a file.
It also made me wonder about all the fuss about the auto corrections during loading as it took a release version of LDCad about 90 seconds to load and correct all 8193 parts (recursively) in the parts folder of the 1401 lib.
This taking into account loading 8000 parts (taking just below 1GB of memory) will probably never happen in a normal drawing session.
Anyone still interested in processing those files by hand?
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-16
Quote:I'm not sure about "type 5 line with missing or duplicate control points (degrade to normal line)"It really happens though, take for example this one:
Re: Auto correcting (some) defects in the official library - Magnus Forsberg - 2014-11-16
Could you give us some example files?
Before and after fix?
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-16
This might be easier to get the idea.
One side note, when I feed the corrected library again there will still be a couple of problems but loading the entiry library only generated about 65 warnings/errors, these are things like quads with multiple duplicate points, triangle with dup pnt, line with dup pnt, Those kinds of problems are not fixed and therefore currently also not saved.
I could do something like adding a 0 !INVALID in front of them, although it's probably easier for the reviewers to just follow the notes in the log file.
relevant log portion:
Code:
00033 | 2014-11-16_23:11:15 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2507p01.dat | line 41 | CLIP OFF not supported, culling disabled for whole part.
00034 | 2014-11-16_23:11:19 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2677.dat | line 321 | Duplicate point detected, line will not be used.
00035 | 2014-11-16_23:11:19 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\2677.dat | line 500 | Duplicate point detected, line will not be used.
00036 | 2014-11-16_23:11:40 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\u9064.dat | line 27 | CLIP OFF not supported, culling disabled for whole part.
00037 | 2014-11-16_23:11:41 | Warning | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4116604.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00038 | 2014-11-16_23:11:50 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 210 | Multiple duplicate points detected, quad will not be used.
00039 | 2014-11-16_23:11:50 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 214 | Multiple duplicate points detected, quad will not be used.
00040 | 2014-11-16_23:11:50 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 219 | Multiple duplicate points detected, quad will not be used.
00041 | 2014-11-16_23:11:50 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\4629.dat | line 223 | Multiple duplicate points detected, quad will not be used.
00042 | 2014-11-16_23:11:55 | Warning | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\54869.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00043 | 2014-11-16_23:11:57 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\57895p01.dat | line 17 | CLIP OFF not supported, culling disabled for whole part.
00044 | 2014-11-16_23:11:57 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\s\59228s03.dat | line 1303 | Duplicate point detected, line will not be used.
00045 | 2014-11-16_23:12:00 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6156.dat | line 101 | Duplicate point detected, triangle will not be used.
00046 | 2014-11-16_23:12:00 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6156.dat | line 117 | Duplicate point detected, line will not be used.
00047 | 2014-11-16_23:12:00 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6159.dat | line 70 | Multiple duplicate points detected, quad will not be used.
00048 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 688 | Duplicate point detected, line will not be used.
00049 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 689 | Duplicate point detected, line will not be used.
00050 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 690 | Duplicate point detected, line will not be used.
00051 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 691 | Duplicate point detected, line will not be used.
00052 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 692 | Duplicate point detected, line will not be used.
00053 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 693 | Duplicate point detected, line will not be used.
00054 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 694 | Duplicate point detected, line will not be used.
00055 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 695 | Duplicate point detected, line will not be used.
00056 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 696 | Duplicate point detected, line will not be used.
00057 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 697 | Duplicate point detected, line will not be used.
00058 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 698 | Duplicate point detected, line will not be used.
00059 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 699 | Duplicate point detected, line will not be used.
00060 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 701 | Multiple duplicate points detected, quad will not be used.
00061 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 702 | Multiple duplicate points detected, quad will not be used.
00062 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 703 | Multiple duplicate points detected, quad will not be used.
00063 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 704 | Multiple duplicate points detected, quad will not be used.
00064 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 705 | Multiple duplicate points detected, quad will not be used.
00065 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 706 | Multiple duplicate points detected, quad will not be used.
00066 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 707 | Multiple duplicate points detected, quad will not be used.
00067 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 708 | Multiple duplicate points detected, quad will not be used.
00068 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 709 | Multiple duplicate points detected, quad will not be used.
00069 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 710 | Multiple duplicate points detected, quad will not be used.
00070 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 711 | Multiple duplicate points detected, quad will not be used.
00071 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 712 | Multiple duplicate points detected, quad will not be used.
00072 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 713 | Multiple duplicate points detected, quad will not be used.
00073 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 714 | Multiple duplicate points detected, quad will not be used.
00074 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 715 | Multiple duplicate points detected, quad will not be used.
00075 | 2014-11-16_23:12:01 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6252.dat | line 716 | Multiple duplicate points detected, quad will not be used.
00076 | 2014-11-16_23:12:02 | Warning | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\62930.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00077 | 2014-11-16_23:12:04 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6551.dat | line 868 | Duplicate point detected, line will not be used.
00078 | 2014-11-16_23:12:04 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\6551.dat | line 879 | Duplicate point detected, line will not be used.
00079 | 2014-11-16_23:12:05 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\70022.dat | line 78 | CLIP OFF not supported, culling disabled for whole part.
00080 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 243 | Duplicate point detected, line will not be used.
00081 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 245 | Duplicate point detected, line will not be used.
00082 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 259 | Duplicate point detected, line will not be used.
00083 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 261 | Duplicate point detected, line will not be used.
00084 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 275 | Duplicate point detected, line will not be used.
00085 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 277 | Duplicate point detected, line will not be used.
00086 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 291 | Duplicate point detected, line will not be used.
00087 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 293 | Duplicate point detected, line will not be used.
00088 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 307 | Duplicate point detected, line will not be used.
00089 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 309 | Duplicate point detected, line will not be used.
00090 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 323 | Duplicate point detected, line will not be used.
00091 | 2014-11-16_23:12:09 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\877.dat | line 325 | Duplicate point detected, line will not be used.
00092 | 2014-11-16_23:12:12 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\973p60.dat | line 81 | Duplicate point detected, triangle will not be used.
00093 | 2014-11-16_23:12:12 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\973p60.dat | line 82 | Duplicate point detected, triangle will not be used.
00094 | 2014-11-16_23:12:13 | Error | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\99148p01.dat | line 22 | CLIP OFF not supported, culling disabled for whole part.
00095 | 2014-11-16_23:12:14 | Warning | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\979.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.
00096 | 2014-11-16_23:12:14 | Warning | LDraw file load | C:\Users\Roland\Documents\LDraw\parts\980.dat | Description prefix marks subfile as a physical color part, but no such information was given in a !LDRAW_ORG meta.Re: Auto correcting (some) defects in the official library - Philippe Hurbain - 2014-11-17
Quote:Anyone still interested in processing those files by hand?By hand, no, but I would appreciate if all parts with "obvious" errors were automatically corrected!
- "This seems to be mostly because of case mismatches though," What do you mean, reference to eg. 4-4DISC.DAT instead of 4-4disc.dat? If so, that's an excellent exemple of thing that can be automatically corrected without further ado.
- "CLIP OFF not supported, culling disabled for whole part." NOCLIP is presently in BFC specification, does this means that we should change spec? This has been used for transparent patterned parts here, where we should see back face of pattern through part. Latest LDView beta does work fine now even without NOCLIP, but LDCad do not show patterns on back face (except for those that contain NOCLIP!).
Re: Auto correcting (some) defects in the official library - Nicola - 2014-11-17
To me, autofixing some hundreds files already looks like a great improvement.
Roland Melkert Wrote:Although I'm wondering do things like bow ties really have to go though the full review stage? Isn't it easier to just look at the differences text wise combined with a quick LDView checkup.
I agree. These are things routinely done at runtime, i'm pretty sure the software is tried and tested. And even in the worst case, it's not like we cando much damage, since all this parts are already broken to begin with.
Re: Auto correcting (some) defects in the official library - Philippe Hurbain - 2014-11-17
+1 !!!
Go for an automatic cleanup as much as possible!
Re: Auto correcting (some) defects in the official library - Merlijn Wissink - 2014-11-17
I agree 100% with this.
Instead of talking for days or weeks about fixing broken parts, let's just fix them.
Re: Auto correcting (some) defects in the official library - Michael Heidemann - 2014-11-17
How should a corrected "bowtie quad" be controlled to have the correct winding???
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-17
Quote:NOCLIP is presently in BFC specification, does this means that we should change spec?No, those errors are the result of the bfc implementation (clipping on/off at part level) I choose for LDCad it's not a line formatting error. The above log chunk are all >=warning messages while reading the entire (corrected) library, most of them are unrecoverable malformed lines though.
Quote:By hand, no, but I would appreciate if all parts with "obvious" errors were automatically corrected!I did also a test leaving out the case mismatch only files which resulted in ~300 files, which might be a more manageable chunk.
I'll compile zip files with the clanged files (one with the cap change only files, and one with the rest) later tonight so people can take a look them selves.
Re: Auto correcting (some) defects in the official library - Roland Melkert - 2014-11-17
Michael Heidemann Wrote:How should a corrected "bowtie quad" be controlled to have the correct winding???A part with proper BFC wouldn't have bow ties, so any bow tie corrected file will be ether non bfc or have a false bfc status to begin with.
Resulting patch (was Re: Auto correcting..) - Roland Melkert - 2014-11-17
I've ran the last version of my tweaks on a clean 1401 library (re downloaded/installed complete.zip to be sure) and put all changed files in a
7MB 7zip file
The archive contains two lib root folders, one containing files whom only needed cap corrections and one with all the other auto fixed stuff.
I've also included the LDCad logfiles of the batch run for them both so you can check what was changed in detail, although I recommend using something like winmerge or svn to detect / inspect the changes.
Finally there is also a log file of a batch run on the already corrected library which shows the 'fatal' errors which remain in a couple of dozen files.
I'm not in anyway suggesting these files should be bluntly put into the part tracker as they might need human intervention adjustments in some cases.
Anyway here you go
Re: Resulting patch (was Re: Auto correcting..) - Chris Dee - 2014-11-17
Thanks for undertaking this work.
To ease the processing of these files through the Parts Tracker, would it be possible to update the !LDRAW_ORG line to Unofficial and add a new !HISTORY line? Otherwise I need to admin edit every file.
If the basis for this was the latest official library, how do you propose merging your corrections with changes in files already on the parts tracker. I've not counted, but there is at least one case of this (2599.dat).
Re: Resulting patch (was Re: Auto correcting..) - Roland Melkert - 2014-11-17
You could probably use a simple search and replace for the !LDRAW_ORG line (I like using grepwin for such things)
But the history lines might be a problem. Current version of LDCad does not really manage those lines, I could add a simple 'insert after last one (no matter where it might be)' function though.
Or maybe someone here could demonstrate some regular expression search and replace skills in order to add those lines
As for files already on the trackar, I would ignore those as datheader can do (and probably already has done) the same corrections on those files as part of their review process??.
Re: Resulting patch (was Re: Auto correcting..) - Magnus Forsberg - 2014-11-17
No, Please don't do that.
Updated part now present at the PT should not be overwritten.
We've spent hour correcting them !
Re: Resulting patch (was Re: Auto correcting..) - Chris Dee - 2014-11-18
I re-iterate, it is great that you are taking this on. Many people will appreciate a cleaner library, but please don't under-estimate the effort involved.
Roland Melkert Wrote:You could probably use a simple search and replace for the !LDRAW_ORG line (I like using grepwin for such things)Yes, there are many ways that I could do this, but if you want to help get these submitted to the Parts Tracker, I think you should make as much effort as possible to fit in with the change control process.
The current official LDRAW_ORG line should be updated something like:
0 !LDRAW_ORG Part UPDATE 2003-03
to
0 !LDRAW_ORG Unofficial_Part
Roland Melkert Wrote:But the history lines might be a problem. Current version of LDCad does not really manage those lines, I could add a simple 'insert after last one (no matter where it might be)' function though.They will only be in the header so inserting something like this after the last one would be fine.
0 !HISTORY 2014-11-18 [roland] Auto-corrected with LDCad Vxx.x
The parts release script uses the following algorithm (but it's written in Perl, so I guess not much use for you).
- Read and re-write header to "0 !LICENSE" license line (in fact in official library files this should always be exactly five lines)
- Read and re-write rest of header until one of the following REGEXs is satisfied:
Code:
/^\s*[1-5]\s/
/^\s*0\s*BFC\s*INVERTNEXT\s*/
/^\s*0\s*\/\/\s*/You will also need to request Part Author rights for me to be able to proxy-submit these files under your name, viz:
Part Tracker Wrote:... send an email to the Parts Library Admin (Chris Dee) with your true first and last names and your LDraw.org username. You must also include the statement "I accept the LDraw.org Contributor Agreement with regards to all past and future contributions I make to LDraw.org".
If possible please separate files which already exist as updated versions on the Parts Tracker - these will need special treatment to avoid losing any changes (which may be nothing to do with the corrections you are making).
Re: Auto correcting (some) defects in the official library - Steffen - 2014-11-18
let's leave out the uppercase lowercase changes for now
the remaining 300 files can be manually reviewed, that's doable.
the uppercase/lowercase change should happen as a separate batch, for example
done by Chris with the next parts release IMHO
Re: Resulting patch (was Re: Auto correcting..) - Roland Melkert - 2014-11-18
Initially I only intended to process these files to help kick start the cleaning process no need to credit me etc. Hence the request (in the first post) for some one else to do something with them in regards of the pt etc.
But if it helps things get along more quickly I'll happily make those header changes during the export, it will just take me a couple of days to find the time to make the necessary changes to LDCad and process the lib again.
@Magnus: Concerning the files already on the pt, like I wrote before those should indeed be ignored/excluded.
Is there a list of all parts currently on the tracker whom are also part of 1401?, if so I can exclude them in code to make 100% sure they won't cause problems.
Re: Resulting patch (was Re: Auto correcting..) - Nicola - 2014-11-19
Roland Melkert Wrote:I've ran the last version of my tweaks on a clean 1401 library (re downloaded/installed complete.zip to be sure) and put all changed files in a
7MB 7zip file
The archive contains two lib root folders, one containing files whom only needed cap corrections and one with all the other auto fixed stuff.
I've also included the LDCad logfiles of the batch run for them both so you can check what was changed in detail, although I recommend using something like winmerge or svn to detect / inspect the changes.
Finally there is also a log file of a batch run on the already corrected library which shows the 'fatal' errors which remain in a couple of dozen files.
I'm not in anyway suggesting these files should be bluntly put into the part tracker as they might need human intervention adjustments in some cases.
Anyway here you go
Awesome work! Thanks!
Supposing we decide to use these parts, what kind of test should we made? A visual inspection on LDView would be enought? We could divide them among us and verify them rather quickly
Re: Resulting patch (was Re: Auto correcting..) - Philippe Hurbain - 2014-11-19
Quote:Is there a list of all parts currently on the tracker whom are also part of 1401?, if so I can exclude them in code to make 100% sure they won't cause problems.I just made one, for each folder of library (attached).
Re: Resulting patch (was Re: Auto correcting..) - Chris Dee - 2014-11-19
Roland Melkert Wrote:Initially I only intended to process these files to help kick start the cleaning process no need to credit me etc. Hence the request (in the first post) for some one else to do something with them in regards of the pt etc.This is your work, so to incorporate that work into the LDraw library you do need to affirm the Contributor Agreement (CA). You don't need to (and for official parts that are not alrready being re-worked, you cannot) submit them to the PT yourself. I can handle that but the new HISTORY lines need to contain your LDraw userID, as a recognition of your affirmation of the CA. Otherwise the files become non-redistributable - and we don't want that.
Re: Resulting patch (was Re: Auto correcting..) - Magnus Forsberg - 2014-11-19
Among the PT Tools there is now a dynamic list of non-bfc parts.
It shows all the current parts that have a missing or 'nocertify' bfc statment.
If I understand it correctly, that list will adapt it's content when a bfc-fixed official or unofficial part is uploded to PT.
Is it time for a recap?
What are the errors you have fixed?
What is the next step?
1.)
Upload all of these files to PT and a normal review process?
2.)
Not recycling them through PT, but sending them to PT admin and a release without review?
3.)
Am I expected to review them as they are?
If I remember it correctly, it started with a 'broken display' in webgl rendering.
Is this all an effort to make our parts look good in those renderers?
I've now had a quick look into some of the parts, and can't help but wonder What has been changed? I see the same flawed part, but maybe software don't get confused. I don't know. I'm not a programmer, only a part author.
The major issue for making a good rendering in webgl is all the files that are not BFC'd.
I can't see that any of the auto fixed parts has become BFC'd. Or have I missed them?
Don't get me wrong. This is an atempt as good as any, but personally I'd rather pick files from that list at PT to work on, and improve them one at a time.
Re: Resulting patch (was Re: Auto correcting..) - Chris Dee - 2014-11-19
They need to go through the Parts Tracker. That is the safest way to get them packaged into a release without broken dependencies.
Re: Resulting patch (was Re: Auto correcting..) - Roland Melkert - 2014-11-19
Magnus Forsberg Wrote:What are the errors you have fixed?These are the things whom are detected and fixed during loading of LDCad and thus end up in these patches.
- "0 BFC CERTIFY INVERTNEXT" -> "0 BFC INVERTNEXT"
- type 1 ref name case mismatch
- all zero matrix in type 1 line -> id matrix
- all zero row or col in type 1 line matrix -> tries to correct by adding 1 or -1 at suitable position.
- type 1 line with color 24 -> color 16
- type 4 line with hourglass sequence, tries to fix by swapping/rotating points around
- type 4 line with duplicate points -> degrades to triangle (but only if the other 3 are unique).
- non flat type 4 line (0.0015 dotprod diff -> ~3deg) -> degrades to two type 3 lines.
- type 5 line using a control point which is equal to one of it's line points -> degrade to type 2 line.
Most made corrections are the case mismatch, the hourglass or non flat quads though. But all these corrections where added to LDCad because it caused problems in one way or another at some point (in the past).
Magnus Forsberg Wrote:I've now had a quick look into some of the parts, and can't help but wonder What has been changed? I see the same flawed part, but maybe software don't get confused. I don't know. I'm not a programmer, only a part author.Some parts might still look weird because they have many more flaws (like e.g. 206b.dat) but at least all/most invalid formatted LDraw lines should be corrected (like all the zero row/col matrix errors in 206b).
Magnus Forsberg Wrote:Don't get me wrong. This is an atempt as good as any, but personally I'd rather pick files from that list at PT to work on, and improve them one at a time.Just use the ones you like and ignore the others, I'm doing these mass corrections only to help not to hinder.
Re: Resulting patch (was Re: Auto correcting..) - Roland Melkert - 2014-11-19
Thanks,
I'll put them in a single file and prevent saving on matches.
Re: Resulting patch (was Re: Auto correcting..) - Steffen - 2014-11-20
additionally, the added history line makes it easier for us parts reviewers to quickly grep for
all those parts affected by that batch change.
Resulting patch v2 (was Re: Auto correcting..) - Roland Melkert - 2014-11-21
Ok, I've made a new archive this time I also changed the headers and excluded the files on Philio's lists.
It resulted in a 2MB 7zip
or a version which preserves the Physical_Colour tags in cap only files (also a 2MB 7zip)
It again holds separate lib trees for files having just cap fixes, and files with the other fixes (also incl caps if they needed any other fix). Also included again are the relevant LDCad logs. Do note the 2nd pass one is now much larger as it still finds errors/problems with the excluded files.
It's a lot smaller as I discovered the previous one also included the unchanged \s files, sorry about that so it's best to completly forget about the previous .7z (I removed it on my site too).
I've also requested a pt account like Chris asked.
And again feel free to only use a subset or just a hand full, The whole point of this exercise is to help clean the library not to hinder or annoy the part authors
.
Re: Resulting patch v2 (was Re: Auto correcting..) - Steffen - 2014-11-21
Thank you very much.
The "LDraw-1401-autoFixExclCapOnly" now only contains a fairly "small" bunch of files:
1 primitive
232 parts
53 subparts
I would find it tolerable to simply put these on the PT now and let them sink through there as usual,
so we can find the remaining errors like this
800.dat (Brick 2 x 4 with Wheels Holder for Car Steering-Gear Axle)
WARNING "800.dat" Line 168: Identical to line 165: 4 16 -36 16 -8 -36 4 -8 -8 4 -8 -8 16 -8
NOTABENE "34b.dat" referenced by "800.dat": Moved to 3110
NOTABENE "34d.dat" referenced by "800.dat": Moved to 3111
NOTABENE "ring2.dat" referenced by "3111.dat": Moved to 4-4ring2
and maybe others.
Re: Resulting patch (was Re: Auto correcting..) - Steffen - 2014-11-21
maybe we should update the quite outdated
http://www.ldraw.org/library/tracker/ref/reviewfaq/
sometimes
)))what I do [usually] is:
- check syntax with l3p (yes, the ancient one
- look at the file header manually (does it have correct part type, license etc.)
- check the part title if it fits consistently into the library
- check the part with DATHeader
- open the file in LDView with turned on coordinate axes to check proper origin and orientation
- visual inspection with LDView to look for missing lines
- visual inspection with LDView, turning BFC red/green/blue coloring on and off to find BFC errors
- visual inspection with LDView, turning on random color mode to find overlaps
- staring at the part some more
- if I find problems, some closer inspection or running additional tools like edger etc.
Would be interesting to hear other people's check routines
Re: Resulting patch (was Re: Auto correcting..) - Santeri Piippo - 2014-11-21
My routines aren't really as sophiscated.. I usually just look at the header as displayed by the parts tracker and inspect the part in LDView with the BFC red/green/blue mode on, check the origin, orientation, check in edges-only mode to find missing edgelines and, for patterns, check in random colors too. DATHeader doesn't work on my linux system at all so I just have to hope someone else scans the part.
My rule of thumb generally is that if the origin/orientation is good and the part doesn't show visual deficits and it's technically correct then it's good enough.
Re: Resulting patch v2 (was Re: Auto correcting..) - Roland Melkert - 2014-11-21
One side note:
I noticed I've forgot about the qualifier (e.g. Physical_Colour) while changing 'part' into 'Unofficial_Part' But I'm not sure it's needed / wanted at all?
If so I'll do another run only difference being using 'unoffical_part Physical_Colour' if it was 'part Physical_Colour" etc.
Re: Resulting patch v2 (was Re: Auto correcting..) - Roland Melkert - 2014-11-21
As it was only a small change I made an alternative archive
This one preserves the Physical_Colour tag in the cap only fix files (there where none in the exclcap branch).
I've also added the Physical_Colour tag to 979.dat and 980.dat manually, it only being two files and all, as they seem to be the only two files missing it in the originals.
Re: Resulting patch v2 (was Re: Auto correcting..) - Chris Dee - 2014-11-21
Yes, the qualifiers 'Physical_Colour' and 'Alias' need to be retained.
Re: Resulting patch v2 (was Re: Auto correcting..) - Philippe Hurbain - 2014-12-01
So guys, what should we do next?
Re: Resulting patch v2 (was Re: Auto correcting..) - Michael Heidemann - 2014-12-01
I think Chris answered this question already in the past. All files need to be uploaded to the PT for certifying process.
Re: Resulting patch v2 (was Re: Auto correcting..) - Philippe Hurbain - 2014-12-01
Yes, but who does this, Chris?
Re: Resulting patch v2 (was Re: Auto correcting..) - Magnus Forsberg - 2014-12-01
I can only interpret what has been said earlier that it is up to Roland to send these files to Chris, but only if they are supplemented with the missing information required to be possible to upload them to the PT
Re: Resulting patch v2 (was Re: Auto correcting..) - Roland Melkert - 2014-12-01
I assumed Chris would batch 'proxy' upload the ones he liked,
Or at least that's what I understood from this message
Re: Resulting patch v2 (was Re: Auto correcting..) - Chris Dee - 2014-12-01
Yes - I will proxy submit these. At present there is no batch script to do that, and I have been away all weekend without server-side access.
Re: Resulting patch v2 (was Re: Auto correcting..) - Chris Dee - 2014-12-23
Almost all of the files in the 'LDraw-1401-autoFixExclCapOnly' archive have now been submitted to the Parts Tracker.
The following have appeared on the Parts Tracker since Roland made this file, so have not been over-written with the auto-fixed versions:
s/4492b1.dat
s/4492b2.dat
s/4492e3.dat
s/4492m.dat
s/4492s.dat
s/4492t.dat
973p02.dat
2397.dat
2571.dat
3846p4e.dat
4491a.dat
4491b.dat
4493.dat
4494.dat
4587.dat
Many thanks to Roland for doing this. Hopefully they can get certified reasonably quickly.
Since this auto-fix file was built against 2014-01, I wanted to make sure these were submitted before the 2014-02 official part update confused matters further. Hopefully I can get 2014-02 out before the end of the year.