[PATCH] Bogus chapter at the end of title

[macrule]

I encountered some DVDs which had one chapter too many for the main title (Hot Fuzz, Region 2 being only one of them). It looked somewhat like this:

Code: Select all

scan: title 1 has 29 chapters
scan: chap 1 c=0->0, b=0->56984 (56985), 139247 ms
scan: chap 2 c=1->1, b=56985->125385 (68401), 169598 ms
scan: chap 3 c=2->2, b=125386->168825 (43440), 107244 ms
scan: chap 4 c=3->3, b=168826->282218 (113393), 282580 ms
scan: chap 5 c=4->4, b=282219->327410 (45192), 112441 ms
scan: chap 6 c=5->5, b=327411->501946 (174536), 433593 ms
scan: chap 7 c=6->6, b=501947->598822 (96876), 238601 ms
scan: chap 8 c=7->7, b=598823->667773 (68951), 171430 ms
scan: chap 9 c=8->8, b=667774->746028 (78255), 193630 ms
scan: chap 10 c=9->9, b=746029->822960 (76932), 191297 ms
scan: chap 11 c=10->10, b=822961->907836 (84876), 211324 ms
scan: chap 12 c=11->11, b=907837->979007 (71171), 176447 ms
scan: chap 13 c=12->12, b=979008->1042601 (63594), 157532 ms
scan: chap 14 c=13->13, b=1042602->1146960 (104359), 259599 ms
scan: chap 15 c=14->14, b=1146961->1274918 (127958), 318589 ms
scan: chap 16 c=15->15, b=1274919->1432693 (157775), 389664 ms
scan: chap 17 c=16->16, b=1432694->1610400 (177707), 440773 ms
scan: chap 18 c=17->17, b=1610401->1785579 (175179), 435876 ms
scan: chap 19 c=18->19, b=1785580->1837272 (51693), 129634 ms
scan: chap 20 c=20->20, b=1837273->2054843 (217571), 561005 ms
scan: chap 21 c=21->21, b=2054844->2112722 (57879), 154377 ms
scan: chap 22 c=22->22, b=2112723->2326333 (213611), 550981 ms
scan: chap 23 c=23->23, b=2326334->2422556 (96223), 245540 ms
scan: chap 24 c=24->24, b=2422557->2540602 (118046), 305431 ms
scan: chap 25 c=25->25, b=2540603->2567548 (26946), 69303 ms
scan: chap 26 c=26->26, b=2567549->2609527 (41979), 108145 ms
scan: chap 27 c=27->27, b=2609528->2639968 (30441), 77464 ms
scan: chap 28 c=28->28, b=2639969->2747133 (107165), 318639 ms
scan: chap 29 c=29->29, b=0->56984 (56985), 1001 ms

Note that chapter 29 references the same blocks as chapter 1, but with a different duration. I tried the following patch which recognizes if the last chapter references the same start block as the first chapter, and then removes the last chapter from the list.

For me the following patch works, but I think someone who actually understands how this stuff works should look at it. Also I'm not sure how memory management in HB works, so the simple removal of the chapter may leak memory.

Code: Select all

Index: libhb/dvd.c
===================================================================
--- libhb/dvd.c	(revision 2320)
+++ libhb/dvd.c	(working copy)
@@ -578,6 +578,18 @@
         c++;
     }
 
+    /* Sometimes the last chapter is really a fake reference to the first chapter.
+       We check for that and delete such a chapter. */
+    if (hb_list_count( title->list_chapter ) > 1)
+    {
+        hb_chapter_t * first_chapter = hb_list_item( title->list_chapter, 0 );
+        hb_chapter_t * last_chapter  = hb_list_item( title->list_chapter, hb_list_count( title->list_chapter ) - 1 );
+    	if (first_chapter->block_start == last_chapter->block_start) {
+    	    /* The last chapter would repeat the first chapter. That would be wrong. */
+    	    hb_list_rem( title->list_chapter, last_chapter );
+    	}
+    }
+    
     /* The durations we get for chapters aren't precise. Scale them so
        the total matches the title duration */
     duration = 0;

[kvaruni]

I have a DVD with the same problem. I will report on whether the rip succeeds or not (that will be in about 3 hours). By looking at your code this seems like a clean patch, but it is probably best if someone with more in-depth knowledge of HB takes a look at it, as you suggested. I do wonder if such a loop could only occur as the last chapter. I have another DVD that keeps playing the first chapter. However, skipping forward moves you to the actual next chapters, so this problem may be unrelated.

edit: The rip has completed and the first chapter is still there, as the final chapter. I will look into this problem a bit more to see where I have gone wrong or why my rip is different. I will keep you posted.

[macrule]

I found another DVD which showed the same problem, but the patch did not catch it. In this case the bogus chapter started before the first chapter as shown below:

Code: Select all

[20:02:45] scan: chap 1 c=0->0, b=2268693->2852577 (583885), 1454454 ms
[20:02:45] scan: chap 2 c=1->1, b=2852578->3108347 (255770), 644199 ms
[20:02:45] scan: chap 3 c=2->2, b=3108348->3479311 (370964), 916254 ms
[20:02:45] scan: chap 4 c=3->3, b=3479312->3956086 (476775), 1198230 ms
[20:02:45] scan: chap 5 c=4->4, b=0->582639 (582640), 1000 ms

I altered the patch like this:

Code: Select all

Index: libhb/dvd.c
===================================================================
--- libhb/dvd.c   (revision 2320)
+++ libhb/dvd.c   (working copy)
@@ -578,6 +578,18 @@
         c++;
     }

+    /* Sometimes the last chapter is really a fake reference to a bogus chapter.
+       We check for that and delete such a chapter. */
+    if (hb_list_count( title->list_chapter ) > 1)
+    {
+        hb_chapter_t * first_chapter = hb_list_item( title->list_chapter, 0 );
+        hb_chapter_t * last_chapter  = hb_list_item( title->list_chapter, hb_list_count( title->list_chapter ) - 1 );
+       if (first_chapter->block_start <= last_chapter->block_start) {
+           /* The last chapter starts before the first chapter, or is identical to it. That would be wrong. */
+           hb_list_rem( title->list_chapter, last_chapter );
+       }
+    }
+    
     /* The durations we get for chapters aren't precise. Scale them so
        the total matches the title duration */
     duration = 0;

While this works for encoding, some of the preview pictures are still wrong. The last preview picture I got was from the next title on the DVD. Perhaps someone knows offhand what needs to be altered to correct this.

[kvaruni]

My apologies for this late reply. The DVD I have has a similar problem as your last post:

Code: Select all

[14:46:58] scan: chap 1 c=0->45, b=179->74451 (58152), 171394 ms
[14:46:58] scan: chap 2 c=46->46, b=74452->145442 (70991), 180217 ms
[14:46:58] scan: chap 3 c=47->47, b=145443->189782 (44340), 192389 ms
[14:46:58] scan: chap 4 c=48->48, b=189783->294680 (104898), 374276 ms
[14:46:58] scan: chap 5 c=49->49, b=294681->326082 (31402), 107935 ms
[14:46:58] scan: chap 6 c=50->50, b=326083->357326 (31244), 107814 ms
[14:46:58] scan: chap 7 c=51->51, b=357327->399436 (42110), 163094 ms
[14:46:58] scan: chap 8 c=52->52, b=399437->413149 (13713), 36377 ms
[14:46:58] scan: chap 9 c=53->53, b=413150->467149 (54000), 197228 ms
[14:46:58] scan: chap 10 c=54->54, b=467150->507619 (40470), 141123 ms
[14:46:58] scan: chap 11 c=55->55, b=507620->570508 (62889), 230668 ms
[14:46:58] scan: chap 12 c=56->56, b=570509->643700 (73192), 280043 ms
[14:46:58] scan: chap 13 c=57->57, b=643701->728762 (85062), 287759 ms
[14:46:58] scan: chap 14 c=58->58, b=728763->759294 (30532), 97673 ms
[14:46:58] scan: chap 15 c=59->59, b=759295->847652 (88358), 305073 ms
[14:46:58] scan: chap 16 c=60->60, b=847653->885153 (37501), 129131 ms
[14:46:58] scan: chap 17 c=61->61, b=885154->948738 (63585), 234400 ms
[14:46:58] scan: chap 18 c=62->62, b=948739->1015962 (67224), 231634 ms
[14:46:58] scan: chap 19 c=63->63, b=1015963->1091453 (75491), 241603 ms
[14:46:58] scan: chap 20 c=64->64, b=1091454->1152682 (61229), 201303 ms
[14:46:58] scan: chap 21 c=65->65, b=1152683->1220172 (67490), 250617 ms
[14:46:58] scan: chap 22 c=66->66, b=1220173->1248676 (28504), 100882 ms
[14:46:58] scan: chap 23 c=67->68, b=1248677->1297871 (49195), 158205 ms
[14:46:58] scan: chap 24 c=69->69, b=1297872->1361840 (63969), 187410 ms
[14:46:58] scan: chap 25 c=70->70, b=1361841->1467904 (106064), 343060 ms
[14:46:58] scan: chap 26 c=71->71, b=1467905->1485468 (17564), 44355 ms
[14:46:58] scan: chap 27 c=72->72, b=1485469->1519385 (33917), 100802 ms
[14:46:58] scan: chap 28 c=73->73, b=1519386->1563747 (44362), 262629 ms
[14:46:58] scan: chap 29 c=74->77, b=16913->186 (57563), 90 ms

I had written up a patch very similar to yours, but I have improved on it. Perhaps it is best that we run over all the issues that we may face and that are undesirable? I can think of the following list:

• We do not want chapters that end earlier than they begin (as in my example)
• We do not want chapters that start before the end time of the last chapter. (as in both your examples)

The following patch works for me:

Code: Select all

*** dvd-orig.c	Fri Apr 17 16:10:32 2009
--- dvd.c	Sun Apr 19 09:15:13 2009
***************
*** 578,583 ****
--- 578,608 ----
          c++;
      }
  
+ 	/* Sometimes the last chapter is really a fake reference to a bogus chapter.
+ 	   We check for that and delete such a chapter. */
+ 	if (hb_list_count( title->list_chapter ) > 1)
+ 	{
+ 		/* First, we need to ensure that each chapter starts before it ends. */
+ 		for ( i = hb_list_count( title->list_chapter ) - 1; i >= 0; i-- )
+ 		{
+ 			hb_chapter_t * current_chapter = hb_list_item( title->list_chapter, i );
+ 			if (current_chapter->block_start >= current_chapter->block_end) 
+ 			{
+ 				hb_list_rem( title->list_chapter, current_chapter );
+ 			}
+ 		}
+ 		/* Then, we need to verify that each chapter starts no sooner than the previous chapter ends. */
+ 		for ( i = hb_list_count( title->list_chapter ) - 1; i > 0; i-- )
+ 		{
+ 			hb_chapter_t * previous_chapter = hb_list_item( title->list_chapter, i - 1 );
+ 			hb_chapter_t * current_chapter = hb_list_item( title->list_chapter, i );
+ 			if (current_chapter->block_start < previous_chapter->block_end) 
+ 			{
+ 				hb_list_rem( title->list_chapter, current_chapter );
+ 			}
+ 		}
+ 	}
+ 
      /* The durations we get for chapters aren't precise. Scale them so
         the total matches the title duration */
      duration = 0;

However, I do not believe that it solves the Preview issue. Can you verify that this patch works otherwise?

(As a side note, what am I doing wrong with my patches? The header appears to be different from all other patches and I cannot readily find the answer in the stickies or wiki.)

[kvaruni]

Ok, so the patch is not really waterproof: when encoding Pursuit of Happyness it dropped the last chapter, which would seem obvious given the information below:

Code: Select all

[11:22:54] scan: chap 1 c=0->32, b=0->116937 (92608), 295995 ms
[11:22:54] scan: chap 2 c=33->33, b=116938->214170 (97233), 242527 ms
[11:22:54] scan: chap 3 c=34->34, b=214171->283081 (68911), 186315 ms
[11:22:54] scan: chap 4 c=35->35, b=283082->428156 (145075), 366680 ms
[11:22:54] scan: chap 5 c=36->36, b=428157->511825 (83669), 199406 ms
[11:22:54] scan: chap 6 c=37->37, b=511826->555129 (43304), 111422 ms
[11:22:54] scan: chap 7 c=38->38, b=555130->638109 (82980), 219359 ms
[11:22:54] scan: chap 8 c=39->39, b=638110->767927 (129818), 310879 ms
[11:22:54] scan: chap 9 c=40->40, b=767928->871905 (103978), 270683 ms
[11:22:54] scan: chap 10 c=41->41, b=871906->944038 (72133), 175407 ms
[11:22:54] scan: chap 11 c=42->42, b=944039->1036664 (92626), 258463 ms
[11:22:54] scan: chap 12 c=43->43, b=1036665->1087591 (50927), 142313 ms
[11:22:54] scan: chap 13 c=44->44, b=1087592->1177553 (89962), 258543 ms
[11:22:54] scan: chap 14 c=45->45, b=1177554->1242991 (65438), 177570 ms
[11:22:54] scan: chap 15 c=46->46, b=1242992->1281845 (38854), 98160 ms
[11:22:54] scan: chap 16 c=47->47, b=1281846->1395380 (113535), 273508 ms
[11:22:54] scan: chap 17 c=48->48, b=1395381->1477002 (81622), 212548 ms
[11:22:54] scan: chap 18 c=49->49, b=1477003->1553200 (76198), 193527 ms
[11:22:54] scan: chap 19 c=50->50, b=1553201->1687554 (134354), 316879 ms
[11:22:54] scan: chap 20 c=51->51, b=1687555->1749751 (62197), 171380 ms
[11:22:54] scan: chap 21 c=52->52, b=1749752->1824741 (74990), 190402 ms
[11:22:54] scan: chap 22 c=53->54, b=1824742->1894604 (69863), 196481 ms
[11:22:54] scan: chap 23 c=55->55, b=1894605->1983585 (88981), 223566 ms
[11:22:54] scan: chap 24 c=56->56, b=1983586->2104323 (120738), 316538 ms
[11:22:54] scan: chap 25 c=57->57, b=2104324->2300557 (196234), 423754 ms
[11:22:54] scan: chap 26 c=58->58, b=2300558->2385952 (85395), 218437 ms
[11:22:54] scan: chap 27 c=59->59, b=2385953->2463991 (78039), 195520 ms
[11:22:54] scan: chap 28 c=60->72, b=2463992->7 (192300), 515745 ms

However, this was a valid chapter and is an actual part of the image. I am not sure why this is a good chapter and why the other encode I did with similar features was in fact a bogus chapter at the end. Nevertheless, stay clear of my patch since it is a bit too eager in removing possibly bogus chapters.

edit: I guess that removing the first loop is all that is needed to resolve the issue.

[macrule]

Your logic made sense to me. Shame that it doesn't work. I noticed something that is weird about the scan you posted. The chapter numbers are not contious. Not sure if this is normal, but I haven't seen it anywhere else yet. Here for example, the first chapter contains "chapter" 0 to 32:

Code: Select all

[11:22:54] scan: chap 1 c=0->32, b=0->116937 (92608), 295995 ms

(also: 116937 - 0 != 92608)

Perhaps the key to success lies in the duration of a chapter. I noticed that all bogus chapters so far were listed as "1000 ms" or less, even if they spanned all the blocks of the DVD. I have no clue where that calculation is done though, and how it is done.

Right now I'm puzzled by another DVD which doesn't show any oddities in the scan list but also repeats the first chapter after the last one.
edit: it seems http://forum.handbrake.fr/viewtopic.php ... aa6#p54726 was the reason because this particular one failed. The chapter patch couldn't work on it. Haven't actually tested encoding again, though.

Btw: I think your diff's just look different because you use the real "diff". I usually work on a soure tree checked out from subversion, and "svn diff filename" or just "svn diff" comes in handy then.

[kvaruni]

I am sorry to say so, but this morning I got a bogus chapter that is ... 1012ms long.

Code: Select all

[07:58:37] scan: chap 1 c=0->0, b=0->10453 (10454), 34595 ms
[07:58:37] scan: chap 2 c=1->1, b=10454->18925 (8472), 24420 ms
[07:58:37] scan: chap 3 c=2->2, b=18926->22330 (3405), 11147 ms
[07:58:37] scan: chap 4 c=3->3, b=22331->30547 (8217), 28673 ms
[07:58:37] scan: chap 5 c=4->4, b=30548->52327 (21780), 60748 ms
[07:58:37] scan: chap 6 c=5->5, b=52328->68436 (16109), 64149 ms
[07:58:37] scan: chap 7 c=6->6, b=68437->73848 (5412), 15247 ms
[07:58:37] scan: chap 8 c=7->7, b=73849->81226 (7378), 21383 ms
[07:58:37] scan: chap 9 c=8->8, b=81227->83515 (2289), 4070 ms
[07:58:37] scan: chap 10 c=9->9, b=83516->96916 (13401), 41885 ms
[07:58:37] scan: chap 11 c=10->10, b=96917->103225 (6309), 14194 ms
[07:58:37] scan: chap 12 c=11->11, b=103226->107998 (4773), 15551 ms
[07:58:37] scan: chap 13 c=12->12, b=107999->114692 (6694), 22446 ms
[07:58:37] scan: chap 14 c=13->13, b=114693->119913 (5221), 18467 ms
[07:58:37] scan: chap 15 c=14->14, b=119914->127223 (7310), 21474 ms
[07:58:37] scan: chap 16 c=15->15, b=127224->138780 (11557), 46654 ms
[07:58:37] scan: chap 17 c=16->16, b=138781->141149 (2369), 6317 ms
[07:58:37] scan: chap 18 c=17->17, b=141150->146873 (5724), 22446 ms
[07:58:37] scan: chap 19 c=18->18, b=146874->149517 (2644), 8352 ms
[07:58:37] scan: chap 20 c=19->19, b=149518->160511 (10994), 36631 ms
[07:58:37] scan: chap 21 c=20->20, b=160512->162811 (2300), 5345 ms
[07:58:37] scan: chap 22 c=21->21, b=162812->166575 (3764), 10205 ms
[07:58:37] scan: chap 23 c=22->22, b=166576->178836 (12261), 27397 ms
[07:58:37] scan: chap 24 c=23->23, b=178837->181413 (2577), 6408 ms
[07:58:37] scan: chap 25 c=24->24, b=181414->183569 (2156), 5436 ms
[07:58:37] scan: chap 26 c=25->25, b=183570->185188 (1619), 4070 ms
[07:58:37] scan: chap 27 c=26->26, b=185189->213180 (27992), 86393 ms
[07:58:37] scan: chap 28 c=27->27, b=0->10453 (10454), 1012 ms

Nevertheless, the second loop of my patch and your original patch quite nicely eliminate this chapter.

BTW, thanks for the info on using svn to create diffs. I had no clue you could do that. It does make it a whole lot easier .

[kvaruni]

Yet another special case: my patch was too eager and delete the last two chapters of the following title. This was not desirable, since both 50 and 51 contained the credits.

Code: Select all

[21:29:08] scan: chap 1 c=0->0, b=0->5519 (5520), 19181 ms
[21:29:08] scan: chap 2 c=1->2, b=5520->70476 (64141), 88317 ms
[21:29:08] scan: chap 3 c=3->5, b=70477->191414 (119778), 347242 ms
[21:29:08] scan: chap 4 c=6->6, b=191415->228476 (37062), 115600 ms
[21:29:08] scan: chap 5 c=7->8, b=228477->257459 (28983), 79222 ms
[21:29:08] scan: chap 6 c=9->9, b=257460->298455 (40996), 120603 ms
[21:29:08] scan: chap 7 c=10->10, b=298456->326353 (27898), 82341 ms
[21:29:08] scan: chap 8 c=11->11, b=326354->367502 (41149), 138491 ms
[21:29:08] scan: chap 9 c=12->12, b=367503->431632 (64130), 198783 ms
[21:29:08] scan: chap 10 c=13->14, b=431633->469971 (38339), 99567 ms
[21:29:08] scan: chap 11 c=15->15, b=469972->534006 (64035), 160761 ms
[21:29:08] scan: chap 12 c=16->17, b=534007->617679 (83673), 249911 ms
[21:29:08] scan: chap 13 c=18->19, b=617680->665689 (48010), 122599 ms
[21:29:08] scan: chap 14 c=20->20, b=665690->746967 (81278), 240646 ms
[21:29:08] scan: chap 15 c=21->22, b=746968->842333 (95366), 285095 ms
[21:29:08] scan: chap 16 c=23->24, b=842334->902605 (60272), 198994 ms
[21:29:08] scan: chap 17 c=25->26, b=902606->1051222 (148617), 449857 ms
[21:29:08] scan: chap 18 c=27->28, b=1051223->1110916 (59694), 171991 ms
[21:29:08] scan: chap 19 c=29->30, b=1110917->1177132 (66216), 195855 ms
[21:29:08] scan: chap 20 c=31->31, b=1177133->1265406 (88274), 229797 ms
[21:29:08] scan: chap 21 c=32->32, b=1265407->1342680 (77274), 177807 ms
[21:29:08] scan: chap 22 c=33->33, b=1342681->1455410 (112730), 262915 ms
[21:29:08] scan: chap 23 c=34->34, b=1455411->1477390 (21980), 68303 ms
[21:29:08] scan: chap 24 c=35->36, b=1477391->1590357 (112967), 291031 ms
[21:29:08] scan: chap 25 c=37->38, b=1590358->1636038 (45681), 102725 ms
[21:29:08] scan: chap 26 c=39->39, b=1636039->1690728 (54690), 132485 ms
[21:29:08] scan: chap 27 c=40->41, b=1690729->1790108 (99380), 222136 ms
[21:29:08] scan: chap 28 c=42->42, b=1790109->1848104 (57996), 140627 ms
[21:29:08] scan: chap 29 c=43->44, b=1848105->1931468 (83364), 181868 ms
[21:29:08] scan: chap 30 c=45->45, b=1931469->1957173 (25705), 64312 ms
[21:29:08] scan: chap 31 c=46->46, b=1957174->2013884 (56711), 121325 ms
[21:29:08] scan: chap 32 c=47->47, b=2013885->2116474 (102590), 242701 ms
[21:29:08] scan: chap 33 c=48->49, b=2116475->2162990 (46516), 99737 ms
[21:29:08] scan: chap 34 c=50->50, b=2162991->2242156 (79166), 198713 ms
[21:29:08] scan: chap 35 c=51->52, b=2242157->2292884 (50728), 122458 ms
[21:29:08] scan: chap 36 c=53->53, b=2292885->2333723 (40839), 93581 ms
[21:29:08] scan: chap 37 c=54->54, b=2333724->2383867 (50144), 104320 ms
[21:29:08] scan: chap 38 c=55->55, b=2383868->2448702 (64835), 131563 ms
[21:29:08] scan: chap 39 c=56->58, b=2448703->2624153 (175451), 367637 ms
[21:29:08] scan: chap 40 c=59->59, b=2624154->2695656 (71503), 139424 ms
[21:29:08] scan: chap 41 c=60->61, b=2695657->2732796 (37140), 74449 ms
[21:29:08] scan: chap 42 c=62->62, b=2732797->2756268 (23472), 48490 ms
[21:29:08] scan: chap 43 c=63->63, b=2756269->2785598 (29330), 59289 ms
[21:29:08] scan: chap 44 c=64->64, b=2785599->2830083 (44485), 89610 ms
[21:29:08] scan: chap 45 c=65->65, b=2830084->2860256 (30173), 74339 ms
[21:29:08] scan: chap 46 c=66->66, b=2860257->2891325 (31069), 68223 ms
[21:29:08] scan: chap 47 c=67->67, b=2891326->2911528 (20203), 54245 ms
[21:29:08] scan: chap 48 c=68->68, b=2911529->2939036 (27508), 63209 ms
[21:29:08] scan: chap 49 c=69->71, b=2939037->2980601 (40927), 80435 ms
[21:29:08] scan: chap 50 c=72->73, b=2979964->3270931 (290595), 369091 ms
[21:29:08] scan: chap 51 c=74->75, b=3270559->3271271 (189), 1133 ms