超參重新調整後，開始運行程序訓練yolov3，開始以爲是調整程序，修改參數導致的錯誤。但在恢復原參數，還是照樣錯誤，通過對錯誤代碼的解讀，是GPU內存不夠導致。但比較奇怪的是對於IBM雲平臺，只能訓練被指定分配的GPU，被其他人的程序佔用，導致報錯。
2019-09-07 17:10:57.137765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fa605800 of size 2048
2019-09-07 17:10:57.137771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fa606000 of size 2048
2019-09-07 17:10:57.137777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fa606800 of size 2048
2019-09-07 17:10:57.137783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fa607000 of size 2048
2019-09-07 17:10:57.137789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fa607800 of size 4718592
2019-09-07 17:10:57.137795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faa87800 of size 4718592
2019-09-07 17:10:57.137801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faf07800 of size 256
2019-09-07 17:10:57.137807: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faf07900 of size 4096
2019-09-07 17:10:57.137813: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faf08900 of size 4096
2019-09-07 17:10:57.137819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faf09900 of size 4096
2019-09-07 17:10:57.137824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84faf0a900 of size 18874368
2019-09-07 17:10:57.137831: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc10a900 of size 2048
2019-09-07 17:10:57.137837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc10b100 of size 2048
2019-09-07 17:10:57.137843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc10b900 of size 2048
2019-09-07 17:10:57.137849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc10c100 of size 2048
2019-09-07 17:10:57.137854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc10c900 of size 2097152
2019-09-07 17:10:57.137860: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc30c900 of size 4096
2019-09-07 17:10:57.137868: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc30d900 of size 4096
2019-09-07 17:10:57.137874: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc30e900 of size 4096
2019-09-07 17:10:57.137882: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc30f900 of size 4096
2019-09-07 17:10:57.137888: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fc310900 of size 18874368
2019-09-07 17:10:57.137894: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd510900 of size 2048
2019-09-07 17:10:57.137900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd511100 of size 2048
2019-09-07 17:10:57.137906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd511900 of size 2048
2019-09-07 17:10:57.137912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd512100 of size 2048
2019-09-07 17:10:57.137918: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd512900 of size 2097152
2019-09-07 17:10:57.137924: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd712900 of size 4096
2019-09-07 17:10:57.137930: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd713900 of size 4096
2019-09-07 17:10:57.137936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd714900 of size 4096
2019-09-07 17:10:57.137942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd715900 of size 4096
2019-09-07 17:10:57.137948: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fd716900 of size 18874368
2019-09-07 17:10:57.137954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fe916900 of size 2048
2019-09-07 17:10:57.137960: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fe917100 of size 2048
2019-09-07 17:10:57.137965: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fe917900 of size 2048
2019-09-07 17:10:57.137971: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fe918100 of size 2048
2019-09-07 17:10:57.137977: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fe918900 of size 2097152
2019-09-07 17:10:57.137983: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84feb18900 of size 4096
2019-09-07 17:10:57.137989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84feb19900 of size 4096
2019-09-07 17:10:57.137994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84feb1a900 of size 4096
2019-09-07 17:10:57.138000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84feb1b900 of size 4096
2019-09-07 17:10:57.138006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84feb1c900 of size 18874368
2019-09-07 17:10:57.138012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd1c900 of size 256
2019-09-07 17:10:57.138018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd1ca00 of size 256
2019-09-07 17:10:57.138024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd1cb00 of size 256
2019-09-07 17:10:57.138030: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd1cc00 of size 256
2019-09-07 17:10:57.138036: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd1cd00 of size 32768
2019-09-07 17:10:57.138042: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd24d00 of size 2048
2019-09-07 17:10:57.138048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd25500 of size 2048
2019-09-07 17:10:57.138054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd25d00 of size 2048
2019-09-07 17:10:57.138060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd26500 of size 2048
2019-09-07 17:10:57.138065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84ffd26d00 of size 2097152
2019-09-07 17:10:57.138071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fff26d00 of size 4096
2019-09-07 17:10:57.138077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fff27d00 of size 4096
2019-09-07 17:10:57.138083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fff28d00 of size 4096
2019-09-07 17:10:57.138088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fff29d00 of size 4096
2019-09-07 17:10:57.138095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f84fff2ad00 of size 18874368
2019-09-07 17:10:57.138101: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850112ad00 of size 512
2019-09-07 17:10:57.138107: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850112af00 of size 512
2019-09-07 17:10:57.138112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850112b100 of size 512
2019-09-07 17:10:57.138119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850112b300 of size 512
2019-09-07 17:10:57.138125: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850112b500 of size 294912
2019-09-07 17:10:57.138130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8501173500 of size 256
2019-09-07 17:10:57.138136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8501173600 of size 256
2019-09-07 17:10:57.138310: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8501173700 of size 256
2019-09-07 17:10:57.138316: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8501173800 of size 256
2019-09-07 17:10:57.138322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8501173900 of size 32768
2019-09-07 17:10:57.138328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850117b900 of size 512
2019-09-07 17:10:57.138334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850117bb00 of size 512
2019-09-07 17:10:57.138340: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850117bd00 of size 512
2019-09-07 17:10:57.138346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850117bf00 of size 512
2019-09-07 17:10:57.138352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850117c100 of size 294912
2019-09-07 17:10:57.138358: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85011c4100 of size 1024
2019-09-07 17:10:57.138364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85011c4500 of size 1024
2019-09-07 17:10:57.138370: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85011c4900 of size 1024
2019-09-07 17:10:57.138376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85011c4d00 of size 1024
2019-09-07 17:10:57.138382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85011c5100 of size 1179648
2019-09-07 17:10:57.138388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e5100 of size 2048
2019-09-07 17:10:57.138394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e5900 of size 2048
2019-09-07 17:10:57.138400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e6100 of size 2048
2019-09-07 17:10:57.138406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e6900 of size 2048
2019-09-07 17:10:57.138411: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e7100 of size 2048
2019-09-07 17:10:57.138417: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e7900 of size 2048
2019-09-07 17:10:57.138425: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85012e8100 of size 2097152
2019-09-07 17:10:57.138432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85014e8100 of size 2097152
2019-09-07 17:10:57.138439: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016e8100 of size 4096
2019-09-07 17:10:57.138445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016e9100 of size 4096
2019-09-07 17:10:57.138451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016ea100 of size 4096
2019-09-07 17:10:57.138457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016eb100 of size 4096
2019-09-07 17:10:57.138463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016ec100 of size 4096
2019-09-07 17:10:57.138469: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016ed100 of size 4096
2019-09-07 17:10:57.138475: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85016ee100 of size 18874368
2019-09-07 17:10:57.138481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85028ee100 of size 18874368
2019-09-07 17:10:57.138491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aee100 of size 1024
2019-09-07 17:10:57.138497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aee500 of size 1024
2019-09-07 17:10:57.138503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aee900 of size 1024
2019-09-07 17:10:57.138509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aeed00 of size 1024
2019-09-07 17:10:57.138515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aef100 of size 1024
2019-09-07 17:10:57.138521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aef500 of size 1024
2019-09-07 17:10:57.138527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503aef900 of size 524288
2019-09-07 17:10:57.138533: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503b6f900 of size 524288
2019-09-07 17:10:57.138539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bef900 of size 2048
2019-09-07 17:10:57.138545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf0100 of size 2048
2019-09-07 17:10:57.138551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf0900 of size 2048
2019-09-07 17:10:57.138557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf1100 of size 2048
2019-09-07 17:10:57.138563: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf1900 of size 2048
2019-09-07 17:10:57.138569: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf2100 of size 2048
2019-09-07 17:10:57.138575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8503bf2900 of size 4718592
2019-09-07 17:10:57.138581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504072900 of size 4718592
2019-09-07 17:10:57.138586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f2900 of size 1024
2019-09-07 17:10:57.138592: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f2d00 of size 1024
2019-09-07 17:10:57.138598: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f3100 of size 1024
2019-09-07 17:10:57.138604: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f3500 of size 1024
2019-09-07 17:10:57.138610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f3900 of size 1024
2019-09-07 17:10:57.138616: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f3d00 of size 1024
2019-09-07 17:10:57.138622: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85044f4100 of size 524288
2019-09-07 17:10:57.138627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504574100 of size 524288
2019-09-07 17:10:57.138633: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f4100 of size 2048
2019-09-07 17:10:57.138639: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f4900 of size 2048
2019-09-07 17:10:57.138645: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f5100 of size 2048
2019-09-07 17:10:57.138651: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f5900 of size 2048
2019-09-07 17:10:57.138657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f6100 of size 2048
2019-09-07 17:10:57.138663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f6900 of size 2048
2019-09-07 17:10:57.138669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85045f7100 of size 4718592
2019-09-07 17:10:57.138675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504a77100 of size 4718592
2019-09-07 17:10:57.138681: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504ef7100 of size 256
2019-09-07 17:10:57.138687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504ef7200 of size 256
2019-09-07 17:10:57.138693: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504ef7300 of size 98304
2019-09-07 17:10:57.138699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f0f300 of size 98304
2019-09-07 17:10:57.138705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27300 of size 512
2019-09-07 17:10:57.138710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27500 of size 512
2019-09-07 17:10:57.138716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27700 of size 512
2019-09-07 17:10:57.138896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27900 of size 512
2019-09-07 17:10:57.138902: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27b00 of size 512
2019-09-07 17:10:57.138908: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27d00 of size 512
2019-09-07 17:10:57.138914: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f27f00 of size 131072
2019-09-07 17:10:57.138921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f47f00 of size 131072
2019-09-07 17:10:57.138926: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f67f00 of size 512
2019-09-07 17:10:57.138932: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68100 of size 512
2019-09-07 17:10:57.138938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68300 of size 512
2019-09-07 17:10:57.138944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68500 of size 512
2019-09-07 17:10:57.138950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68700 of size 512
2019-09-07 17:10:57.138956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68900 of size 512
2019-09-07 17:10:57.138962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68b00 of size 256
2019-09-07 17:10:57.138968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68c00 of size 256
2019-09-07 17:10:57.138974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68d00 of size 256
2019-09-07 17:10:57.138980: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68e00 of size 256
2019-09-07 17:10:57.138986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f68f00 of size 3584
2019-09-07 17:10:57.138992: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f69d00 of size 256
2019-09-07 17:10:57.138998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f69e00 of size 256
2019-09-07 17:10:57.139006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f69f00 of size 256
2019-09-07 17:10:57.139012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f6a000 of size 256
2019-09-07 17:10:57.139020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f6a100 of size 73728
2019-09-07 17:10:57.139026: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f7c100 of size 512
2019-09-07 17:10:57.139032: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f7c300 of size 512
2019-09-07 17:10:57.139038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f7c500 of size 512
2019-09-07 17:10:57.139045: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f7c700 of size 512
2019-09-07 17:10:57.139050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f7c900 of size 131072
2019-09-07 17:10:57.139057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f9c900 of size 1024
2019-09-07 17:10:57.139062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f9cd00 of size 1024
2019-09-07 17:10:57.139068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f9d100 of size 1024
2019-09-07 17:10:57.139074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f9d500 of size 1024
2019-09-07 17:10:57.139080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8504f9d900 of size 1179648
2019-09-07 17:10:57.139086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050bd900 of size 512
2019-09-07 17:10:57.139092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050bdb00 of size 512
2019-09-07 17:10:57.139098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050bdd00 of size 512
2019-09-07 17:10:57.139104: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050bdf00 of size 512
2019-09-07 17:10:57.139109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050be100 of size 131072
2019-09-07 17:10:57.139116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050de100 of size 1024
2019-09-07 17:10:57.139121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050de500 of size 1024
2019-09-07 17:10:57.139127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050de900 of size 1024
2019-09-07 17:10:57.139133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050ded00 of size 1024
2019-09-07 17:10:57.139138: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85050df100 of size 1179648
2019-09-07 17:10:57.139144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85051ff100 of size 512
2019-09-07 17:10:57.139150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85051ff300 of size 512
2019-09-07 17:10:57.139156: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85051ff500 of size 512
2019-09-07 17:10:57.139162: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85051ff700 of size 512
2019-09-07 17:10:57.139168: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85051ff900 of size 131072
2019-09-07 17:10:57.139173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850521f900 of size 1024
2019-09-07 17:10:57.139179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850521fd00 of size 1024
2019-09-07 17:10:57.139185: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505220100 of size 1024
2019-09-07 17:10:57.139191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505220500 of size 1024
2019-09-07 17:10:57.139197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505220900 of size 1179648
2019-09-07 17:10:57.139203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505340900 of size 512
2019-09-07 17:10:57.139209: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505340b00 of size 512
2019-09-07 17:10:57.139215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505340d00 of size 512
2019-09-07 17:10:57.139221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505340f00 of size 512
2019-09-07 17:10:57.139226: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505341100 of size 131072
2019-09-07 17:10:57.139232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505361100 of size 1024
2019-09-07 17:10:57.139238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505361500 of size 1024
2019-09-07 17:10:57.139244: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505361900 of size 1024
2019-09-07 17:10:57.139250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505361d00 of size 1024
2019-09-07 17:10:57.139256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505362100 of size 1179648
2019-09-07 17:10:57.139262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505482100 of size 512
2019-09-07 17:10:57.139268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505482300 of size 512
2019-09-07 17:10:57.139273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505482500 of size 512
2019-09-07 17:10:57.139279: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505482700 of size 512
2019-09-07 17:10:57.139285: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505482900 of size 131072
2019-09-07 17:10:57.139291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85054a2900 of size 1024
2019-09-07 17:10:57.139297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85054a2d00 of size 1024
2019-09-07 17:10:57.139481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85054a3100 of size 1024
2019-09-07 17:10:57.139490: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85054a3500 of size 1024
2019-09-07 17:10:57.139497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85054a3900 of size 1179648
2019-09-07 17:10:57.139503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c3900 of size 256
2019-09-07 17:10:57.139509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c3a00 of size 256
2019-09-07 17:10:57.139515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c3b00 of size 256
2019-09-07 17:10:57.139521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c3c00 of size 256
2019-09-07 17:10:57.139527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c3d00 of size 8192
2019-09-07 17:10:57.139534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c5d00 of size 512
2019-09-07 17:10:57.139539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c5f00 of size 512
2019-09-07 17:10:57.139546: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c6100 of size 512
2019-09-07 17:10:57.139551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c6300 of size 512
2019-09-07 17:10:57.139558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055c6500 of size 131072
2019-09-07 17:10:57.139563: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055e6500 of size 1024
2019-09-07 17:10:57.139570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055e6900 of size 1024
2019-09-07 17:10:57.139575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055e6d00 of size 1024
2019-09-07 17:10:57.139581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055e7100 of size 1024
2019-09-07 17:10:57.139587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85055e7500 of size 1179648
2019-09-07 17:10:57.139593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707500 of size 256
2019-09-07 17:10:57.139599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707600 of size 256
2019-09-07 17:10:57.139606: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707700 of size 256
2019-09-07 17:10:57.139611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707800 of size 256
2019-09-07 17:10:57.139617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707900 of size 256
2019-09-07 17:10:57.139624: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707a00 of size 256
2019-09-07 17:10:57.139631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707b00 of size 256
2019-09-07 17:10:57.139638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707c00 of size 256
2019-09-07 17:10:57.139645: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707d00 of size 256
2019-09-07 17:10:57.139650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707e00 of size 256
2019-09-07 17:10:57.139656: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505707f00 of size 256
2019-09-07 17:10:57.139662: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708000 of size 256
2019-09-07 17:10:57.139669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708100 of size 256
2019-09-07 17:10:57.139675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708200 of size 256
2019-09-07 17:10:57.139681: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708300 of size 256
2019-09-07 17:10:57.139687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708400 of size 256
2019-09-07 17:10:57.139693: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708500 of size 256
2019-09-07 17:10:57.139699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708600 of size 256
2019-09-07 17:10:57.139705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708700 of size 256
2019-09-07 17:10:57.139710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708800 of size 256
2019-09-07 17:10:57.139716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708900 of size 256
2019-09-07 17:10:57.139722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708a00 of size 256
2019-09-07 17:10:57.139728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708b00 of size 256
2019-09-07 17:10:57.139734: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708c00 of size 256
2019-09-07 17:10:57.139740: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708d00 of size 256
2019-09-07 17:10:57.139746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708e00 of size 256
2019-09-07 17:10:57.139752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505708f00 of size 256
2019-09-07 17:10:57.139758: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709000 of size 256
2019-09-07 17:10:57.139764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709100 of size 256
2019-09-07 17:10:57.139770: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709200 of size 256
2019-09-07 17:10:57.139776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709300 of size 256
2019-09-07 17:10:57.139782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709400 of size 256
2019-09-07 17:10:57.139788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709500 of size 256
2019-09-07 17:10:57.139794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709600 of size 256
2019-09-07 17:10:57.139800: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709700 of size 256
2019-09-07 17:10:57.139805: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709800 of size 256
2019-09-07 17:10:57.139811: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709900 of size 256
2019-09-07 17:10:57.139817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709a00 of size 256
2019-09-07 17:10:57.139823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709b00 of size 256
2019-09-07 17:10:57.139829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709c00 of size 256
2019-09-07 17:10:57.139834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709d00 of size 256
2019-09-07 17:10:57.139840: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709e00 of size 256
2019-09-07 17:10:57.139846: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505709f00 of size 256
2019-09-07 17:10:57.139851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a000 of size 256
2019-09-07 17:10:57.139857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a100 of size 256
2019-09-07 17:10:57.139863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a200 of size 256
2019-09-07 17:10:57.139869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a300 of size 256
2019-09-07 17:10:57.139875: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a400 of size 256
2019-09-07 17:10:57.139881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a500 of size 256
2019-09-07 17:10:57.139887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a600 of size 256
2019-09-07 17:10:57.140074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a700 of size 256
2019-09-07 17:10:57.140080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a800 of size 256
2019-09-07 17:10:57.140087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570a900 of size 256
2019-09-07 17:10:57.140092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570aa00 of size 256
2019-09-07 17:10:57.140098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ab00 of size 256
2019-09-07 17:10:57.140104: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ac00 of size 256
2019-09-07 17:10:57.140110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ad00 of size 256
2019-09-07 17:10:57.140116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ae00 of size 256
2019-09-07 17:10:57.140122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570af00 of size 256
2019-09-07 17:10:57.140128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b000 of size 256
2019-09-07 17:10:57.140134: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b100 of size 256
2019-09-07 17:10:57.140140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b200 of size 256
2019-09-07 17:10:57.140146: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b300 of size 256
2019-09-07 17:10:57.140152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b400 of size 256
2019-09-07 17:10:57.140158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b500 of size 256
2019-09-07 17:10:57.140163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b600 of size 256
2019-09-07 17:10:57.140169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b700 of size 256
2019-09-07 17:10:57.140175: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b800 of size 256
2019-09-07 17:10:57.140181: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570b900 of size 256
2019-09-07 17:10:57.140187: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ba00 of size 256
2019-09-07 17:10:57.140195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570bb00 of size 256
2019-09-07 17:10:57.140202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570bc00 of size 256
2019-09-07 17:10:57.140210: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570bd00 of size 256
2019-09-07 17:10:57.140216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570be00 of size 256
2019-09-07 17:10:57.140222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570bf00 of size 256
2019-09-07 17:10:57.140228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c000 of size 256
2019-09-07 17:10:57.140234: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c100 of size 256
2019-09-07 17:10:57.140240: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c200 of size 256
2019-09-07 17:10:57.140246: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c300 of size 256
2019-09-07 17:10:57.140252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c400 of size 256
2019-09-07 17:10:57.140258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c500 of size 256
2019-09-07 17:10:57.140264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c600 of size 256
2019-09-07 17:10:57.140270: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c700 of size 256
2019-09-07 17:10:57.140276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c800 of size 256
2019-09-07 17:10:57.140282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570c900 of size 256
2019-09-07 17:10:57.140288: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ca00 of size 256
2019-09-07 17:10:57.140294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570cb00 of size 256
2019-09-07 17:10:57.140300: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570cc00 of size 256
2019-09-07 17:10:57.140305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570cd00 of size 256
2019-09-07 17:10:57.140311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ce00 of size 256
2019-09-07 17:10:57.140317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570cf00 of size 256
2019-09-07 17:10:57.140323: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d000 of size 256
2019-09-07 17:10:57.140329: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d100 of size 256
2019-09-07 17:10:57.140335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d200 of size 256
2019-09-07 17:10:57.140340: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d300 of size 256
2019-09-07 17:10:57.140346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d400 of size 256
2019-09-07 17:10:57.140352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d500 of size 256
2019-09-07 17:10:57.140358: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d600 of size 256
2019-09-07 17:10:57.140364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d700 of size 256
2019-09-07 17:10:57.140369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d800 of size 256
2019-09-07 17:10:57.140375: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570d900 of size 256
2019-09-07 17:10:57.140381: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570da00 of size 256
2019-09-07 17:10:57.140387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570db00 of size 256
2019-09-07 17:10:57.140393: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570dc00 of size 256
2019-09-07 17:10:57.140398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570dd00 of size 256
2019-09-07 17:10:57.140404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570de00 of size 256
2019-09-07 17:10:57.140410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570df00 of size 256
2019-09-07 17:10:57.140416: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e000 of size 256
2019-09-07 17:10:57.140421: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e100 of size 256
2019-09-07 17:10:57.140427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e200 of size 256
2019-09-07 17:10:57.140433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e300 of size 256
2019-09-07 17:10:57.140439: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e400 of size 256
2019-09-07 17:10:57.140445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e500 of size 256
2019-09-07 17:10:57.140451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e600 of size 256
2019-09-07 17:10:57.140457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e700 of size 256
2019-09-07 17:10:57.140463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e800 of size 256
2019-09-07 17:10:57.140468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570e900 of size 256
2019-09-07 17:10:57.140474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ea00 of size 256
2019-09-07 17:10:57.140655: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570eb00 of size 256
2019-09-07 17:10:57.140662: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ec00 of size 256
2019-09-07 17:10:57.140668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ed00 of size 256
2019-09-07 17:10:57.140674: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ee00 of size 256
2019-09-07 17:10:57.140680: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ef00 of size 256
2019-09-07 17:10:57.140686: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f000 of size 256
2019-09-07 17:10:57.140692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f100 of size 256
2019-09-07 17:10:57.140698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f200 of size 256
2019-09-07 17:10:57.140704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f300 of size 256
2019-09-07 17:10:57.140710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f400 of size 256
2019-09-07 17:10:57.140716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f500 of size 256
2019-09-07 17:10:57.140722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f600 of size 256
2019-09-07 17:10:57.140728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f700 of size 256
2019-09-07 17:10:57.140734: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f800 of size 256
2019-09-07 17:10:57.140740: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570f900 of size 256
2019-09-07 17:10:57.140746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570fa00 of size 256
2019-09-07 17:10:57.140752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570fb00 of size 256
2019-09-07 17:10:57.140757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570fc00 of size 256
2019-09-07 17:10:57.140763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570fd00 of size 256
2019-09-07 17:10:57.140771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570fe00 of size 256
2019-09-07 17:10:57.140779: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850570ff00 of size 256
2019-09-07 17:10:57.140786: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710000 of size 256
2019-09-07 17:10:57.140793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710100 of size 256
2019-09-07 17:10:57.140798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710200 of size 256
2019-09-07 17:10:57.140804: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710300 of size 256
2019-09-07 17:10:57.140810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710400 of size 256
2019-09-07 17:10:57.140816: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710500 of size 256
2019-09-07 17:10:57.140822: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710600 of size 256
2019-09-07 17:10:57.140828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710700 of size 256
2019-09-07 17:10:57.140834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710800 of size 256
2019-09-07 17:10:57.140840: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710900 of size 256
2019-09-07 17:10:57.140845: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710a00 of size 256
2019-09-07 17:10:57.140851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710b00 of size 256
2019-09-07 17:10:57.140857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710c00 of size 256
2019-09-07 17:10:57.140863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710d00 of size 256
2019-09-07 17:10:57.140869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710e00 of size 256
2019-09-07 17:10:57.140875: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505710f00 of size 256
2019-09-07 17:10:57.140881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711000 of size 256
2019-09-07 17:10:57.140887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711100 of size 256
2019-09-07 17:10:57.140892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711200 of size 256
2019-09-07 17:10:57.140898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711300 of size 256
2019-09-07 17:10:57.140904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711400 of size 256
2019-09-07 17:10:57.140910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711500 of size 256
2019-09-07 17:10:57.140916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711600 of size 256
2019-09-07 17:10:57.140921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711700 of size 256
2019-09-07 17:10:57.140927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711800 of size 256
2019-09-07 17:10:57.140933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711900 of size 256
2019-09-07 17:10:57.140938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711a00 of size 256
2019-09-07 17:10:57.140944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711b00 of size 256
2019-09-07 17:10:57.140950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711c00 of size 256
2019-09-07 17:10:57.140956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711d00 of size 256
2019-09-07 17:10:57.140962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711e00 of size 256
2019-09-07 17:10:57.140968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505711f00 of size 256
2019-09-07 17:10:57.140973: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712000 of size 256
2019-09-07 17:10:57.140979: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712100 of size 256
2019-09-07 17:10:57.140985: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712200 of size 256
2019-09-07 17:10:57.140990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712300 of size 256
2019-09-07 17:10:57.140996: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712400 of size 256
2019-09-07 17:10:57.141002: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712500 of size 256
2019-09-07 17:10:57.141008: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712600 of size 256
2019-09-07 17:10:57.141014: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712700 of size 256
2019-09-07 17:10:57.141019: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712800 of size 256
2019-09-07 17:10:57.141025: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712900 of size 256
2019-09-07 17:10:57.141031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712a00 of size 256
2019-09-07 17:10:57.141036: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712b00 of size 256
2019-09-07 17:10:57.141042: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712c00 of size 256
2019-09-07 17:10:57.141048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712d00 of size 256
2019-09-07 17:10:57.141054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712e00 of size 256
2019-09-07 17:10:57.141231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505712f00 of size 256
2019-09-07 17:10:57.141237: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713000 of size 256
2019-09-07 17:10:57.141243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713100 of size 256
2019-09-07 17:10:57.141249: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713200 of size 256
2019-09-07 17:10:57.141255: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713300 of size 256
2019-09-07 17:10:57.141261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713400 of size 256
2019-09-07 17:10:57.141267: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713500 of size 256
2019-09-07 17:10:57.141273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713600 of size 256
2019-09-07 17:10:57.141279: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713700 of size 256
2019-09-07 17:10:57.141285: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713800 of size 256
2019-09-07 17:10:57.141291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713900 of size 256
2019-09-07 17:10:57.141297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713a00 of size 256
2019-09-07 17:10:57.141303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713b00 of size 256
2019-09-07 17:10:57.141309: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713c00 of size 256
2019-09-07 17:10:57.141315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713d00 of size 256
2019-09-07 17:10:57.141321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713e00 of size 256
2019-09-07 17:10:57.141327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505713f00 of size 256
2019-09-07 17:10:57.141333: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714000 of size 256
2019-09-07 17:10:57.141339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714100 of size 256
2019-09-07 17:10:57.141347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714200 of size 256
2019-09-07 17:10:57.141353: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714300 of size 256
2019-09-07 17:10:57.141360: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714400 of size 256
2019-09-07 17:10:57.141366: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714500 of size 256
2019-09-07 17:10:57.141372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714600 of size 256
2019-09-07 17:10:57.141378: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714700 of size 256
2019-09-07 17:10:57.141384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714800 of size 256
2019-09-07 17:10:57.141390: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714900 of size 256
2019-09-07 17:10:57.141396: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714a00 of size 256
2019-09-07 17:10:57.141402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714b00 of size 256
2019-09-07 17:10:57.141407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714c00 of size 256
2019-09-07 17:10:57.141413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714d00 of size 256
2019-09-07 17:10:57.141419: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714e00 of size 256
2019-09-07 17:10:57.141425: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505714f00 of size 256
2019-09-07 17:10:57.141431: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715000 of size 256
2019-09-07 17:10:57.141436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715100 of size 256
2019-09-07 17:10:57.141442: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715200 of size 256
2019-09-07 17:10:57.141448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715300 of size 256
2019-09-07 17:10:57.141454: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715400 of size 256
2019-09-07 17:10:57.141460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715500 of size 256
2019-09-07 17:10:57.141466: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715600 of size 256
2019-09-07 17:10:57.141472: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715700 of size 256
2019-09-07 17:10:57.141477: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715800 of size 256
2019-09-07 17:10:57.141487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715900 of size 256
2019-09-07 17:10:57.141493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715a00 of size 256
2019-09-07 17:10:57.141499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715b00 of size 256
2019-09-07 17:10:57.141505: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715c00 of size 256
2019-09-07 17:10:57.141511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715d00 of size 256
2019-09-07 17:10:57.141517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715e00 of size 256
2019-09-07 17:10:57.141523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505715f00 of size 256
2019-09-07 17:10:57.141529: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716000 of size 256
2019-09-07 17:10:57.141535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716100 of size 256
2019-09-07 17:10:57.141541: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716200 of size 256
2019-09-07 17:10:57.141547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716300 of size 256
2019-09-07 17:10:57.141553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716400 of size 256
2019-09-07 17:10:57.141559: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716500 of size 256
2019-09-07 17:10:57.141564: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716600 of size 256
2019-09-07 17:10:57.141570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716700 of size 256
2019-09-07 17:10:57.141576: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716800 of size 256
2019-09-07 17:10:57.141582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716900 of size 256
2019-09-07 17:10:57.141587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716a00 of size 256
2019-09-07 17:10:57.141593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716b00 of size 256
2019-09-07 17:10:57.141599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716c00 of size 256
2019-09-07 17:10:57.141605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716d00 of size 256
2019-09-07 17:10:57.141610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716e00 of size 256
2019-09-07 17:10:57.141617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505716f00 of size 256
2019-09-07 17:10:57.141622: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717000 of size 256
2019-09-07 17:10:57.141628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717100 of size 256
2019-09-07 17:10:57.141634: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717200 of size 256
2019-09-07 17:10:57.141789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717300 of size 256
2019-09-07 17:10:57.141795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717400 of size 256
2019-09-07 17:10:57.141801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717500 of size 256
2019-09-07 17:10:57.141807: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717600 of size 256
2019-09-07 17:10:57.141813: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717700 of size 256
2019-09-07 17:10:57.141819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717800 of size 256
2019-09-07 17:10:57.141825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717900 of size 256
2019-09-07 17:10:57.141831: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717a00 of size 256
2019-09-07 17:10:57.141837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717b00 of size 256
2019-09-07 17:10:57.141843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717c00 of size 256
2019-09-07 17:10:57.141849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717d00 of size 256
2019-09-07 17:10:57.141854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717e00 of size 256
2019-09-07 17:10:57.141861: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505717f00 of size 256
2019-09-07 17:10:57.141866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718000 of size 256
2019-09-07 17:10:57.141872: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718100 of size 256
2019-09-07 17:10:57.141878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718200 of size 256
2019-09-07 17:10:57.141884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718300 of size 256
2019-09-07 17:10:57.141892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718400 of size 256
2019-09-07 17:10:57.141898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718500 of size 256
2019-09-07 17:10:57.141905: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718600 of size 256
2019-09-07 17:10:57.141911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718700 of size 256
2019-09-07 17:10:57.141917: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718800 of size 256
2019-09-07 17:10:57.141924: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718900 of size 256
2019-09-07 17:10:57.141930: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718a00 of size 256
2019-09-07 17:10:57.141936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718b00 of size 256
2019-09-07 17:10:57.141942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718c00 of size 256
2019-09-07 17:10:57.141948: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718d00 of size 256
2019-09-07 17:10:57.141954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718e00 of size 256
2019-09-07 17:10:57.141960: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505718f00 of size 256
2019-09-07 17:10:57.141966: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719000 of size 256
2019-09-07 17:10:57.141972: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719100 of size 256
2019-09-07 17:10:57.141978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719200 of size 256
2019-09-07 17:10:57.141984: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719300 of size 256
2019-09-07 17:10:57.141989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719400 of size 256
2019-09-07 17:10:57.141995: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719500 of size 256
2019-09-07 17:10:57.142001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719600 of size 256
2019-09-07 17:10:57.142007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719700 of size 256
2019-09-07 17:10:57.142013: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719800 of size 256
2019-09-07 17:10:57.142019: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719900 of size 256
2019-09-07 17:10:57.142024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719a00 of size 256
2019-09-07 17:10:57.142030: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719b00 of size 256
2019-09-07 17:10:57.142036: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719c00 of size 256
2019-09-07 17:10:57.142042: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719d00 of size 256
2019-09-07 17:10:57.142047: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719e00 of size 256
2019-09-07 17:10:57.142053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505719f00 of size 256
2019-09-07 17:10:57.142059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a000 of size 256
2019-09-07 17:10:57.142065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a100 of size 256
2019-09-07 17:10:57.142071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a200 of size 256
2019-09-07 17:10:57.142077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a300 of size 256
2019-09-07 17:10:57.142082: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a400 of size 256
2019-09-07 17:10:57.142088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a500 of size 256
2019-09-07 17:10:57.142094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a600 of size 256
2019-09-07 17:10:57.142100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a700 of size 256
2019-09-07 17:10:57.142106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a800 of size 256
2019-09-07 17:10:57.142112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571a900 of size 256
2019-09-07 17:10:57.142117: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571aa00 of size 256
2019-09-07 17:10:57.142123: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ab00 of size 256
2019-09-07 17:10:57.142129: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ac00 of size 256
2019-09-07 17:10:57.142135: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ad00 of size 256
2019-09-07 17:10:57.142141: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ae00 of size 2048
2019-09-07 17:10:57.142147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571b600 of size 2048
2019-09-07 17:10:57.142153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571be00 of size 2048
2019-09-07 17:10:57.142159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571c600 of size 2048
2019-09-07 17:10:57.142165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ce00 of size 4096
2019-09-07 17:10:57.142170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571de00 of size 4096
2019-09-07 17:10:57.142176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571ee00 of size 4096
2019-09-07 17:10:57.142182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850571fe00 of size 4096
2019-09-07 17:10:57.142188: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505720e00 of size 1024
2019-09-07 17:10:57.142348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505721200 of size 1024
2019-09-07 17:10:57.142355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505721600 of size 1024
2019-09-07 17:10:57.142361: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505721a00 of size 1024
2019-09-07 17:10:57.142367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505721e00 of size 2048
2019-09-07 17:10:57.142373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505722600 of size 2048
2019-09-07 17:10:57.142379: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505722e00 of size 2048
2019-09-07 17:10:57.142385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505723600 of size 2048
2019-09-07 17:10:57.142391: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505723e00 of size 1024
2019-09-07 17:10:57.142397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505724200 of size 1024
2019-09-07 17:10:57.142402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505724600 of size 1024
2019-09-07 17:10:57.142409: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505724a00 of size 1024
2019-09-07 17:10:57.142414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505724e00 of size 2048
2019-09-07 17:10:57.142420: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505725600 of size 2048
2019-09-07 17:10:57.142426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505725e00 of size 2048
2019-09-07 17:10:57.142432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505726600 of size 2048
2019-09-07 17:10:57.142438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505726e00 of size 512
2019-09-07 17:10:57.142444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727000 of size 512
2019-09-07 17:10:57.142450: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727200 of size 512
2019-09-07 17:10:57.142456: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727400 of size 512
2019-09-07 17:10:57.142462: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727600 of size 512
2019-09-07 17:10:57.142468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727800 of size 512
2019-09-07 17:10:57.142473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727a00 of size 512
2019-09-07 17:10:57.142481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727c00 of size 512
2019-09-07 17:10:57.142492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505727e00 of size 1024
2019-09-07 17:10:57.142500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505728200 of size 1024
2019-09-07 17:10:57.142506: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505728600 of size 1024
2019-09-07 17:10:57.142512: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505728a00 of size 1024
2019-09-07 17:10:57.142518: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505728e00 of size 512
2019-09-07 17:10:57.142525: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729000 of size 512
2019-09-07 17:10:57.142530: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729200 of size 512
2019-09-07 17:10:57.142537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729400 of size 512
2019-09-07 17:10:57.142542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729600 of size 1024
2019-09-07 17:10:57.142549: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729a00 of size 1024
2019-09-07 17:10:57.142555: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505729e00 of size 1024
2019-09-07 17:10:57.142561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572a200 of size 1024
2019-09-07 17:10:57.142567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572a600 of size 2048
2019-09-07 17:10:57.142572: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572ae00 of size 2048
2019-09-07 17:10:57.142578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572b600 of size 2048
2019-09-07 17:10:57.142584: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572be00 of size 2048
2019-09-07 17:10:57.142590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572c600 of size 512
2019-09-07 17:10:57.142596: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572c800 of size 512
2019-09-07 17:10:57.142602: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572ca00 of size 512
2019-09-07 17:10:57.142608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572cc00 of size 512
2019-09-07 17:10:57.142613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572ce00 of size 1024
2019-09-07 17:10:57.142619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572d200 of size 1024
2019-09-07 17:10:57.142625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572d600 of size 1024
2019-09-07 17:10:57.142631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572da00 of size 1024
2019-09-07 17:10:57.142637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572de00 of size 4096
2019-09-07 17:10:57.142643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572ee00 of size 4096
2019-09-07 17:10:57.142648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850572fe00 of size 4096
2019-09-07 17:10:57.142654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505730e00 of size 4096
2019-09-07 17:10:57.142660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505731e00 of size 2048
2019-09-07 17:10:57.142666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505732600 of size 2048
2019-09-07 17:10:57.142672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505732e00 of size 2048
2019-09-07 17:10:57.142678: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505733600 of size 2048
2019-09-07 17:10:57.142683: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505733e00 of size 4096
2019-09-07 17:10:57.142689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505734e00 of size 4096
2019-09-07 17:10:57.142695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505735e00 of size 4096
2019-09-07 17:10:57.142701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505736e00 of size 4096
2019-09-07 17:10:57.142707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505737e00 of size 1024
2019-09-07 17:10:57.142713: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505738200 of size 1024
2019-09-07 17:10:57.142719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505738600 of size 1024
2019-09-07 17:10:57.142726: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505738a00 of size 1024
2019-09-07 17:10:57.142731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505738e00 of size 1024
2019-09-07 17:10:57.142737: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505739200 of size 1024
2019-09-07 17:10:57.142743: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505739600 of size 1024
2019-09-07 17:10:57.142749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505739a00 of size 1024
2019-09-07 17:10:57.142755: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505739e00 of size 2048
2019-09-07 17:10:57.142921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573a600 of size 2048
2019-09-07 17:10:57.142928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573ae00 of size 2048
2019-09-07 17:10:57.142934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573b600 of size 2048
2019-09-07 17:10:57.142939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573be00 of size 256
2019-09-07 17:10:57.142945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573bf00 of size 256
2019-09-07 17:10:57.142951: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c000 of size 256
2019-09-07 17:10:57.142957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c100 of size 256
2019-09-07 17:10:57.142963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c200 of size 256
2019-09-07 17:10:57.142969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c300 of size 256
2019-09-07 17:10:57.142975: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c400 of size 256
2019-09-07 17:10:57.142981: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c500 of size 256
2019-09-07 17:10:57.142987: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c600 of size 256
2019-09-07 17:10:57.142993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c700 of size 256
2019-09-07 17:10:57.142998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c800 of size 256
2019-09-07 17:10:57.143004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573c900 of size 256
2019-09-07 17:10:57.143010: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573ca00 of size 256
2019-09-07 17:10:57.143016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573cb00 of size 256
2019-09-07 17:10:57.143022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573cc00 of size 256
2019-09-07 17:10:57.143028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573cd00 of size 256
2019-09-07 17:10:57.143037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573ce00 of size 256
2019-09-07 17:10:57.143043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573cf00 of size 256
2019-09-07 17:10:57.143050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573d000 of size 256
2019-09-07 17:10:57.143056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573d100 of size 2048
2019-09-07 17:10:57.143062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573d900 of size 2048
2019-09-07 17:10:57.143068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573e100 of size 2048
2019-09-07 17:10:57.143074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573e900 of size 3840
2019-09-07 17:10:57.143081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850573f800 of size 512
2019-09-07 17:10:57.143086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f850573fa00 of size 2048
2019-09-07 17:10:57.143092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740200 of size 256
2019-09-07 17:10:57.143098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740300 of size 256
2019-09-07 17:10:57.143104: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740400 of size 256
2019-09-07 17:10:57.143110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740500 of size 256
2019-09-07 17:10:57.143116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740600 of size 256
2019-09-07 17:10:57.143122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740700 of size 256
2019-09-07 17:10:57.143128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740800 of size 256
2019-09-07 17:10:57.143133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740900 of size 256
2019-09-07 17:10:57.143139: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740a00 of size 256
2019-09-07 17:10:57.143145: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740b00 of size 256
2019-09-07 17:10:57.143150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740c00 of size 256
2019-09-07 17:10:57.143156: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740d00 of size 256
2019-09-07 17:10:57.143162: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740e00 of size 256
2019-09-07 17:10:57.143168: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505740f00 of size 256
2019-09-07 17:10:57.143174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8505741000 of size 512
2019-09-07 17:10:57.143179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741200 of size 256
2019-09-07 17:10:57.143186: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741300 of size 256
2019-09-07 17:10:57.143192: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741400 of size 1024
2019-09-07 17:10:57.143198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8505741800 of size 1280
2019-09-07 17:10:57.143203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741d00 of size 256
2019-09-07 17:10:57.143210: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741e00 of size 256
2019-09-07 17:10:57.143215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505741f00 of size 1024
2019-09-07 17:10:57.143221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505742300 of size 1024
2019-09-07 17:10:57.143227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505742700 of size 1024
2019-09-07 17:10:57.143233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505742b00 of size 1024
2019-09-07 17:10:57.143239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505742f00 of size 4096
2019-09-07 17:10:57.143245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505743f00 of size 4096
2019-09-07 17:10:57.143251: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505744f00 of size 14336
2019-09-07 17:10:57.143257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505748700 of size 256
2019-09-07 17:10:57.143262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8505748800 of size 512
2019-09-07 17:10:57.143268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505748a00 of size 512
2019-09-07 17:10:57.143274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505748c00 of size 256
2019-09-07 17:10:57.143281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505748d00 of size 1024
2019-09-07 17:10:57.143286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505749100 of size 1024
2019-09-07 17:10:57.143293: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505749500 of size 2048
2019-09-07 17:10:57.143298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505749d00 of size 2048
2019-09-07 17:10:57.143304: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850574a500 of size 256
2019-09-07 17:10:57.143311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850574a600 of size 12288
2019-09-07 17:10:57.143317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850574d600 of size 12288
2019-09-07 17:10:57.143323: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505750600 of size 12288
2019-09-07 17:10:57.143497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505753600 of size 12288
2019-09-07 17:10:57.143504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8505756600 of size 157952
2019-09-07 17:10:57.143510: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850577cf00 of size 3309824
2019-09-07 17:10:57.143517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505aa5000 of size 827648
2019-09-07 17:10:57.143523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b6f100 of size 4096
2019-09-07 17:10:57.143529: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b70100 of size 4096
2019-09-07 17:10:57.143535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b71100 of size 2048
2019-09-07 17:10:57.143541: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b71900 of size 1024
2019-09-07 17:10:57.143547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b71d00 of size 256
2019-09-07 17:10:57.143553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b71e00 of size 256
2019-09-07 17:10:57.143559: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b71f00 of size 256
2019-09-07 17:10:57.143565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8505b72000 of size 256
2019-09-07 17:10:57.143571: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b72100 of size 4096
2019-09-07 17:10:57.143577: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b73100 of size 4096
2019-09-07 17:10:57.143583: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b74100 of size 2048
2019-09-07 17:10:57.143588: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b74900 of size 2048
2019-09-07 17:10:57.143595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b75100 of size 97536
2019-09-07 17:10:57.143603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505b8ce00 of size 389376
2019-09-07 17:10:57.143611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505bebf00 of size 24576
2019-09-07 17:10:57.143618: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8505bf1f00 of size 11960832
2019-09-07 17:10:57.143625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850675a100 of size 24576
2019-09-07 17:10:57.143632: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506760100 of size 133888
2019-09-07 17:10:57.143639: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506780c00 of size 12288
2019-09-07 17:10:57.143645: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506783c00 of size 12288
2019-09-07 17:10:57.143651: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506786c00 of size 389376
2019-09-07 17:10:57.143657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85067e5d00 of size 133888
2019-09-07 17:10:57.143663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506806800 of size 255488
2019-09-07 17:10:57.143669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506844e00 of size 12288
2019-09-07 17:10:57.143676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506847e00 of size 194816
2019-09-07 17:10:57.143682: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506877700 of size 194816
2019-09-07 17:10:57.143688: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85068a7000 of size 194816
2019-09-07 17:10:57.143694: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85068d6900 of size 2141696
2019-09-07 17:10:57.143700: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506ae1700 of size 194816
2019-09-07 17:10:57.143706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b11000 of size 97536
2019-09-07 17:10:57.143712: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b28d00 of size 48896
2019-09-07 17:10:57.143718: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b34c00 of size 24576
2019-09-07 17:10:57.143724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b3ac00 of size 24576
2019-09-07 17:10:57.143730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b40c00 of size 12288
2019-09-07 17:10:57.143736: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b43c00 of size 12288
2019-09-07 17:10:57.143742: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b46c00 of size 12288
2019-09-07 17:10:57.143747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b49c00 of size 12288
2019-09-07 17:10:57.143753: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b4cc00 of size 12288
2019-09-07 17:10:57.143759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8506b4fc00 of size 2048
2019-09-07 17:10:57.143765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b50400 of size 2048
2019-09-07 17:10:57.143771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b50c00 of size 2048
2019-09-07 17:10:57.143777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8506b51400 of size 2048
2019-09-07 17:10:57.143783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b51c00 of size 12288
2019-09-07 17:10:57.143789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b54c00 of size 15360
2019-09-07 17:10:57.143795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b58800 of size 48896
2019-09-07 17:10:57.143801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506b64700 of size 535552
2019-09-07 17:10:57.143808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506be7300 of size 48896
2019-09-07 17:10:57.143814: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506bf3200 of size 4153344
2019-09-07 17:10:57.143820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8506fe9200 of size 2076672
2019-09-07 17:10:57.143827: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85071e4200 of size 2097152
2019-09-07 17:10:57.143833: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85073e4200 of size 4132864
2019-09-07 17:10:57.143839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85077d5200 of size 12288
2019-09-07 17:10:57.143846: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85077d8200 of size 194816
2019-09-07 17:10:57.143852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8507807b00 of size 48896
2019-09-07 17:10:57.143859: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8507813a00 of size 16613376
2019-09-07 17:10:57.143865: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85087eba00 of size 2076672
2019-09-07 17:10:57.143871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85089e6a00 of size 4153344
2019-09-07 17:10:57.143877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8508ddca00 of size 4153344
2019-09-07 17:10:57.143883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85091d2a00 of size 4153344
2019-09-07 17:10:57.143889: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85095c8a00 of size 2076672
2019-09-07 17:10:57.143895: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85097c3a00 of size 2097152
2019-09-07 17:10:57.143900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85099c3a00 of size 2076672
2019-09-07 17:10:57.143906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8509bbea00 of size 2076672
2019-09-07 17:10:57.143912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8509db9a00 of size 2240512
2019-09-07 17:10:57.144071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8509fdca00 of size 4153344
2019-09-07 17:10:57.144078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850a3d2a00 of size 4153344
2019-09-07 17:10:57.144084: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850a7c8a00 of size 4153344
2019-09-07 17:10:57.144090: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850abbea00 of size 4153344
2019-09-07 17:10:57.144096: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850afb4a00 of size 2097152
2019-09-07 17:10:57.144102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b1b4a00 of size 2076672
2019-09-07 17:10:57.144108: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b3afa00 of size 2076672
2019-09-07 17:10:57.144115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b5aaa00 of size 1562624
2019-09-07 17:10:57.144121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b728200 of size 1038336
2019-09-07 17:10:57.144127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b825a00 of size 1716224
2019-09-07 17:10:57.144134: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850b9c8a00 of size 4153344
2019-09-07 17:10:57.144140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850bdbea00 of size 4153344
2019-09-07 17:10:57.144146: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850c1b4a00 of size 4153344
2019-09-07 17:10:57.144152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850c5aaa00 of size 4153344
2019-09-07 17:10:57.144159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850c9a0a00 of size 1038336
2019-09-07 17:10:57.144165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850ca9e200 of size 194816
2019-09-07 17:10:57.144171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850cacdb00 of size 5181184
2019-09-07 17:10:57.144179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850cfbea00 of size 4153344
2019-09-07 17:10:57.144187: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850d3b4a00 of size 4153344
2019-09-07 17:10:57.144194: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850d7aaa00 of size 12460032
2019-09-07 17:10:57.144201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e38ca00 of size 133888
2019-09-07 17:10:57.144207: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3ad500 of size 33536
2019-09-07 17:10:57.144215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f850e3b5800 of size 8192
2019-09-07 17:10:57.144221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3b7800 of size 133888
2019-09-07 17:10:57.144227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3d8300 of size 24576
2019-09-07 17:10:57.144233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3de300 of size 12288
2019-09-07 17:10:57.144239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3e1300 of size 24576
2019-09-07 17:10:57.144245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3e7300 of size 12288
2019-09-07 17:10:57.144252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e3ea300 of size 133888
2019-09-07 17:10:57.144258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e40ae00 of size 12288
2019-09-07 17:10:57.144264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e40de00 of size 12288
2019-09-07 17:10:57.144270: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e410e00 of size 244736
2019-09-07 17:10:57.144277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e44ca00 of size 4153344
2019-09-07 17:10:57.144283: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850e842a00 of size 4153344
2019-09-07 17:10:57.144288: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850ec38a00 of size 4153344
2019-09-07 17:10:57.144294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850f02ea00 of size 13025280
2019-09-07 17:10:57.144301: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f850fc9aa00 of size 8306688
2019-09-07 17:10:57.144307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8510486a00 of size 12288
2019-09-07 17:10:57.144313: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8510489a00 of size 133888
2019-09-07 17:10:57.144319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f85104aa500 of size 1536
2019-09-07 17:10:57.144325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85104aab00 of size 8306688
2019-09-07 17:10:57.144331: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8510c96b00 of size 12288
2019-09-07 17:10:57.144337: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8510c99b00 of size 133888
2019-09-07 17:10:57.144343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8510cba600 of size 8306688
2019-09-07 17:10:57.144349: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85114a6600 of size 133888
2019-09-07 17:10:57.144355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85114c7100 of size 4677632
2019-09-07 17:10:57.144362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851193d100 of size 4153344
2019-09-07 17:10:57.144367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8511d33100 of size 4153344
2019-09-07 17:10:57.144374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8512129100 of size 4153344
2019-09-07 17:10:57.144380: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851251f100 of size 1536
2019-09-07 17:10:57.144386: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f851251f700 of size 2048
2019-09-07 17:10:57.144391: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851251ff00 of size 2048
2019-09-07 17:10:57.144397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8512520700 of size 4096
2019-09-07 17:10:57.144403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8512521700 of size 13025280
2019-09-07 17:10:57.144410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851318d700 of size 8306688
2019-09-07 17:10:57.144416: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8513979700 of size 24576
2019-09-07 17:10:57.144422: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851397f700 of size 48896
2019-09-07 17:10:57.144428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851398b600 of size 24576
2019-09-07 17:10:57.144435: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8513991600 of size 13011200
2019-09-07 17:10:57.144441: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85145f9f00 of size 24576
2019-09-07 17:10:57.144447: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85145fff00 of size 24576
2019-09-07 17:10:57.144452: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8514605f00 of size 8306688
2019-09-07 17:10:57.144459: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8514df1f00 of size 24576
2019-09-07 17:10:57.144464: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8514df7f00 of size 24576
2019-09-07 17:10:57.144471: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8514dfdf00 of size 4677632
2019-09-07 17:10:57.144477: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8515273f00 of size 8192
2019-09-07 17:10:57.144636: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515275f00 of size 4153344
2019-09-07 17:10:57.144643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f851566bf00 of size 4096
2019-09-07 17:10:57.144649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851566cf00 of size 2048
2019-09-07 17:10:57.144655: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f851566d700 of size 6144
2019-09-07 17:10:57.144661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851566ef00 of size 24576
2019-09-07 17:10:57.144667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515674f00 of size 6144
2019-09-07 17:10:57.144673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8515676700 of size 4096
2019-09-07 17:10:57.144679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515677700 of size 4153344
2019-09-07 17:10:57.144686: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515a6d700 of size 4153344
2019-09-07 17:10:57.144692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515e63700 of size 24576
2019-09-07 17:10:57.144698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515e69700 of size 24576
2019-09-07 17:10:57.144704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8515e6f700 of size 2207744
2019-09-07 17:10:57.144711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851608a700 of size 2076672
2019-09-07 17:10:57.144717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8516285700 of size 12288
2019-09-07 17:10:57.144723: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8516288700 of size 133888
2019-09-07 17:10:57.144729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162a9200 of size 24576
2019-09-07 17:10:57.144735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162af200 of size 24576
2019-09-07 17:10:57.144741: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f85162b5200 of size 4096
2019-09-07 17:10:57.144750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162b6200 of size 24576
2019-09-07 17:10:57.144757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162bc200 of size 24576
2019-09-07 17:10:57.144765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162c2200 of size 24576
2019-09-07 17:10:57.144771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162c8200 of size 24576
2019-09-07 17:10:57.144777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162ce200 of size 24576
2019-09-07 17:10:57.144783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162d4200 of size 24576
2019-09-07 17:10:57.144790: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162da200 of size 24576
2019-09-07 17:10:57.144796: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162e0200 of size 24576
2019-09-07 17:10:57.144803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162e6200 of size 24576
2019-09-07 17:10:57.144809: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f85162ec200 of size 8192
2019-09-07 17:10:57.144815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85162ee200 of size 8312064
2019-09-07 17:10:57.144822: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8516adb700 of size 8306688
2019-09-07 17:10:57.144828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85172c7700 of size 24576
2019-09-07 17:10:57.144834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85172cd700 of size 8306688
2019-09-07 17:10:57.144841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8517ab9700 of size 2076672
2019-09-07 17:10:57.144847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8517cb4700 of size 8306688
2019-09-07 17:10:57.144853: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f85184a0700 of size 2076672
2019-09-07 17:10:57.144859: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f851869b700 of size 877056
2019-09-07 17:10:57.144866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8518771900 of size 778752
2019-09-07 17:10:57.144872: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f851882fb00 of size 2076672
2019-09-07 17:10:57.144878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8518a2ab00 of size 8306688
2019-09-07 17:10:57.144884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f8519216b00 of size 8306688
2019-09-07 17:10:57.144890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f8519a02b00 of size 18928896
2019-09-07 17:10:57.144896: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2019-09-07 17:10:57.144906: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 506 Chunks of size 256 totalling 126.5KiB
2019-09-07 17:10:57.144913: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 150 Chunks of size 512 totalling 75.0KiB
2019-09-07 17:10:57.144921: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 249 Chunks of size 1024 totalling 249.0KiB
2019-09-07 17:10:57.144928: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2019-09-07 17:10:57.144934: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1536 totalling 1.5KiB
2019-09-07 17:10:57.144941: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 206 Chunks of size 2048 totalling 412.0KiB
2019-09-07 17:10:57.144948: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 3584 totalling 3.5KiB
2019-09-07 17:10:57.144955: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 3840 totalling 3.8KiB
2019-09-07 17:10:57.144962: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 58 Chunks of size 4096 totalling 232.0KiB
2019-09-07 17:10:57.144968: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 6144 totalling 6.0KiB
2019-09-07 17:10:57.144975: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8192 totalling 8.0KiB
2019-09-07 17:10:57.144982: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 21 Chunks of size 12288 totalling 252.0KiB
2019-09-07 17:10:57.144989: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 14336 totalling 14.0KiB
2019-09-07 17:10:57.144996: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 15360 totalling 15.0KiB
2019-09-07 17:10:57.145004: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 27 Chunks of size 24576 totalling 648.0KiB
2019-09-07 17:10:57.145010: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 32768 totalling 64.0KiB
2019-09-07 17:10:57.145018: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 33536 totalling 32.8KiB
2019-09-07 17:10:57.145024: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 48896 totalling 238.8KiB
2019-09-07 17:10:57.145032: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 49152 totalling 144.0KiB
2019-09-07 17:10:57.145039: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 73728 totalling 144.0KiB
2019-09-07 17:10:57.145046: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 97536 totalling 190.5KiB
2019-09-07 17:10:57.145053: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 98304 totalling 288.0KiB
2019-09-07 17:10:57.145060: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 15 Chunks of size 131072 totalling 1.88MiB
2019-09-07 17:10:57.145066: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 133888 totalling 1.15MiB
2019-09-07 17:10:57.145073: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 194816 totalling 1.11MiB
2019-09-07 17:10:57.145079: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 196608 totalling 1.12MiB
2019-09-07 17:10:57.145244: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 244736 totalling 239.0KiB
2019-09-07 17:10:57.145251: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 255488 totalling 249.5KiB
2019-09-07 17:10:57.145258: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 294912 totalling 864.0KiB
2019-09-07 17:10:57.145265: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 389376 totalling 760.5KiB
2019-09-07 17:10:57.145272: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 15 Chunks of size 524288 totalling 7.50MiB
2019-09-07 17:10:57.145279: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 535552 totalling 523.0KiB
2019-09-07 17:10:57.145286: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 778752 totalling 760.5KiB
2019-09-07 17:10:57.145293: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 786432 totalling 2.25MiB
2019-09-07 17:10:57.145300: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 827648 totalling 808.2KiB
2019-09-07 17:10:57.145307: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 877056 totalling 856.5KiB
2019-09-07 17:10:57.145314: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1038336 totalling 1.98MiB
2019-09-07 17:10:57.145321: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 16 Chunks of size 1179648 totalling 18.00MiB
2019-09-07 17:10:57.145328: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1562624 totalling 1.49MiB
2019-09-07 17:10:57.145334: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1716224 totalling 1.64MiB
2019-09-07 17:10:57.145344: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 2076672 totalling 19.80MiB
2019-09-07 17:10:57.145351: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 14 Chunks of size 2097152 totalling 28.00MiB
2019-09-07 17:10:57.145359: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2141696 totalling 2.04MiB
2019-09-07 17:10:57.145366: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2207744 totalling 2.11MiB
2019-09-07 17:10:57.145373: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2240512 totalling 2.14MiB
2019-09-07 17:10:57.145379: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 3309824 totalling 3.16MiB
2019-09-07 17:10:57.145387: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4132864 totalling 3.94MiB
2019-09-07 17:10:57.145394: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 23 Chunks of size 4153344 totalling 91.10MiB
2019-09-07 17:10:57.145401: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4677632 totalling 8.92MiB
2019-09-07 17:10:57.145408: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 16 Chunks of size 4718592 totalling 72.00MiB
2019-09-07 17:10:57.145415: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5181184 totalling 4.94MiB
2019-09-07 17:10:57.145422: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 8306688 totalling 79.22MiB
2019-09-07 17:10:57.145429: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8312064 totalling 7.93MiB
2019-09-07 17:10:57.145436: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 11960832 totalling 11.41MiB
2019-09-07 17:10:57.145443: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12460032 totalling 11.88MiB
2019-09-07 17:10:57.145451: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 13011200 totalling 12.41MiB
2019-09-07 17:10:57.145458: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 13025280 totalling 24.84MiB
2019-09-07 17:10:57.145465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16613376 totalling 15.84MiB
2019-09-07 17:10:57.145472: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 18874368 totalling 216.00MiB
2019-09-07 17:10:57.145480: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 663.82MiB
2019-09-07 17:10:57.145495: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 717291520
InUse: 696068608
MaxInUse: 711933952
NumAllocs: 2850
MaxAllocSize: 273106432
2019-09-07 17:10:57.145580: W tensorflow/core/common_runtime/bfc_allocator.cc:279]
2019-09-07 17:10:57.145465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16613376 totalling 15.84MiB
2019-09-07 17:10:57.145472: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 18874368 totalling 216.00MiB
2019-09-07 17:10:57.145480: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 663.82MiB
2019-09-07 17:10:57.145495: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 717291520
InUse: 696068608
MaxInUse: 711933952
NumAllocs: 2850
MaxAllocSize: 273106432
2019-09-07 17:10:57.145580: W tensorflow/core/common_runtime/bfc_allocator.cc:279] **************************************************************************************************__
2019-09-07 17:10:57.145611: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at concat_op.cc:152 : Resource exhausted: OOM when allocating tensor with shape[6,384,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6,384,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: yolov3/yolov3_head/concat_1 = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/concat_1-0-TransposeNHWCToNCHW-LayoutOptimizer, yolov3/darknet53_body/add_10, yolov3/yolov3_head/concat-2-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: add_20/_807 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_51259_add_20", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 140, in <module>
feed_dict={is_training: True})
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6,384,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: yolov3/yolov3_head/concat_1 = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/concat_1-0-TransposeNHWCToNCHW-LayoutOptimizer, yolov3/darknet53_body/add_10, yolov3/yolov3_head/concat-2-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: add_20/_807 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_51259_add_20", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'yolov3/yolov3_head/concat_1', defined at:
File "train.py", line 74, in <module>
pred_feature_maps = yolo_model.forward(image, is_training=is_training)
File "/gpfsssd/sigs/competition17-gpu-notebook-sig/Jupyter-5.4.0/competition17-gpu-notebook-sig/9ea0d7c5-23dd-4950-803b-4a4d9749a80f/Jupyter-5-4-0-225/notebooks/yolo_v3/model.py", line 72, in forward
concat2 = tf.concat([inter2, route_1], axis=3)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1114, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1030, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/gpfsssd/anacondas/ana1/anaconda/envs/env2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[6,384,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: yolov3/yolov3_head/concat_1 = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/concat_1-0-TransposeNHWCToNCHW-LayoutOptimizer, yolov3/darknet53_body/add_10, yolov3/yolov3_head/concat-2-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: add_20/_807 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_51259_add_20", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
sh-4.2$
輸入nvidia-smi，發現GPU-0已經被佔用，導致訓練內存不夠，程序報錯。
Sat Sep 7 17:11:53 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.37 Driver Version: 396.37 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P40 Off | 00000000:2D:00.0 Off | 0 |
| N/A 80C P0 176W / 250W | 21928MiB / 22919MiB | 90% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla P40 Off | 00000000:31:00.0 Off | 0 |
| N/A 29C P0 48W / 250W | 21789MiB / 22919MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla P40 Off | 00000000:35:00.0 Off | 0 |
| N/A 30C P0 49W / 250W | 21789MiB / 22919MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla P40 Off | 00000000:39:00.0 Off | 0 |
| N/A 31C P0 49W / 250W | 21789MiB / 22919MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 17149 C ...ndas/ana1/anaconda/envs/env1/bin/python 165MiB |
| 0 46136 C python 21749MiB |
| 1 46136 C python 21775MiB |
| 2 46136 C python 21775MiB |
| 3 46136 C python 21775MiB |
+-----------------------------------------------------------------------------+
sh-4.2$
2019-09-07