项目描述 ：
2019-09-05 17:37:38.843481: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 127 Chunks of size 835584 totalling 101.20MiB 2019-09-05 17:37:38.911620: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 851968 totalling 4.06MiB 2019-09-05 17:37:38.928072: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 156 Chunks of size 860160 totalling 127.97MiB 2019-09-05 17:37:39.003703: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 868352 totalling 4.14MiB 2019-09-05 17:37:39.081868: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 884736 totalling 4.22MiB 2019-09-05 17:37:39.097021: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 901120 totalling 4.30MiB 2019-09-05 17:37:39.110337: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 917504 totalling 2.63MiB 2019-09-05 17:37:39.185849: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 933888 totalling 2.67MiB 2019-09-05 17:37:39.203580: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 950272 totalling 2.72MiB 2019-09-05 17:37:39.249281: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 966656 totalling 2.77MiB 2019-09-05 17:37:39.260632: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 983040 totalling 9.38MiB 2019-09-05 17:37:39.272127: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 62 Chunks of size 1032192 totalling 61.03MiB 2019-09-05 17:37:39.336373: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 190 Chunks of size 1048576 totalling 190.00MiB 2019-09-05 17:37:39.394669: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 20 Chunks of size 1114112 totalling 21.25MiB 2019-09-05 17:37:39.406105: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 70 Chunks of size 1179648 totalling 78.75MiB 2019-09-05 17:37:39.416286: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 1327104 totalling 12.66MiB 2019-09-05 17:37:39.425292: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 1572864 totalling 15.00MiB 2019-09-05 17:37:39.461690: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 122 Chunks of size 1597440 totalling 185.86MiB 2019-09-05 17:37:39.473597: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 1638400 totalling 15.63MiB 2019-09-05 17:37:39.482200: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 122 Chunks of size 1671168 totalling 194.44MiB 2019-09-05 17:37:39.542267: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 62 Chunks of size 1769472 totalling 104.63MiB 2019-09-05 17:37:39.621103: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 1966080 totalling 18.75MiB 2019-09-05 17:37:39.637613: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 116 Chunks of size 2097152 totalling 232.00MiB 2019-09-05 17:37:39.717971: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 158 Chunks of size 2119936 totalling 319.43MiB 2019-09-05 17:37:39.733892: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 2211840 totalling 21.09MiB 2019-09-05 17:37:39.814283: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 2293760 totalling 21.88MiB 2019-09-05 17:37:39.830941: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 248 Chunks of size 2359296 totalling 558.00MiB 2019-09-05 17:37:39.901954: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 2621440 totalling 25.00MiB 2019-09-05 17:37:39.916748: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 14 Chunks of size 2654208 totalling 35.44MiB 2019-09-05 17:37:39.928031: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 14 Chunks of size 2981888 totalling 39.81MiB 2019-09-05 17:37:39.993204: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 3145728 totalling 30.00MiB 2019-09-05 17:37:40.044704: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 3317760 totalling 25.31MiB 2019-09-05 17:37:40.055407: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 14 Chunks of size 3538944 totalling 47.25MiB 2019-09-05 17:37:40.064791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 3670016 totalling 35.00MiB 2019-09-05 17:37:40.135295: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 62 Chunks of size 3727360 totalling 220.39MiB 2019-09-05 17:37:40.151758: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 3981312 totalling 37.97MiB 2019-09-05 17:37:40.213282: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 94 Chunks of size 4194304 totalling 376.00MiB 2019-09-05 17:37:40.225309: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 4423680 totalling 33.75MiB 2019-09-05 17:37:40.236866: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 70 Chunks of size 4718592 totalling 315.00MiB 2019-09-05 17:37:40.298752: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 17 Chunks of size 6193152 totalling 100.41MiB 2019-09-05 17:37:40.314460: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 6291456 totalling 48.00MiB 2019-09-05 17:37:40.383969: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 6422528 totalling 18.38MiB 2019-09-05 17:37:40.399374: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 56 Chunks of size 8388608 totalling 448.00MiB 2019-09-05 17:37:40.409447: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 361 Chunks of size 9437184 totalling 3.17GiB 2019-09-05 17:37:40.476294: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 12582912 totalling 96.00MiB 2019-09-05 17:37:40.540315: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 12779520 totalling 97.50MiB 2019-09-05 17:37:40.552041: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 44497664 totalling 42.44MiB 2019-09-05 17:37:40.560529: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 48664576 totalling 92.82MiB 2019-09-05 17:37:40.612848: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 104000256 totalling 99.18MiB 2019-09-05 17:37:40.631611: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 8.40GiB 2019-09-05 17:37:40.644748: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 9019431321 InUse: 9019427584 MaxInUse: 9019430144 NumAllocs: 4847671 MaxAllocSize: 2706898944 2019-09-05 17:37:40.737348: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ***************************************************** *********************************************** 2019-09-05 17:37:40.813154: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at conv_ops.cc:735 : Resource exhausted: O OM when allocating tensor with shape[128,64,1,1] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "cnn_test_auto.py", line 128, in test_main() File "cnn_test_auto.py", line 119, in test_main loss,acuracy = test(data_path,generate_test, model_path) File "cnn_test_auto.py", line 76, in test loss, accuracy = my_spatial_model.evaluate_generator(generate_test, steps=test_step) #98需要才能重新确定值的大小 File "D:\python 3.6.4\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "D:\python 3.6.4\lib\site-packages\keras\engine\training.py", line 1472, in evaluate_generator verbose=verbose) File "D:\python 3.6.4\lib\site-packages\keras\engine\training_generator.py", line 346, in evaluate_generator outs = model.test_on_batch(x, y, sample_weight=sample_weight) File "D:\python 3.6.4\lib\site-packages\keras\engine\training.py", line 1256, in test_on_batch outputs = self.test_function(ins) File "D:\python 3.6.4\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "D:\python 3.6.4\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "D:\python 3.6.4\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__ run_metadata_ptr) File "D:\python 3.6.4\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[128,64,1,1] and type float on /job :localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node block2_sepconv1_1/separable_conv2d}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[{{node metrics_33/acc/Mean_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

高级语言：

项目主页：

项目地址: git://github.com/fucker007/out-of-GPU-memoery.git

创建时间： 2019-09-05T10:20:14Z
项目社区：https://github.com/fucker007/out-of-GPU-memoery
开源协议：
下载