之前用的tensorflow_cpu版训练太慢,换成gpu后训练直接结束进程了是什么情况
model.fit(
x=train_dataloader,
steps_per_epoch=epoch_step,
validation_data=val_dataloader,
validation_steps=epoch_step_val,
epochs=end_epoch,
initial_epoch=start_epoch,
use_multiprocessing=True if num_workers > 1 else False,
workers=num_workers,
callbacks = callbacks
)
GeForce RTX 3050 Ti Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
Epoch 1: LearningRateScheduler setting learning rate to 2.9999999999999997e-05.
Epoch 1/50
C:\Users\97394\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model_1/yolo_loss/while/gradients/model_1/yolo_loss/while/cond_grad/gradients/model_1/yolo_loss/while/cond/GatherNd_2_grad/Squeeze:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model_1/yolo_loss/while/gradients/model_1/yolo_loss/while/cond_grad/gradients/grad_ys_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model_1/yolo_loss/while/gradients/model_1/yolo_loss/while/cond_grad/gradients/model_1/yolo_loss/while/cond/GatherNd_2_grad/Shape:0", shape=(2,), dtype=int64))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
2022-12-08 19:25:08.232598: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8101
进程已结束,退出代码-1073740791 (0xC0000409)
会是gpu版本的cuda跟cudnn的问题吗,但我重装了好几遍了,也可能是内存溢出的情况
GeForce RTX 3050 Ti Laptop GPU这块显卡的显存只有4G,做深度学习似乎有点太小了,你可以把batch_size设置为1,再看看能不能走通