为什么loss和acc陡然下降 如何调整为宜？(深度学习 影像分割 分割 二值分类 TensorFlow keras unet )

问题遇到的现象和发生背景

深度学习 影像图斑分割 二分类
使用框架TensorFlow keras unet
显卡P5000
初始遥感影像样本旋转了3次 裁切得到训练及验证样本集 验证样本随机获取

问题相关代码，请勿粘贴截图

batch_size =8
input_size=256,256,3
epochs=260
learning_rate=5e-5
train_num=10456
validation_num=1306

用于配置训练模型（优化器、目标函数、模型评估标准）

model.compile(optimizer = Adam(lr = learning_rate), loss = 'categorical_crossentropy', metrics = ['accuracy'])

运行结果及报错内容

2022-08-16 20:43:33.834002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2022-08-16 20:43:34.821246: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
2022-08-16 20:43:34.871322: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
1307/1307 [ ] - 1045s 800ms/step - loss: 0.6964 - accuracy: 0.8024 - val_loss: 0.6732 - val_accuracy: 0.8048

Epoch 00001: loss improved from inf to 0.69645, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 2/260
1307/1307 [ ] - 1047s 801ms/step - loss: 0.6232 - accuracy: 0.9028 - val_loss: 0.6156 - val_accuracy: 0.8981

Epoch 00002: loss improved from 0.69645 to 0.62322, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 3/260
1307/1307 [ ] - 1024s 784ms/step - loss: 0.5814 - accuracy: 0.9059 - val_loss: 0.5668 - val_accuracy: 0.9157

Epoch 00003: loss improved from 0.62322 to 0.58136, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 4/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.5437 - accuracy: 0.9053 - val_loss: 0.5916 - val_accuracy: 0.8800

Epoch 00004: loss improved from 0.58136 to 0.54374, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 5/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.5069 - accuracy: 0.9104 - val_loss: 0.5355 - val_accuracy: 0.9004

Epoch 00005: loss improved from 0.54374 to 0.50688, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 6/260
1307/1307 [ ] - 959s 734ms/step - loss: 0.4756 - accuracy: 0.9103 - val_loss: 0.4023 - val_accuracy: 0.9107

Epoch 00006: loss improved from 0.50688 to 0.47559, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 7/260
1307/1307 [ ] - 960s 735ms/step - loss: 0.4449 - accuracy: 0.9140 - val_loss: 0.4558 - val_accuracy: 0.8947

Epoch 00007: loss improved from 0.47559 to 0.44491, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 8/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.4195 - accuracy: 0.9141 - val_loss: 0.3656 - val_accuracy: 0.8967

Epoch 00008: loss improved from 0.44491 to 0.41947, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 9/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.3982 - accuracy: 0.9130 - val_loss: 0.3337 - val_accuracy: 0.9070

Epoch 00009: loss improved from 0.41947 to 0.39823, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 10/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.3777 - accuracy: 0.9132 - val_loss: 0.3189 - val_accuracy: 0.8995

Epoch 00010: loss improved from 0.39823 to 0.37770, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 11/260
1307/1307 [ ] - 960s 735ms/step - loss: 0.3639 - accuracy: 0.9138 - val_loss: 0.2769 - val_accuracy: 0.9046

Epoch 00011: loss improved from 0.37770 to 0.36389, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 12/260
1307/1307 [ ] - 960s 735ms/step - loss: 0.3607 - accuracy: 0.9160 - val_loss: 0.3507 - val_accuracy: 0.9007

Epoch 00012: loss improved from 0.36389 to 0.36071, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 13/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.3594 - accuracy: 0.9172 - val_loss: 0.2733 - val_accuracy: 0.8945

Epoch 00013: loss improved from 0.36071 to 0.35935, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 14/260
1307/1307 [ ] - 960s 734ms/step - loss: 0.3529 - accuracy: 0.9225 - val_loss: 0.2974 - val_accuracy: 0.8967

Epoch 00014: loss improved from 0.35935 to 0.35292, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 15/260
1307/1307 [ ] - 960s 735ms/step - loss: 0.3586 - accuracy: 0.9162 - val_loss: 0.2786 - val_accuracy: 0.9005

Epoch 00015: loss did not improve from 0.35292
Epoch 16/260
1307/1307 [ ] - 959s 734ms/step - loss: 0.3506 - accuracy: 0.9223 - val_loss: 0.3221 - val_accuracy: 0.9024

Epoch 00016: loss improved from 0.35292 to 0.35065, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 17/260
1307/1307 [ ] - 959s 734ms/step - loss: 0.3480 - accuracy: 0.9220 - val_loss: 0.4305 - val_accuracy: 0.8949

Epoch 00017: loss improved from 0.35065 to 0.34804, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 18/260
1307/1307 [ ] - 956s 731ms/step - loss: 0.3792 - accuracy: 0.8988 - val_loss: 0.8560 - val_accuracy: 0.1180

Epoch 00018: loss did not improve from 0.34804
Epoch 19/260
1307/1307 [ ] - 958s 733ms/step - loss: 0.3761 - accuracy: 0.8980 - val_loss: 0.7498 - val_accuracy: 0.1046

Epoch 00019: loss did not improve from 0.34804
Epoch 20/260
1307/1307 [ ] - 957s 732ms/step - loss: 0.3722 - accuracy: 0.8960 - val_loss: 0.7169 - val_accuracy: 0.1014

Epoch 00020: loss did not improve from 0.34804
Epoch 21/260
1307/1307 [ ] - 958s 733ms/step - loss: 0.3674 - accuracy: 0.8944 - val_loss: 0.6812 - val_accuracy: 0.8971

Epoch 00021: loss did not improve from 0.34804
Epoch 22/260
1307/1307 [ ] - 958s 733ms/step - loss: 0.3503 - accuracy: 0.9013 - val_loss: 0.6408 - val_accuracy: 0.8844

Epoch 00022: loss did not improve from 0.34804
Epoch 23/260
1307/1307 [ ] - 958s 733ms/step - loss: 0.0431 - accuracy: 0.1980 - val_loss: 1.1921e-07 - val_accuracy: 0.0978

Epoch 00023: loss improved from 0.34804 to 0.04306, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 24/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1007 - val_loss: 1.1921e-07 - val_accuracy: 0.1002

Epoch 00024: loss improved from 0.04306 to 0.00000, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 25/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1008 - val_loss: 1.1921e-07 - val_accuracy: 0.0957

Epoch 00025: loss improved from 0.00000 to 0.00000, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 26/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0996 - val_loss: 1.1921e-07 - val_accuracy: 0.1071

Epoch 00026: loss did not improve from 0.00000
Epoch 27/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1041 - val_loss: 1.1921e-07 - val_accuracy: 0.1075

Epoch 00027: loss did not improve from 0.00000
Epoch 28/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0996 - val_loss: 1.1921e-07 - val_accuracy: 0.0995

Epoch 00028: loss did not improve from 0.00000
Epoch 29/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0992 - val_loss: 1.1921e-07 - val_accuracy: 0.0937

Epoch 00029: loss did not improve from 0.00000
Epoch 30/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0985 - val_loss: 1.1921e-07 - val_accuracy: 0.1015

Epoch 00030: loss did not improve from 0.00000
Epoch 31/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1025 - val_loss: 1.1921e-07 - val_accuracy: 0.0973

Epoch 00031: loss did not improve from 0.00000
Epoch 32/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0972 - val_loss: 1.1921e-07 - val_accuracy: 0.1082

Epoch 00032: loss did not improve from 0.00000
Epoch 33/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0989 - val_loss: 1.1921e-07 - val_accuracy: 0.0937

Epoch 00033: loss did not improve from 0.00000
Epoch 34/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0995 - val_loss: 1.1921e-07 - val_accuracy: 0.0986

Epoch 00034: loss did not improve from 0.00000
Epoch 35/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0995 - val_loss: 1.1921e-07 - val_accuracy: 0.1008

Epoch 00035: loss improved from 0.00000 to 0.00000, saving model to F:\Data\QQCT2025\RSICPRJ\model\model.hdf5
Epoch 36/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1017 - val_loss: 1.1921e-07 - val_accuracy: 0.0910

Epoch 00036: loss did not improve from 0.00000
Epoch 37/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1003 - val_loss: 1.1921e-07 - val_accuracy: 0.0993

Epoch 00037: loss did not improve from 0.00000
Epoch 38/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1041 - val_loss: 1.1921e-07 - val_accuracy: 0.0965

Epoch 00038: loss did not improve from 0.00000
Epoch 39/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1024 - val_loss: 1.1921e-07 - val_accuracy: 0.0895

Epoch 00039: loss did not improve from 0.00000
Epoch 40/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.0992 - val_loss: 1.1921e-07 - val_accuracy: 0.0962

Epoch 00040: loss did not improve from 0.00000
Epoch 41/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1041 - val_loss: 1.1921e-07 - val_accuracy: 0.0978

Epoch 00041: loss did not improve from 0.00000
Epoch 42/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1008 - val_loss: 1.1921e-07 - val_accuracy: 0.0985

Epoch 00042: loss did not improve from 0.00000
Epoch 43/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1034 - val_loss: 1.1921e-07 - val_accuracy: 0.1030

Epoch 00043: loss did not improve from 0.00000
Epoch 44/260
1307/1307 [ ] - 958s 733ms/step - loss: 1.1921e-07 - accuracy: 0.1032 - val_loss: 1.1921e-07 - val_accuracy: 0.1070

Epoch 00044: loss did not improve from 0.00000
Epoch 45/260
653/1307 [=>] - ETA: 1:17:37 - loss: 1.1921e-07 - accuracy: 0.1055

我的解答思路和尝试过的方法

深度学习半路出家 初次开展这么多样本的训练
昨天 开始训练时就出现过acc较低的情况 开始几轮0.1的acc 重启了几次 把lr从2e-5调整为5e-5
训练acc前几轮达到0.8左右 之后继续训练
今天发现 在18轮出现loss和acc陡然下降 之后 acc维持在0.1左右 才训练至45轮

我想要达到的结果

请问loss和acc陡然下降是什么原因？
怎样调整或检查排除才能让loss 和acc正常 曲线拟合至最佳状态？
就遥感影像地类要素分类来说 基础样本旋转3次得到10000左右训练样本是否可用 或还有哪些增强方式可用？

  1. 数据增强使用CutMix和Cutout，其中CutMix就是将一部分区域cut掉但不填充0像素而是随机填充训练集中的其他数据的区域像素值，分类结果按一定的比例分配；Cutout:随机的将样本中的部分区域cut掉，并且填充0像素值，分类的结果不变
  2. 激活函数可尝试使用swish或者mish
  3. 训练trick可使用学习率预热+学习率余弦衰减，建议先试预热，然后试两者合起来。
    学习率预热的代码为：
# 在10个epoch学习率从5e-8逐渐提升为5e-7
from tensorflow.keras.callbacks import LearningRateScheduler

lr_schedule = [10]

def schedule(epoch_idx):
    if epoch_idx  < lr_schedule[0]:
        return 5e-7 / 10 * (epoch_idx+1)
   
    return 5e-7

scheduler = LearningRateScheduler(schedule=schedule)

model.fit(X_train, X_train_label,
                  validation_data=(X_val, X_val_label),
                  epochs=100, batch_size=64,
                  shuffle=True,
                  callbacks=[scheduler] #回调代码

两者合起来的代码为：

from tensorflow.keras.callbacks import LearningRateScheduler

lr_schedule = [10]

def schedule(epoch_idx):
    if epoch_idx < lr_schedule[0]:
        return 5e-7 / 10 * (epoch_idx+1)
    else:
        t = (epoch_idx - 10) * math.pi / 100 
        return  1/2 * (1 + math.cos(t)) * 0.1

scheduler = LearningRateScheduler(schedule=schedule)

下降的原因：1.数据集存在问题。2.优化器参数设置问题。
处理办法：1.数据问题的话需要分析数据。2.参数问题的话，建议使用冻结训练，以及模型预热。
增强方式：1.图像的几何变换，翻转,旋转,平移等，在transform函数中有很多。2.cutout,cutmix,mixup等等数据增强的方法。3.labelsmothing标签平滑。4.dropout模块等等。
具体可以私信一起研究。

1、首先，批次大小过小，batch size抽样训练的数据过小会导致模型学习的特征并不能代表整体，可能会导致你每个批次的内容差别过大，使精度和损失高数震荡，一般调成32 64 128等
2、学习率不合适，学习率过大会导致模型精确度来回震荡无法收敛，学习率过小会导致精确度处于一个局部最优点，无法达到最终的收敛
3、建议使用keras中的迁移模型，有更深的网络层数可以挖掘到更深的特征，会具有更高的精确度以及更稳定的效果
4、数据本身有问题，是否有做数据清洗，标准归一化处理？
5、可能是神经网络选型的问题，例如bp等梯度型神经网络，处理多极点的模型，容易进入局部最小点

建议使用迁移模型

Q1：可能是优化的时候突然越过了局部最优点，建议学习率进行优化，比如采用动态学习率或者早停。
Q2：旋转算是数据增强里面最常用的了，常见的还有翻转（上下翻转，水平翻转）、缩放、平移、裁剪，或添加高斯噪声。

从问题描述来看，
loss下降对应的是训练集，
acc对应的是测试集，
不管设置到多小的学习率，到达一定的acc后，就突然下降。
提供以下几种尝试方法：
1、下降之后会不会再次上升到之前的acc附近，如果是，看第2步。
2、查看数据读取的顺序是否一成不变的，一般会每次打乱顺序，让读取的批次数据样式不固定；
3、做一下数据增强，
4、在模型中添加注意力机制，
5、降低模型复杂度。

这种可能性有很多，程序问题、数据问题等都有可能，如果真是刚入行的话，建议改用简单的、确定的数据进行训练，比如黑白棋、五子棋等棋局数据

建议使用pytorch

从三大因素考虑:数据、算法、算力。
数据方面:可能存在低质量的数据，建议对数据进一步预处理
算法方面:可能受学习率、批大小、迭代次数、模型其他参数影响，可以通过tensorboard找到超参数，再进行训练
算力方面:这里影响不大
（抛砖引玉，祝你成功）

在原数据集上也有这个情况么，可以试试