RecomputeOptimizer¶

api_attr: 声明式编程(静态图)专用API

class paddle.fluid.optimizer.RecomputeOptimizer(optimizer)[源代码]¶

通常来讲，一个深度学习的训练流程包含了三个子步骤：首先，运行前向算子来计算Variable和loss的值；其次，运行反向算子来计算参数的梯度；最后，应用优化算法以更新参数值。

在前向运算过程中，反向运算会用到的Variable都会保存在内存中，当模型深度很深时，这会占用大量的内存。

重计算将深度学习网络切分为k个部分（segments）。在每个segment，运行反向运算时会首先运算前向计算。在重计算模式下，前向计算除了checkpoint和一些必须存储在内存中的特殊Variable，其他临时Variable都会被释放，这对节省内存非常有益。

把一个深度学习网络切分为k个segments的Variables被称为checkpoints。用户在使用运行RecomputeOptimizer之前需要先设置checkpoints。

参数¶

optimizer (Optimizer)-内部优化器

代码示例¶

import paddle.fluid as fluid
import numpy as np
def gen_data():
    return {"x": np.random.random(size=(32, 32)).astype('float32'),
    "y": np.random.randint(2, size=(32, 1)).astype('int64')}
def mlp(input_x, input_y, hid_dim=128, label_dim=2):
    print(input_x)
    fc_1 = fluid.layers.fc(input=input_x, size=hid_dim)
    prediction = fluid.layers.fc(input=[fc_1], size=label_dim, act='softmax')
    cost = fluid.layers.cross_entropy(input=prediction, label=input_y)
    sum_cost = fluid.layers.reduce_mean(cost)
    return sum_cost, fc_1, prediction
input_x = fluid.layers.data(name="x", shape=[32], dtype='float32')
input_y = fluid.layers.data(name="y", shape=[1], dtype='int64')
cost, fc_1, pred = mlp(input_x, input_y)

sgd = fluid.optimizer.Adam(learning_rate=0.01)
sgd = fluid.optimizer.RecomputeOptimizer(sgd)
sgd._set_checkpoints([fc_1, pred])
sgd.minimize(cost)

print("Finished optimize")
place = fluid.CPUPlace()
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
step = 10

for i in range(step):
    cost_val = exe.run(feed=gen_data(),
           program=fluid.default_main_program(),
           fetch_list=[cost.name])
    print("step=%d cost=%f" % (i, cost_val[0]))

方法¶

apply_gradients(params_grads)¶

调用self.apply_gradients

参数

params_grads (list)- 用于优化的(param, grad)对组成的列表

返回: 附加在当前Program的优化算子组成的列表
返回类型: list

代码示例

import paddle.fluid as fluid
import paddle.fluid.framework as framework

def mlp(input_x, input_y, hid_dim=128, label_dim=2):
    fc_1 = fluid.layers.fc(input=input_x, size=hid_dim)
    prediction = fluid.layers.fc(input=[fc_1], size=label_dim, act='softmax')
    cost = fluid.layers.cross_entropy(input=prediction, label=input_y)
    sum_cost = fluid.layers.reduce_mean(cost)
    return sum_cost, fc_1, prediction


input_x = fluid.layers.data(name="x", shape=[32], dtype='float32')
input_y = fluid.layers.data(name="y", shape=[1], dtype='int64')
cost, fc_1, pred = mlp(input_x, input_y)
print("Finished FF")

sgd = fluid.optimizer.Adam(learning_rate=0.01)
sgd = fluid.optimizer.RecomputeOptimizer(sgd)
params_grads = sgd.backward(
    cost,
    startup_program=None,
    parameter_list=None,
    no_grad_set=None)

program = cost.block.program
with framework.program_guard(program, None):
    optimize_ops = sgd.apply_gradients(params_grads)

print("Finished apply gradients")

apply_optimize(loss, startup_program, params_grads)¶

调用self._optimizer的apply_optimize函数

参数

loss (Variable) – 用于优化过程的损失值变量

startup_program (Program) – 用于初始化在parameter_list中参数的startup_program

params_grads (list)- 用于优化的(param, grad)对组成的列表

返回: 附加在当前Program的算子组成的列表
返回类型: list

代码示例

import paddle.fluid as fluid

def mlp(input_x, input_y, hid_dim=128, label_dim=2):
    fc_1 = fluid.layers.fc(input=input_x, size=hid_dim)
    prediction = fluid.layers.fc(input=[fc_1], size=label_dim, act='softmax')
    cost = fluid.layers.cross_entropy(input=prediction, label=input_y)
    sum_cost = fluid.layers.reduce_mean(cost)
    return sum_cost, fc_1, prediction

input_x = fluid.layers.data(name="x", shape=[32], dtype='float32')
input_y = fluid.layers.data(name="y", shape=[1], dtype='int64')
cost, fc_1, pred = mlp(input_x, input_y)
print("Finished FF")

sgd = fluid.optimizer.Adam(learning_rate=0.01)
sgd = fluid.optimizer.RecomputeOptimizer(sgd)
params_grads = sgd.backward(
    cost,
    startup_program=None,
    parameter_list=None,
    no_grad_set=None)

optimize_ops = sgd.apply_optimize(
    cost, startup_program=None, params_grads=params_grads)

print("Finished apply_optimize")

backward(loss, startup_program=None, parameter_list=None, no_grad_set=None, callbacks=None)¶

带checkpoint的backward函数

参数

loss (Variable) – 需要最小化的损失值变量

startup_program (Program, 可选) – 用于初始化parameter_list中参数的 Program , 默认值为None，此时将使用 default_startup_program

parameter_list (list, 可选) – 待更新的Parameter或者Parameter.name组成的列表，默认值为None，此时将更新所有的Parameter

no_grad_set (set, 可选) – 不需要更新的Parameter或者Parameter.name组成的的集合，默认值为None

callbacks (list, 可选) – 当为某参数附加反向算子时所要运行的callables组成的列表

checkpoints (list, 可选) – 一批作为checkpoints的Variables

返回: 由(param, grad)对构成的列表，其中param是参数，grad是其对应的梯度
返回类型: list

代码示例

import paddle.fluid as fluid

def mlp(input_x, input_y, hid_dim=128, label_dim=2):
    fc_1 = fluid.layers.fc(input=input_x, size=hid_dim)
    prediction = fluid.layers.fc(input=[fc_1], size=label_dim, act='softmax')
    cost = fluid.layers.cross_entropy(input=prediction, label=input_y)
    sum_cost = fluid.layers.reduce_mean(cost)
    return sum_cost, fc_1, prediction


input_x = fluid.layers.data(name="x", shape=[32], dtype='float32')
input_y = fluid.layers.data(name="y", shape=[1], dtype='int64')
cost, fc_1, pred = mlp(input_x, input_y)
print("Finished FF")

sgd = fluid.optimizer.Adam(learning_rate=0.01)
sgd = fluid.optimizer.RecomputeOptimizer(sgd)
params_grads = sgd.backward(
    cost,
    startup_program=None,
    parameter_list=None,
    no_grad_set=None)
print("Finished backward")