Training learning rate schedule based on inputs dict with epoch number as keys

tefla.core.lr_policy.StepDecayPolicy (schedule, start_epoch=1)

Args

  • schedule: a dict, epoch number as keys and learning rate as values
  • start_epoch: training start epoch number

Methods

epoch_update (learning_rate, training_history)

Args
  • learning_rate: previous epoch learning rate
  • training_histoty: a dict with epoch, training loss, validation loss as keys
Returns

updated learning rate

Polynomial learning rate decay policy

tefla.core.lr_policy.PolyDecayPolicy (base_lr, power=10.0, max_epoch=500, n_iters_per_epoch=1094)

the effective learning rate follows a polynomial decay, to be zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)

Args

  • base_lr: a float, starting learning rate
  • power: a float, decay factor
  • max_epoch: a int, max training epoch
  • n_iters_per_epoch: number of interations per epoch, e.g. total_training_samples/batch_size

Methods

batch_update (learning_rate, iter_idx)

it follows a polynomial decay policy

Args
  • learning_rate: current batch learning rate
  • iter_idx: iteration number, e.g. number_of_iterations_per_batch*epoch+current_batch_iteration_number
Returns

updated_lr

Cosine learning rate decay policy

tefla.core.lr_policy.CosineDecayPolicy (base_lr, cycle_steps=100000, **kwargs)

the effective learning rate follows a polynomial decay, to be zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)

Args

  • base_lr: a float, starting learning rate
  • cycle_steps: a float, decay steps one cycle, learning rate decayed by inv(cycle_steps)

Methods

batch_update (learning_rate, iter_idx)

it follows a polynomial decay policy

Args
  • learning_rate: current batch learning rate
  • iter_idx: iteration number, e.g. number_of_iterations_per_batch*epoch+current_batch_iteration_number
Returns

updated_lr

Cyclic learning rate decay policy

tefla.core.lr_policy.CyclicDecayPolicy (base_lr, cycle_steps=100000, **kwargs)

the effective learning rate follows a polynomial decay, to be zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)

Args

  • base_lr: a float, starting learning rate
  • cycle_steps: a float, decay steps one cycle, learning rate decayed by inv(cycle_steps)

Methods

batch_update (learning_rate, iter_idx)

it follows a polynomial decay policy

Args
  • learning_rate: current batch learning rate
  • iter_idx: iteration number, e.g. number_of_iterations_per_batch*epoch+current_batch_iteration_number
Returns

updated_lr