Fork me on GitHub
Arraymancer Technical reference Tutorial Spellbook (How-To's) Under the hood

optimizers

Types

Sgd[TT] = object
  params*: seq[Variable[TT]]
  lr*: TT.T
Stochastic gradient descent   Source Edit
Adam[TT] = object
  params: seq[Variable[TT]]    ## Learnable weights
  learning_rate: TT.T
  beta1, beta2: TT.T            ## Decays on first and second moment
  beta1_t, beta2_t: TT.T        ## Current decay
  first_moments: seq[TT]       ## Exponential moving averages (mean estimation)
  second_moments: seq[TT]      ## Exponential moving averages squared (uncentered variance)
  epsilon: TT.T                ## Epsilon for numerical stability when dividing
  
Adaptative Moment Estimation   Source Edit
Optimizer[TT] = Sgd[TT] or Adam[TT]
  Source Edit

Procs

proc newSGD[T](params: varargs[Variable[Tensor[T]]]; learning_rate: T): Sgd[Tensor[T]] {...}{.
    deprecated: "Use the optimizer macro instead".}
  Source Edit
proc update(self: Sgd)
  Source Edit
proc update(self: var Adam)
  Source Edit
proc zeroGrads(o: Optimizer)
  Source Edit

Funcs

func optimizerSGD[M, T](model: M; learning_rate: T): Sgd[Tensor[T]]
Create a SGD optimizer that will update the model weight   Source Edit
func optimizerAdam[M, T](model: M; learning_rate: T = T(0.001); beta1 = T(0.9);
                       beta2 = T(0.999); eps = T(1e-08)): Adam[Tensor[T]]
Create a Adam optimizer that will update the model weight   Source Edit