Distance-based Losses

Loss functions that belong to the category "distance-based" are primarily used in regression problems. They utilize the numeric difference between the predicted output and the true target as a proxy variable to quantify the quality of individual predictions.

This section lists all the subtypes of DistanceLoss that are implemented in this package.

LPDistLoss

LossFunctions.LPDistLoss — Type.

LPDistLoss{P} <: DistanceLoss

The P-th power absolute distance loss. It is Lipschitz continuous iff P == 1, convex if and only if P >= 1, and strictly convex iff P > 1.

\[L(r) = |r|^P\]

source

Lossfunction	Derivative

$L(r) = \mid r \mid ^p$	$L'(r) = p \cdot r \cdot \mid r \mid ^{p-2}$

L1DistLoss

LossFunctions.L1DistLoss — Type.

L1DistLoss <: DistanceLoss

The absolute distance loss. Special case of the LPDistLoss with P=1. It is Lipschitz continuous and convex, but not strictly convex.

\[L(r) = |r|\]

              Lossfunction                     Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    3 │\.                     ./│    1 │            ┌------------│
      │ '\.                 ./' │      │            |            │
      │   \.               ./   │      │            |            │
      │    '\.           ./'    │      │_           |           _│
    L │      \.         ./      │   L' │            |            │
      │       '\.     ./'       │      │            |            │
      │         \.   ./         │      │            |            │
    0 │          '\./'          │   -1 │------------┘            │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -3                        3
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \mid r \mid$	$L'(r) = \textrm{sign}(r)$

L2DistLoss

LossFunctions.L2DistLoss — Type.

L2DistLoss <: DistanceLoss

The least squares loss. Special case of the LPDistLoss with P=2. It is strictly convex.

\[L(r) = |r|^2\]

              Lossfunction                     Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    9 │\                       /│    3 │                   .r/   │
      │".                     ."│      │                 .r'     │
      │ ".                   ." │      │              _./'       │
      │  ".                 ."  │      │_           .r/         _│
    L │   ".               ."   │   L' │         _:/'            │
      │    '\.           ./'    │      │       .r'               │
      │      \.         ./      │      │     .r'                 │
    0 │        "-.___.-"        │   -3 │  _/r'                   │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -2                        2
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \mid r \mid ^2$	$L'(r) = 2 r$

LogitDistLoss

LossFunctions.LogitDistLoss — Type.

LogitDistLoss <: DistanceLoss

The distance-based logistic loss for regression. It is strictly convex and Lipschitz continuous.

\[L(r) = - \ln \frac{4 e^r}{(1 + e^r)^2}\]

              Lossfunction                     Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    2 │                         │    1 │                   _--'''│
      │\                       /│      │                ./'      │
      │ \.                   ./ │      │              ./         │
      │  '.                 .'  │      │_           ./          _│
    L │   '.               .'   │   L' │           ./            │
      │     \.           ./     │      │         ./              │
      │      '.         .'      │      │       ./                │
    0 │        '-.___.-'        │   -1 │___.-''                  │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -4                        4
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = - \ln \frac{4 e^r}{(1 + e^r)^2}$	$L'(r) = \tanh \left( \frac{r}{2} \right)$

HuberLoss

LossFunctions.HuberLoss — Type.

HuberLoss <: DistanceLoss

Loss function commonly used for robustness to outliers. For large values of d it becomes close to the L1DistLoss, while for small values of d it resembles the L2DistLoss. It is Lipschitz continuous and convex, but not strictly convex.

\[L(r) = \begin{cases} \frac{r^2}{2} & \quad \text{if } | r | \le \alpha \\ \alpha | r | - \frac{\alpha^3}{2} & \quad \text{otherwise}\\ \end{cases}\]

              Lossfunction (d=1)               Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    2 │                         │    1 │                .+-------│
      │                         │      │              ./'        │
      │\.                     ./│      │             ./          │
      │ '.                   .' │      │_           ./          _│
    L │   \.               ./   │   L' │           /'            │
      │     \.           ./     │      │          /'             │
      │      '.         .'      │      │        ./'              │
    0 │        '-.___.-'        │   -1 │-------+'                │
      └────────────┴────────────┘      └────────────┴────────────┘
      -2                        2      -2                        2
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \begin{cases} \frac{r^2}{2} & \quad \text{if } \mid r \mid \le \alpha \\ \alpha \mid r \mid - \frac{\alpha^2}{2} & \quad \text{otherwise}\\ \end{cases}$	$L'(r) = \begin{cases} r & \quad \text{if } \mid r \mid \le \alpha \\ \alpha \cdot \textrm{sign}(r) & \quad \text{otherwise}\\ \end{cases}$

L1EpsilonInsLoss

LossFunctions.L1EpsilonInsLoss — Type.

L1EpsilonInsLoss <: DistanceLoss

The $ϵ$-insensitive loss. Typically used in linear support vector regression. It ignores deviances smaller than $ϵ$, but penalizes larger deviances linarily. It is Lipschitz continuous and convex, but not strictly convex.

\[L(r) = \max \{ 0, | r | - \epsilon \}\]

              Lossfunction (ϵ=1)               Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    2 │\                       /│    1 │                  ┌------│
      │ \                     / │      │                  |      │
      │  \                   /  │      │                  |      │
      │   \                 /   │      │_      ___________!     _│
    L │    \               /    │   L' │      |                  │
      │     \             /     │      │      |                  │
      │      \           /      │      │      |                  │
    0 │       \_________/       │   -1 │------┘                  │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -2                        2
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \max \{ 0, \mid r \mid - \epsilon \}$	$L'(r) = \begin{cases} \frac{r}{ \mid r \mid } & \quad \text{if } \epsilon \le \mid r \mid \\ 0 & \quad \text{otherwise}\\ \end{cases}$

L2EpsilonInsLoss

LossFunctions.L2EpsilonInsLoss — Type.

L2EpsilonInsLoss <: DistanceLoss

The quadratic $ϵ$-insensitive loss. Typically used in linear support vector regression. It ignores deviances smaller than $ϵ$, but penalizes larger deviances quadratically. It is convex, but not strictly convex.

\[L(r) = \max \{ 0, | r | - \epsilon \}^2\]

              Lossfunction (ϵ=0.5)             Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    8 │                         │    1 │                  /      │
      │:                       :│      │                 /       │
      │'.                     .'│      │                /        │
      │ \.                   ./ │      │_         _____/        _│
    L │  \.                 ./  │   L' │         /               │
      │   \.               ./   │      │        /                │
      │    '\.           ./'    │      │       /                 │
    0 │      '-._______.-'      │   -1 │      /                  │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -2                        2
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \max \{ 0, \mid r \mid - \epsilon \}^2$	$L'(r) = \begin{cases} 2 \cdot \textrm{sign}(r) \cdot \left( \mid r \mid - \epsilon \right) & \quad \text{if } \epsilon \le \mid r \mid \\ 0 & \quad \text{otherwise}\\ \end{cases}$

PeriodicLoss

LossFunctions.PeriodicLoss — Type.

PeriodicLoss <: DistanceLoss

Measures distance on a circle of specified circumference c.

\[L(r) = 1 - \cos \left( \frac{2 r \pi}{c} \right)\]

source

Lossfunction	Derivative

$L(r) = 1 - \cos \left ( \frac{2 r \pi}{c} \right )$	$L'(r) = \frac{2 \pi}{c} \cdot \sin \left( \frac{2r \pi}{c} \right)$

QuantileLoss

LossFunctions.QuantileLoss — Type.

QuantileLoss <: DistanceLoss

The distance-based quantile loss, also known as pinball loss, can be used to estimate conditional τ-quantiles. It is Lipschitz continuous and convex, but not strictly convex. Furthermore it is symmetric if and only if τ = 1/2.

\[L(r) = \begin{cases} -\left( 1 - \tau \right) r & \quad \text{if } r < 0 \\ \tau r & \quad \text{if } r \ge 0 \\ \end{cases}\]

              Lossfunction (τ=0.7)             Derivative
      ┌────────────┬────────────┐      ┌────────────┬────────────┐
    2 │'\                       │  0.3 │            ┌------------│
      │  \.                     │      │            |            │
      │   '\                    │      │_           |           _│
      │     \.                  │      │            |            │
    L │      '\              ._-│   L' │            |            │
      │        \.         ..-'  │      │            |            │
      │         '.     _r/'     │      │            |            │
    0 │           '_./'         │ -0.7 │------------┘            │
      └────────────┴────────────┘      └────────────┴────────────┘
      -3                        3      -3                        3
                 ŷ - y                            ŷ - y

source

Lossfunction	Derivative

$L(r) = \begin{cases} \left( 1 - \tau \right) r & \quad \text{if } r \ge 0 \\ - \tau r & \quad \text{otherwise} \\ \end{cases}$	$L(r) = \begin{cases} 1 - \tau & \quad \text{if } r \ge 0 \\ - \tau & \quad \text{otherwise} \\ \end{cases}$

Note

You may note that our definition of the QuantileLoss looks different to what one usually sees in other literature. The reason is that we have to correct for the fact that in our case $r = \hat{y} - y$ instead of $r_{\textrm{usual}} = y - \hat{y}$, which means that our definition relates to that in the manner of $r = -1 * r_{\textrm{usual}}$.