KGE.loss

Classes

BinaryCrossEntropyLoss	An implementation of Binary Cross Entropy Loss.
Loss	A base module for loss.
PairwiseHingeLoss	An implementation of Pairwise Hinge Loss / Margin Ranking Loss.
PairwiseLogisticLoss	An implementation of Pairwise Logistic Loss.
SelfAdversarialNegativeSamplingLoss	An implementation of Self Adversarial Negative Sampling Loss.
SquareErrorLoss	An implementation of Square Error Loss.

Class Inheritance Diagram

Different loss functions that you can choose when training Knowledge Graph Embedding Model.

Different Knowledge Graph Embedding Models use different loss functions, the default setting for each KGE model is according to the original paper described, for example, TransE using Pairwise Hinge Loss, RotatE using Self Adversarial Negative Sampling Loss.

You can change the loss function to try any possibility in a very easy way:

from KGE.models.translating_based.TransE import TransE
from KGE.loss import SelfAdversarialNegativeSamplingLoss

model = TransE(
    embedding_params={"embedding_size": 10},
    negative_ratio=10,
    corrupt_side="h+t",
    loss_fn=SelfAdversarialNegativeSamplingLoss(margin=3, temperature=1) # specifying loss function you want
)

class KGE.loss.BinaryCrossEntropyLoss

Bases: KGE.loss.Loss

An implementation of Binary Cross Entropy Loss.

Binary Cross Entropy Loss is commonly used in binary classification problem. In KGE, we can also turn the problem into a binary classification problem that classifies triplet into positive or negative \(y_i = 1~or~0\) with the triplet score as logit: \(logit_i = f\left( (h,r,t)_i \right)\)

\[ \begin{align}\begin{aligned}\begin{aligned} \mathscr{L} &= - \sum_i y_i log(\hat{y}_i) + (1-y_i) log(1-\hat{y}_i)\\ &= - \sum_i log\left[\sigma(f((h,r,t)_i^+))\right] - \sum_i log\left[1-\sigma(f((h,r,t)_i^-))\right]\\ &= - \sum_i log\left[\sigma(f((h,r,t)_i^+))\right] - \sum_i log\left[\sigma(-f((h,r,t)_i^-))\right] \end{aligned}\end{aligned}\end{align} \]

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__()

Initialize loss

class KGE.loss.Loss

Bases: object

A base module for loss.

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__()

Initialize loss

class KGE.loss.PairwiseHingeLoss

Bases: KGE.loss.Loss

An implementation of Pairwise Hinge Loss / Margin Ranking Loss.

Pairwise Hinge Loss or Margin Ranking Loss is a common loss function that used in many models such as UM, SE, TransE, TransH, TransR, TransD, DistMult.

For each pair of postive triplet \((h,r,t)_i^+\) and negative triplet \((h,r,t)_i^-\), Pairwise Hinge Loss compare the difference of scores between postivie triplet and negative triplet:

\[\Delta_i = f\left( (h,r,t)_i^- \right) - f\left( (h,r,t)_i^+ \right)\]

Since the socre of triplet \(f(h,r,t)\) measures how plausible \((h,r,t)\) is, so \(\Delta_i < 0\) is favorable. If the difference \(\Delta_i\) does not execeed the given margin \(\gamma\), Pairwise Hinge Loss penalize this pair:

\[\mathscr{L} = \sum_i max \left( 0, \gamma + \Delta_i \right)\]

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__(margin)

Initialize loss

class KGE.loss.PairwiseLogisticLoss

Bases: KGE.loss.Loss

An implementation of Pairwise Logistic Loss.

Described in Loss Functions in Knowledge Graph Embedding Models.

For each pair of postive triplet \((h,r,t)_i^+\) and negative triplet \((h,r,t)_i^-\), Pairwise Logistic Loss compare the difference of scores between postivie triplet and negative triplet:

\[\Delta_i = f\left( (h,r,t)_i^- \right) - f\left( (h,r,t)_i^+ \right)\]

and define the Pairwise Logistic Loss as:

\[\mathscr{L} = \sum_i log(1+exp(\Delta_i))\]

Pairwise Logistic Loss is a smooth version of Pairwise Hinge Loss while \(\gamma = 0\), you can view function graph here to campare these two functions.

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__()

Initialize loss

class KGE.loss.SelfAdversarialNegativeSamplingLoss

Bases: KGE.loss.Loss

An implementation of Self Adversarial Negative Sampling Loss.

Described in RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.

Self Adversarial Negative Sampling Loss samples negative triples according to the current embedding model. Specifically, it sample negative triples from the following distribution:

\[p\left((h,r,t)_{i,j}^- \vert (h,r,t)_i^+ \right) = \frac{exp~ \alpha f((h,r,t)_{i.j}^-)}{\sum_k exp~ \alpha f((h,r,t)_{i,k}^-)}\]

where \((h,r,t)_i^+\) denotes i-th positive triplet, \((h,r,t)_{i,j}^-\) denotes j-th negative triplet generate from i-th positive triplet, \(\alpha\) is the temperature of sampling.

Since the sampling procedure may be costly, Self Adversarial Negative Sampling Loss treats the above probability as the weight of the negative sample. Therefore, the final negative sampling loss with self-adversarial training takes the following form:

\[\mathscr{L} = - \sum_i log~ \sigma(\gamma + f((h,r,t)_i^+)) - \sum_i \sum_j p\left( (h,r,t)_{i,j}^- \right) log~ \sigma(-\gamma - f((h,r,t)_{i,j}^-))\]

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__(margin, temperature)

Initialize loss

class KGE.loss.SquareErrorLoss

Bases: KGE.loss.Loss

An implementation of Square Error Loss.

Square Error Loss is a loss function used in RESCAL, it computes the squared difference between triplet scores \(f((h,r,t)_i)\) and labels (\(y_i = 1~or~0\)):

\[\mathscr{L} = \sum_i \left( f((h,r,t)_i) - y_i \right)^2\]

Methods

__call__(pos_score, neg_score)

Calculate loss.

__init__()

Initialize loss