Shannon Measures¶
Unified API functions that dispatch to discrete or continuous implementations based on the discrete parameter.
Shorthand Aliases¶
These are the shortest way to access the core measures:
from divergence import entropy, kl_divergence, mutual_information
h = entropy(samples)
kl = kl_divergence(p, q)
mi = mutual_information(x, y, discrete=True)
All aliases accept the same base and discrete parameters as their full-name counterparts.
Unified API¶
entropy_from_samples(sample, base=np.e, discrete=False)
¶
cross_entropy_from_samples(sample_p, sample_q, base=np.e, discrete=False)
¶
relative_entropy_from_samples(sample_p, sample_q, base=np.e, discrete=False)
¶
jensen_shannon_divergence_from_samples(sample_p, sample_q, base=np.e, discrete=False)
¶
mutual_information_from_samples(sample_x, sample_y, base=np.e, discrete=False)
¶
joint_entropy_from_samples(sample_x, sample_y, base=np.e, discrete=False)
¶
conditional_entropy_from_samples(sample_x, sample_y, base=np.e, discrete=False)
¶
Discrete¶
discrete_entropy(sample, base=np.e)
¶
Approximate the entropy of a discrete distribution
H(p) = - E_p[log(p)]
from a sample.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
An approximation of the entropy of the discrete distribution from which the sample is drawn.
|
|
discrete_cross_entropy(sample_p, sample_q, base=np.e)
¶
Approximate the cross entropy of the discrete distribution q relative to the discrete distribution p
H_q(p) = - E_p [log(q)]
from samples of these distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The cross entropy of the distribution q relative to the distribution p.
|
|
discrete_relative_entropy(sample_p, sample_q, base=np.e)
¶
Approximate the relative entropy of the discrete distribution q relative to the discrete distribution p
D_KL(p||q) = E_p [log(p/q)]
from samples of these distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The relative entropy of the distribution q relative to the distribution p.
|
|
discrete_jensen_shannon_divergence(sample_p, sample_q, base=np.e)
¶
Approximate the Jensen-Shannon divergence between discrete distributions p and q
JSD(p||q) = 0.5 * (D_KL(p||m) + D_KL(q||m)), with m = 0.5 * (p + q)
from samples of these distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The Jensen-Shannon divergence between distributions p and q.
|
|
discrete_mutual_information(sample_x, sample_y, base=np.e)
¶
Approximate the mutual information of x and y
I(X; Y) = D_KL(p_{x, y}|| p_x \otimes p_y) =
E_{p_{x, y}} \left[ \log \left( \frac{p_{x, y} (x, y)}{p_x(x) p_y(y)} \right) \right]
from a sample of both distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The mutual information of x and y.
|
|
discrete_joint_entropy(sample_x, sample_y, base=np.e)
¶
Approximate the joint entropy of x and y
H(X, Y) = - E_{p_{x, y}} \left[ \log p_{x, y} (x, y) \right]
from a sample of both distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The joint entropy between of x and y
|
|
discrete_conditional_entropy_of_y_given_x(sample_x, sample_y, base=np.e)
¶
Approximate the conditional entropy of y given x
H(Y|X) = - E_{p_{x, y}} \left[ \log \frac{p_{x, y} (x, y)}{p_x(x)} \right]
from a sample of both distributions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The conditional entropy between of y given x.
|
|
Continuous (sample-based)¶
continuous_entropy_from_sample(sample, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the entropy
H(p) = - E_p[log(p)]
of a sample via approximation by a kernel density estimate and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The entropy of the density approximated by the sample
|
|
continuous_cross_entropy_from_sample(sample_p, sample_q, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the cross entropy of the distribution q relative to the distribution p
H_q(p) = - E_p [log(q)]
from samples of the two distributions via approximation by a kernel density estimate and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The cross entropy of the distribution q relative to the distribution p.
|
|
continuous_relative_entropy_from_sample(sample_p, sample_q, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the relative entropy of the distribution q relative to the distribution p
D_KL(p||q) = E_p [log(p/q)]
from samples of the two distributions via approximation by a kernel density estimate and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The relative entropy of the distribution q relative to the distribution p.
|
|
continuous_jensen_shannon_divergence_from_sample(sample_p, sample_q, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the Jensen-Shannon divergence between distributions p and q
JSD(p||q) = 0.5 * (D_KL(p||m) + D_KL(q||m)), with m = 0.5 * (p + q)
from samples of the two distributions via approximation by a kernel density estimate and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_p
|
ndarray
|
|
required |
sample_q
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The Jensen-Shannon divergence between distributions p and q.
|
|
continuous_mutual_information_from_samples(sample_x, sample_y, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute mutual information of the random variables x and y with joint density p_{x, y} and marginal densities p_x and p_y defined as the KL divergence between the product of marginal densities and the joint density, i.e.
I(X; Y) = D_KL(p_{x, y}|| p_x \otimes p_y) =
E_{p_{x, y}} \left[ \log \left( \frac{p_{x, y} (x, y)}{p_x(x) p_y(y)} \right) \right]
from samples of the two distributions via approximation by kernel density estimates and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The mutual information of the random variables x and y
|
|
continuous_joint_entropy_from_samples(sample_x, sample_y, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute joint entropy of the random variables x and y with joint density p_{x, y} defined as
H(X, Y) = - E_{p_{x, y}} \left[ \log p_{x, y} (x, y) \right]
from samples of the two distributions via approximation by kernel density estimates and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The joint entropy of the random variables x and y
|
|
continuous_conditional_entropy_from_samples(sample_x, sample_y, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute conditional entropy of the random variables x and y with joint density p_{x, y} and marginal density p_x defined as
H(Y|X) = - E_{p_{x, y}} \left[ \log \frac{p_{x, y} (x, y)}{p_x(x)} \right]
from samples of the two distributions via approximation by kernel density estimates and numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_x
|
ndarray
|
|
required |
sample_y
|
ndarray
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The conditional entropy of the random variables x and y
|
|
Continuous (density-based)¶
entropy_from_density_with_support(pdf, a, b, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the entropy
H(p) = - E_p[log(p)]
of the density given in pdf via numerical integration from a to b. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pdf
|
Callable
|
|
required |
a
|
float
|
|
required |
b
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The entropy of the density given by pdf
|
|
cross_entropy_from_densities_with_support(p, q, a, b, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the cross entropy of the distribution q relative to the distribution p
H_q(p) = - E_p [log(q)]
via numerical integration from a to b. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
Callable
|
|
required |
q
|
Callable
|
|
required |
a
|
float
|
|
required |
b
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The cross entropy of the distribution q relative to the distribution p.
|
|
relative_entropy_from_densities_with_support(p, q, a, b, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the relative entropy of the distribution q relative to the distribution p
D_KL(p||q) = E_p [log(p/q)]
via numerical integration from a to b. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
Callable
|
|
required |
q
|
Callable
|
|
required |
a
|
float
|
|
required |
b
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The relative entropy of the distribution q relative to the distribution p.
|
|
jensen_shannon_divergence_from_densities_with_support(p, q, a, b, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute the Jensen-Shannon divergence between distributions p and q
JSD(p||q) = 0.5 * (D_KL(p||m) + D_KL(q||m)), with m = 0.5 * (p + q)
via numerical integration from a to b. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
Callable
|
|
required |
q
|
Callable
|
|
required |
a
|
float
|
|
required |
b
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The Jensen-Shannon divergence between distributions p and q.
|
|
mutual_information_from_densities_with_support(pdf_x, pdf_y, pdf_xy, x_min, x_max, y_min, y_max, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute mutual information of the random variables x and y with joint density p_{x, y} and marginal densities p_x and p_y defined as the KL divergence between the product of marginal densities and the joint density, i.e.
I(X; Y) = D_KL(p_{x, y}|| p_x \otimes p_y) =
E_{p_{x, y}} \left[ \log \left( \frac{p_{x, y} (x, y)}{p_x(x) p_y(y)} \right) \right]
via numerical integration on a rectangular domain aligned with the axes. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pdf_x
|
Callable
|
|
required |
pdf_y
|
Callable
|
|
required |
pdf_xy
|
Callable
|
|
required |
x_min
|
float
|
|
required |
x_max
|
float
|
|
required |
y_min
|
float
|
|
required |
y_max
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The mutual information of the random variables x and y
|
|
joint_entropy_from_densities_with_support(pdf_xy, x_min, x_max, y_min, y_max, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute joint entropy of the random variables x and y with joint density p_{x, y} defined as
H(X, Y) = - E_{p_{x, y}} \left[ \log p_{x, y} (x, y) \right]
via numerical integration on a rectangular domain aligned with the axes. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pdf_xy
|
Callable
|
|
required |
x_min
|
float
|
|
required |
x_max
|
float
|
|
required |
y_min
|
float
|
|
required |
y_max
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The joint entropy of the random variables x and y
|
|
conditional_entropy_from_densities_with_support(pdf_x, pdf_xy, x_min, x_max, y_min, y_max, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute conditional entropy of the random variables x and y with joint density p_{x, y} and marginal density p_x defined as
H(Y|X) = - E_{p_{x, y}} \left[ \log \frac{p_{x, y} (x, y)}{p_x(x)} \right]
via numerical integration on a rectangular domain aligned with the axes. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pdf_x
|
Callable
|
|
required |
pdf_xy
|
Callable
|
|
required |
x_min
|
float
|
|
required |
x_max
|
float
|
|
required |
y_min
|
float
|
|
required |
y_max
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The conditional entropy of the random variables x and y
|
|
Continuous (KDE-based)¶
entropy_from_kde(kde, base=np.e)
¶
Compute the entropy
H(p) = - E_p[log(p)]
of the density given by the statsmodels kde object using grid integration over the KDE's pre-computed support.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kde
|
KDEUnivariate
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The entropy of the density approximated by the kde
|
|
cross_entropy_from_kde(p, q, base=np.e)
¶
Compute the cross entropy of the distribution q relative to the distribution p
H_q(p) = - E_p [log(q)]
given by the statsmodels kde objects using grid integration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
KDEUnivariate
|
|
required |
q
|
KDEUnivariate
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The cross entropy of the distribution q relative to the distribution p.
|
|
relative_entropy_from_kde(p, q, base=np.e)
¶
Compute the relative entropy of the distribution q relative to the distribution p
D_KL(p||q) = E_p [log(p/q)]
given by the statsmodels kde objects using grid integration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
KDEUnivariate
|
|
required |
q
|
KDEUnivariate
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The relative entropy of the distribution q relative to the distribution p.
|
|
jensen_shannon_divergence_from_kde(p, q, base=np.e)
¶
Compute the Jensen-Shannon divergence between distributions p and q
JSD(p||q) = 0.5 * (D_KL(p||m) + D_KL(q||m)), with m = 0.5 * (p + q)
given by the statsmodels kde objects using grid integration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
KDEUnivariate
|
|
required |
q
|
KDEUnivariate
|
|
required |
base
|
float
|
|
e
|
Returns:
| Type | Description |
|---|---|
The Jensen-Shannon divergence between distributions p and q.
|
|
mutual_information_from_kde(kde_x, kde_y, kde_xy, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute mutual information of the random variables x and y with joint density p_{x, y} and marginal densities p_x and p_y defined as the KL divergence between the product of marginal densities and the joint density, i.e.
I(X; Y) = D_KL(p_{x, y}|| p_x \otimes p_y) =
E_{p_{x, y}} \left[ \log \left( \frac{p_{x, y} (x, y)}{p_x(x) p_y(y)} \right) \right]
given by the statsmodels kde objects for the marginal densities and a SciPy gaussian_kde object for the joint density via numerical integration. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kde_x
|
KDEUnivariate
|
|
required |
kde_y
|
KDEUnivariate
|
|
required |
kde_xy
|
gaussian_kde
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The mutual information of the random variables x and y
|
|
joint_entropy_from_kde(kde_xy, x_min, x_max, y_min, y_max, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute joint entropy of the random variables x and y with joint density p_{x, y} defined as
H(X, Y) = - E_{p_{x, y}} \left[ \log p_{x, y} (x, y) \right]
via numerical integration, where the joint density is given by a SciPy gaussian_kde object. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kde_xy
|
gaussian_kde
|
|
required |
x_min
|
float
|
|
required |
x_max
|
float
|
|
required |
y_min
|
float
|
|
required |
y_max
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The joint entropy of the random variables x and y
|
|
conditional_entropy_from_kde(kde_x, kde_xy, y_min, y_max, base=np.e, eps_abs=1.49e-08, eps_rel=1.49e-08)
¶
Compute conditional entropy of the random variables x and y with joint density p_{x, y} and marginal density p_x defined as
H(Y|X) = - E_{p_{x, y}} \left[ \log \frac{p_{x, y} (x, y)}{p_x(x)} \right]
via numerical integration, where the marginal density of x is given by a statsmodels kde object and the joint density by a SciPy gaussian_kde object. The argument base can be used to specify the units in which the entropy is measured. The default choice is the natural logarithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kde_x
|
KDEUnivariate
|
|
required |
kde_y
|
|
required | |
kde_xy
|
gaussian_kde
|
|
required |
y_min
|
float
|
|
required |
y_max
|
float
|
|
required |
base
|
float
|
|
e
|
eps_abs
|
float
|
|
1.49e-08
|
eps_rel
|
float
|
|
1.49e-08
|
Returns:
| Type | Description |
|---|---|
The conditional entropy of the random variables x and y
|
|
Utilities¶
A small interval helper used by the density-based functions when intersecting the supports of two estimated densities.
intersection(a0, b0, a1, b1)
¶
Calculate the intersection of two intervals [a0, b0] and [a1, b1]. If the intervals do not overlap the function returns None. The parameters must satisfy a0 <= b0 and a1 <= b1.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
a0
|
float
|
|
required |
b0
|
float
|
|
required |
a1
|
float
|
|
required |
b1
|
float
|
|
required |