Assert(; cond, msg="")

Asserts all columns of the table by throwing a AssertionError(msg) if cond(column) returns false, otherwise returns the input table.

The msg argument can be a string, or a function that receives the column name and returns a string, e.g.: nm -> "error in column $nm".

Assert(col₁, col₂, ..., colₙ; cond, msg="")
Assert([col₁, col₂, ..., colₙ]; cond, msg="")
Assert((col₁, col₂, ..., colₙ); cond, msg="")

Asserts the selected columns col₁, col₂, ..., colₙ.

Assert(regex; cond, msg="")

Asserts the columns that match with regex.

Examples

Assert(cond=allunique, msg="assertion error")
Assert([2, 3, 5], cond=x -> sum(x) > 100)
Assert([:b, :c, :e], cond=x -> eltype(x) <: Integer)
Assert(("b", "c", "e"), cond=allunique, msg=nm -> "error in column $nm")
Assert(r"[bce]", cond=x -> sum(x) > 100)

Select(col₁, col₂, ..., colₙ)
Select([col₁, col₂, ..., colₙ])
Select((col₁, col₂, ..., colₙ))

The transform that selects columns col₁, col₂, ..., colₙ.

Select(col₁ => newcol₁, col₂ => newcol₂, ..., colₙ => newcolₙ)

Selects the columns col₁, col₂, ..., colₙ and rename them to newcol₁, newcol₂, ..., newcolₙ.

Select(regex)

Selects the columns that match with regex.

Examples

Select(1, 3, 5)
Select([:a, :c, :e])
Select(("a", "c", "e"))
Select(1 => :x, 3 => :y)
Select(:a => :x, :b => :y)
Select("a" => "x", "b" => "y")
Select(r"[ace]")

Reject(col₁, col₂, ..., colₙ)
Reject([col₁, col₂, ..., colₙ])
Reject((col₁, col₂, ..., colₙ))

The transform that discards columns col₁, col₂, ..., colₙ.

Reject(regex)

Discards the columns that match with regex.

Examples

Reject(:b, :d, :f)
Reject(["b", "d", "f"])
Reject((2, 4, 6))
Reject(r"[bdf]")

Satisfies(pred)

Selects the columns where pred(column) returns true.

Examples

Satisfies(allunique)
Satisfies(x -> sum(x) > 100)
Satisfies(x -> eltype(x) <: Integer)

Only(S)

Selects the columns that have scientific type S.

Examples

using DataScienceTraits

Only(Continuous)

Except(S)

Selects the columns that don't have scientific type S.

Examples

using DataScienceTraits

Except(Categorical)

Rename(:col₁ => :newcol₁, :col₂ => :newcol₂, ..., :colₙ => :newcolₙ)
Rename([:col₁ => :newcol₁, :col₂ => :newcol₂, ..., :colₙ => :newcolₙ])

Renames the columns col₁, col₂, ..., colₙ to newcol₁, newcol₂, ..., newcolₙ.

Rename(fun)

Renames the table columns using the modification function fun that takes a string as input and returns another string with the new name.

Examples

Rename(1 => :x, 3 => :y)
Rename(:a => :x, :c => :y)
Rename("a" => "x", "c" => "y")
Rename([1 => "x", 3 => "y"])
Rename([:a => "x", :c => "y"])
Rename(["a", "c"] .=> [:x, :y])
Rename(nm -> nm * "_suffix")

StdNames(spec = :uppersnake)

Standardizes column names according to given spec. Default to :uppersnake case specification.

Specs

• :uppersnake - Upper Snake Case, e.g. COLUMN_NAME
• :uppercamel - Upper Camel Case, e.g. ColumnName
• :upperflat - Upper Flat Case, e.g. COLUMNNAME
• :snake - Snake Case, e.g. column_name
• :camel - Camel Case, e.g. columnName
• :flat - Flat Case, e.g. columnname

StdFeats()

Standardizes the columns of the table based on scientific types:

• Continuous: ZScore
• Categorical: Identity
• Unknown: Identity

Sort(col₁, col₂, ..., colₙ; kwargs...)
Sort([col₁, col₂, ..., colₙ]; kwargs...)
Sort((col₁, col₂, ..., colₙ); kwargs...)

Sort the rows of selected columns col₁, col₂, ..., colₙ by forwarding the kwargs to the sortperm function.

Sort(regex; kwargs...)

Sort the rows of columns that match with regex.

Examples

Sort(:a)
Sort(:a, :c, rev=true)
Sort([1, 3, 5], by=row -> abs.(row))
Sort(("a", "c", "e"))
Sort(r"[ace]")

Sample(size, [weights]; replace=true, ordered=false, rng=default_rng())

Sample size rows of table using weights with or without replacement depending on the option replace. The option ordered can be used to return samples in the same order of the original table.

Examples

Sample(1000)
Sample(1000, replace=false)
Sample(1000, replace=false, ordered=true)

# with rng
using Random
rng = MersenneTwister(2)
Sample(1000, rng=rng)

# with weights
Sample(10, rand(100))

Filter(pred)

Filters the table returning only the rows where the predicate pred is true.

Examples

Filter(row -> sum(row) > 10)
Filter(row -> row.a == true && row.b < 30)
Filter(row -> row."a" == true && row."b" < 30)
Filter(row -> row[1] == true && row[2] < 30)
Filter(row -> row[:a] == true && row[:b] < 30)
Filter(row -> row["a"] == true && row["b"] < 30)

Notes

• The schema of the table is preserved by the transform.

DropMissing()
DropMissing(:)

Drop all rows with missing values in table.

DropMissing(col₁, col₂, ..., colₙ)
DropMissing([col₁, col₂, ..., colₙ])
DropMissing((col₁, col₂, ..., colₙ))

Drop all rows with missing values in selected columns col₁, col₂, ..., colₙ.

DropMissing(regex)

Drop all rows with missing values in columns that match with regex.

Examples

DropMissing()
DropMissing("b", "c", "e")
DropMissing([2, 3, 5])
DropMissing((:b, :c, :e))
DropMissing(r"[bce]")

Notes

• The transform can alter the element type of columns from Union{Missing,T} to T.
• If the transformed column has only missing values, it will be converted to an empty column of type Any.

DropNaN()
DropNaN(:)

Drop all rows with NaN values in table.

DropNaN(col₁, col₂, ..., colₙ)
DropNaN([col₁, col₂, ..., colₙ])
DropNaN((col₁, col₂, ..., colₙ))

Drop all rows with NaN values in selected columns col₁, col₂, ..., colₙ.

DropNaN(regex)

Drop all rows with NaN values in columns that match with regex.

Examples

DropNaN(2, 3, 4)
DropNaN([:b, :c, :d])
DropNaN(("b", "c", "d"))
DropNaN(r"[bcd]")

DropExtrema(; low=0.25, high=0.75)

Drops rows where any of the values in all columns are outside the interval ([quantile(col, low), quantile(col, high)]).

DropExtrema(col₁, col₂, ..., colₙ; low=0.25, high=0.75)
DropExtrema([col₁, col₂, ..., colₙ]; low=0.25, high=0.75)
DropExtrema((col₁, col₂, ..., colₙ); low=0.25, high=0.75)

Drops rows where any of the values in columns col₁, col₂, ..., colₙ are outside the interval.

DropExtrema(regex; low=0.25, high=0.75)

Drops rows where any of the values in columns that match with regex are outside the interval.

Examples

DropExtrema(low=0.3, high=0.7)
DropExtrema(1, low=0.3, high=0.7)
DropExtrema(:a, low=0.2, high=0.8)
DropExtrema("a", low=0.3, high=0.7)
DropExtrema(1, 3, 5, low=0, high=1)
DropExtrema([:a, :c, :e], low=0.3, high=0.7)
DropExtrema(("a", "c", "e"), low=0.25, high=0.75)
DropExtrema(r"[ace]", low=0.3, high=0.7)

DropUnits()
DropUnits(:)

Drop units from all columns in the table.

DropUnits(col₁, col₂, ..., colₙ)
DropUnits([col₁, col₂, ..., colₙ])
DropUnits((col₁, col₂, ..., colₙ))

Drop units from selected columns col₁, col₂, ..., colₙ.

DropUnits(regex)

Drop units from columns that match with regex.

Examples

DropUnits()
DropUnits([2, 3, 5])
DropUnits([:b, :c, :e])
DropUnits(("b", "c", "e"))
DropUnits(r"[bce]")

DropConstant()

Drops the constant columns using the allequal function.

AbsoluteUnits()
AbsoluteUnits(:)

Converts the units of all columns in the table to absolute units.

AbsoluteUnits(col₁, col₂, ..., colₙ)
AbsoluteUnits([col₁, col₂, ..., colₙ])
AbsoluteUnits((col₁, col₂, ..., colₙ))

Converts the units of selected columns col₁, col₂, ..., colₙ to absolute units.

AbsoluteUnits(regex)

Converts the units of columns that match with regex to absolute units.

Examples

AbsoluteUnits()
AbsoluteUnits([2, 3, 5])
AbsoluteUnits([:b, :c, :e])
AbsoluteUnits(("b", "c", "e"))
AbsoluteUnits(r"[bce]")

Unitify()

Add units to columns of the table using bracket syntax. A column named col [unit] will be renamed to a unitful column col with a valid unit from Unitful.jl.

In the case that the unit is not recognized by Unitful.jl, no units are added. Empty brackets are also allowed to represent columns without units, e.g. col [].

Unit(unit)

Converts the units of all columns in the table to unit.

Unit(cols₁ => unit₁, cols₂ => unit₂, ..., colsₙ => unitₙ)

Converts the units of selected columns cols₁, cols₂, ..., colsₙ to unit₁, unit₂, ... unitₙ.

The column selection can be a single column identifier (index or name), a collection of identifiers or a regular expression (regex).

Examples

Unit(u"m")
Unit(1 => u"km", :b => u"K", "c" => u"s")
Unit([2, 3] => u"cm")
Unit([:a, :c] => u"cm")
Unit(["a", "c"] => u"cm")
Unit(r"[abc]" => u"km")

Map(cols₁ => fun₁ => target₁, cols₂ => fun₂, ..., colsₙ => funₙ => targetₙ)

Applies the funᵢ function to the columns selected by colsᵢ using the map function and saves the result in a new column named targetᵢ.

The column selection can be a single column identifier (index or name), a collection of identifiers or a regular expression (regex).

Passing a target column name is optional and when omitted a new name is generated by joining the function name with the selected column names. If the target column already exists in the table, the original column will be replaced.

Examples

Map(1 => sin)
Map(:a => sin, "b" => cos => :cos_b)
Map([2, 3] => ((b, c) -> 2b + c))
Map([:a, :c] => ((a, c) -> 2a * 3c) => :col1)
Map(["c", "a"] => ((c, a) -> 3c / a) => :col1, "c" => tan)
Map(r"[abc]" => ((a, b, c) -> a^2 - 2b + c) => "col1")

Notes

• Anonymous functions must be passed with parentheses as in the examples above;
• Some function names are treated in a special way, they are:
  • Anonymous functions: #1 -> f1;
  • Composed functions: outer ∘ inner -> outer_inner;
  • Base.Fix1 functions: Base.Fix1(f, x) -> fix1_f;
  • Base.Fix2 functions: Base.Fix2(f, x) -> fix2_f;

Replace(cols₁ => pred₁ => new₁, pred₂ => new₂, ..., colsₙ => predₙ => newₙ)

Replaces all values where predᵢ predicate returns true with newᵢ value in the the columns selected by colsᵢ.

Passing a column selection is optional and when omitted all columns in the table will be selected. The column selection can be a single column identifier (index or name), a collection of identifiers, or a regular expression (regex).

The predicate can be a function that accepts a single argument and returns a boolean, or a value. If the predicate is a value, it will be transformed into the following function: x -> x === value.

Examples

Replace(1 => -1, 5 => -5)
Replace(2 => 0.0 => 1.5, 5.0 => 5.5)
Replace(:b => 0.0 => 1.5, 5.0 => 5.5)
Replace("b" => 0.0 => 1.5, 5.0 => 5.5)
Replace([1, 3] => >(5) => 5)
Replace([:a, :c] => isequal(2) => -2)
Replace(["a", "c"] => (x -> 4 < x < 6) => 0)
Replace(r"[abc]" => (x -> isodd(x) && x > 10) => 2)

Notes

• Anonymous functions must be passed with parentheses as in the examples above.
• Replacements are applied in the sequence in which they are defined, therefore, if there is more than one replacement for the same column, the first valid one will be applied.

Coalesce(; value)

Replaces all missing values from the table with value.

Coalesce(col₁, col₂, ..., colₙ; value)
Coalesce([col₁, col₂, ..., colₙ]; value)
Coalesce((col₁, col₂, ..., colₙ); value)

Replaces all missing values from the columns col₁, col₂, ..., colₙ with value.

Coalesce(regex; value)

Replaces all missing values from the columns that match with regex with value.

Examples

Coalesce(value=0)
Coalesce(1, 3, 5, value=1)
Coalesce([:a, :c, :e], value=2)
Coalesce(("a", "c", "e"), value=3)
Coalesce(r"[ace]", value=4)

Notes

• The transform can alter the element type of columns from Union{Missing,T} to T.

Coerce(col₁ => S₁, col₂ => S₂, ..., colₙ => Sₙ)

Return a copy of the table, ensuring that the scientific types of the columns match the new specification.

Coerce(S)

Coerce all columns of the table with scientific type S.

This transform uses the DataScienceTraits.coerce function. Please see their docstring for more details.

Examples

using DataScienceTraits
Coerce(1 => Continuous, 2 => Continuous)
Coerce(:a => Continuous, :b => Continuous)
Coerce("a" => Continuous, "b" => Continuous)

Levels(col₁ => levels₁, col₂ => levels₂, ..., colₙ => levelsₙ; ordered=nothing)

Convert columns col₁, col₂, ..., colₙ to categorical arrays with given levels levels₁, levels₂, ..., levelsₙ. Optionally, specify which columns are ordered.

Examples

Levels(1 => 1:3, 2 => ["a", "b"], ordered=r"a")
Levels(:a => 1:3, :b => ["a", "b"], ordered=[:a])
Levels("a" => 1:3, "b" => ["a", "b"], ordered=["b"])

Indicator(col; k=10, scale=:quantile, categ=false)

Transforms continuous variable into k indicator variables defined by half-intervals of col values in a given scale. Optionally, specify the categ option to return binary categorical values as opposed to raw 1s and 0s.

Given a sequence of increasing threshold values t1 < t2 < ... < tk, the indicator transform converts a continuous variable Z into a sequence of k variables Z_1 = Z <= t1, Z_2 = Z <= t2, ..., Z_k = Z <= tk.

Scales:

• :quantile - threshold values are calculated using the quantile(Z, p) function with a linear range of probabilities.
• :linear - threshold values are calculated using a linear range.

Examples

Indicator(1, k=3)
Indicator(:a, k=6, scale=:linear)
Indicator("a", k=9, scale=:linear, categ=true)

OneHot(col; categ=false)

Transforms categorical column col into one-hot columns of levels returned by the levels function of CategoricalArrays.jl. The categ option can be used to convert resulting columns to categorical arrays as opposed to boolean vectors.

Examples

OneHot(1)
OneHot(:a)
OneHot("a")
OneHot("a", categ=true)

Identity()

The identity transform that maps any object to itself.

Center()

Applies the center transform to all columns of the table. The center transform of the column x, with mean μ, is defined by x .- μ.

Center(col₁, col₂, ..., colₙ)
Center([col₁, col₂, ..., colₙ])
Center((col₁, col₂, ..., colₙ))

Applies the Center transform on columns col₁, col₂, ..., colₙ.

Center(regex)

Applies the Center transform on columns that match with regex.

Examples

Center(1, 3, 5)
Center([:a, :c, :e])
Center(("a", "c", "e"))
Center(r"[ace]")

LowHigh(; low=0.25, high=0.75)

Applies the LowHigh transform to all columns of the table. The LowHigh transform of the column x is defined by (x .- xl) ./ (xh - xl), where xl = quantile(x, low) and xh = quantile(x, high).

LowHigh(col₁, col₂, ..., colₙ; low=0.25, high=0.75)
LowHigh([col₁, col₂, ..., colₙ]; low=0.25, high=0.75)
LowHigh((col₁, col₂, ..., colₙ); low=0.25, high=0.75)

Applies the LowHigh transform on columns col₁, col₂, ..., colₙ.

LowHigh(regex; low=0.25, high=0.75)

Applies the LowHigh transform on columns that match with regex.

Examples

LowHigh()
LowHigh(low=0, high=1)
LowHigh(low=0.3, high=0.7)
LowHigh(1, 3, 5, low=0, high=1)
LowHigh([:a, :c, :e], low=0.3, high=0.7)
LowHigh(("a", "c", "e"), low=0.25, high=0.75)
LowHigh(r"[ace]", low=0.3, high=0.7)

MinMax()

Applies the MinMax transform to all columns of the table. The MinMax transform is equivalent to LowHigh(low=0, high=1).

MinMax(col₁, col₂, ..., colₙ)
MinMax([col₁, col₂, ..., colₙ])
MinMax((col₁, col₂, ..., colₙ))

Applies the MinMax transform on columns col₁, col₂, ..., colₙ.

MinMax(regex)

Applies the MinMax transform on columns that match with regex.

Examples

MinMax(1, 3, 5)
MinMax([:a, :c, :e])
MinMax(("a", "c", "e"))
MinMax(r"[ace]")

See also LowHigh.

Interquartile()

Applies the Interquartile transform to all columns of the table. The Interquartile transform is equivalent to LowHigh(low=0.25, high=0.75).

Interquartile(col₁, col₂, ..., colₙ)
Interquartile([col₁, col₂, ..., colₙ])
Interquartile((col₁, col₂, ..., colₙ))

Applies the Interquartile transform on columns col₁, col₂, ..., colₙ.

Interquartile(regex)

Applies the Interquartile transform on columns that match with regex.

Examples

Interquartile(1, 3, 5)
Interquartile([:a, :c, :e])
Interquartile(("a", "c", "e"))
Interquartile(r"[ace]")

See also LowHigh.

ZScore()

Applies the z-score transform (a.k.a. normal score) to all columns of the table. The z-score transform of the column x, with mean μ and standard deviation σ, is defined by (x .- μ) ./ σ.

ZScore(col₁, col₂, ..., colₙ)
ZScore([col₁, col₂, ..., colₙ])
ZScore((col₁, col₂, ..., colₙ))

Applies the ZScore transform on columns col₁, col₂, ..., colₙ.

ZScore(regex)

Applies the ZScore transform on columns that match with regex.

Examples

ZScore(1, 3, 5)
ZScore([:a, :c, :e])
ZScore(("a", "c", "e"))
ZScore(r"[ace]")

Quantile(; dist=Normal())

The quantile transform to a given distribution.

Quantile(col₁, col₂, ..., colₙ; dist=Normal())
Quantile([col₁, col₂, ..., colₙ]; dist=Normal())
Quantile((col₁, col₂, ..., colₙ); dist=Normal())

Applies the Quantile transform on columns col₁, col₂, ..., colₙ.

Quantile(regex; dist=Normal())

Applies the Quantile transform on columns that match with regex.

Examples

using Distributions

Quantile()
Quantile(dist=Normal())
Quantile(1, 3, 5, dist=Beta())
Quantile([:a, :c, :e], dist=Gamma())
Quantile(("a", "c", "e"), dist=Beta())
Quantile(r"[ace]", dist=Normal())

Functional(fun)

The transform that applies a fun elementwise.

Functional(col₁ => fun₁, col₂ => fun₂, ..., colₙ => funₙ)

Apply the corresponding funᵢ function to each colᵢ column.

Examples

Functional(exp)
Functional(log)
Functional(1 => exp, 2 => log)
Functional(:a => exp, :b => log)
Functional("a" => exp, "b" => log)

EigenAnalysis(proj; [maxdim], [pratio])

The eigenanalysis of the covariance with a given projection proj. Optionally specify the maximum number of dimensions in the output maxdim and the percentage of variance to retain pratio.

Projections

• :V - Uncorrelated variables (PCA transform)
• :VD - Uncorrelated variables and variance one (DRS transform)
• :VDV - Uncorrelated variables and variance one (SDS transformation)

The :V projection used in the PCA transform projects the data on the eigenvectors V of the covariance matrix.

The :VD projection used in the DRS transform. Similar to the :V projection, but the eigenvectors are multiplied by the squared inverse of the eigenvalues D.

The :VDV projection used in the SDS transform. Similar to the :VD transform, but the data is projected back to the basis of the original variables using the Vᵀ matrix.

See https://geostatisticslessons.com/lessons/sphereingmaf for more details about these three variants of eigenanalysis.

Examples

EigenAnalysis(:V)
EigenAnalysis(:VD)
EigenAnalysis(:VDV)
EigenAnalysis(:V, maxdim=3)
EigenAnalysis(:VD, pratio=0.99)
EigenAnalysis(:VDV, maxdim=3, pratio=0.99)

PCA([options])

Principal component analysis.

See EigenAnalysis for detailed description of the available options.

Examples

PCA(maxdim=2)
PCA(pratio=0.86)
PCA(maxdim=2, pratio=0.86)

Notes

• PCA() is shortcut for ZScore() → EigenAnalysis(:V).

DRS([options])

Dimension reduction sphering.

See EigenAnalysis for detailed description of the available options.

Examples

DRS(maxdim=3)
DRS(pratio=0.87)
DRS(maxdim=3, pratio=0.87)

Notes

• DRS() is shortcut for ZScore() → EigenAnalysis(:VD).

SDS([options])

Standardized data sphering.

See EigenAnalysis for detailed description of the available options.

Examples

SDS()
SDS(maxdim=4)
SDS(pratio=0.88)
SDS(maxdim=4, pratio=0.88)

Notes

• SDS() is shortcut for ZScore() → EigenAnalysis(:VDV).

ProjectionPursuit(; tol=1e-6, maxiter=100, deg=5, perc=0.9, n=100, rng=Random.default_rng())

The projection pursuit multivariate transform converts any multivariate distribution into the standard multivariate Gaussian distribution.

This iterative algorithm repeatedly finds a direction of projection α that maximizes a score of non-Gaussianity known as the projection index I(α). The samples projected along α are then transformed with the Quantile transform to remove the non-Gaussian structure. The other coordinates in the rotated orthonormal basis Q = [α ...] are left untouched.

The non-singularity of Q is controlled by assuring that norm(det(Q)) ≥ tol. The iterative process terminates whenever the transformed samples are "more Gaussian" than perc% of n randomly generated samples from the standard multivariate Gaussian distribution, or when the number of iterations reaches a maximum maxiter.

Examples

ProjectionPursuit()
ProjectionPursuit(deg=10)
ProjectionPursuit(perc=0.85, n=50)
ProjectionPursuit(tol=1e-4, maxiter=250, deg=5, perc=0.95, n=100)

# with rng
using Random
rng = MersenneTwister(2)
ProjectionPursuit(perc=0.85, n=50, rng=rng)

See https://doi.org/10.2307/2289161 for further details.

Closure()

The transform that applies the closure operation (i.e. x ./ sum(x)), to all rows of the input table. The rows of the output table sum to one.

See also Remainder.

Remainder([total])

The transform that takes a table with columns x₁, x₂, …, xₙ and returns a new table with an additional column containing the remainder value xₙ₊₁ = total .- (x₁ + x₂ + ⋯ + xₙ) If the total value is not specified, then default to the maximum sum across rows.

See also Closure.

Compose(; as=:CODA)

Converts all columns of the table into parts of a composition in a new column named as, using the CoDa.compose function.

Compose(col₁, col₂, ..., colₙ; as=:CODA)
Compose([col₁, col₂, ..., colₙ]; as=:CODA)
Compose((col₁, col₂, ..., colₙ); as=:CODA)

Converts the selected columns col₁, col₂, ..., colₙ into parts of a composition.

Compose(regex; as=:CODA)

Converts the columns that match with regex into parts of a composition.

Examples

Compose(as=:comp)
Compose([2, 3, 5])
Compose([:b, :c, :e])
Compose(("b", "c", "e"))
Compose(r"[bce]", as="COMP")

ALR([refvar])

Additive log-ratio transform.

Optionally, specify the reference variable refvar for the ratios. Default to the last column of the input table.

CLR()

Centered log-ratio transform.

ILR([refvar])

Isometric log-ratio transform.

Optionally, specify the reference variable refvar for the ratios. Default to the last column of the input table.

RowTable()

The transform that applies the function Tables.rowtable to to the input table.

ColTable()

The transform that applies the function Tables.columntable to to the input table.