API Reference: The grove module#

Multi-merge#

grove.merge(df_list, on=None)#

Merge multiple DataFrames (as an inner join). This module-level function allows more flexibility in passing DataFrames while doing on-the-fly operations, outside a Collection.

Example

>>> grove.merge(
...     [
...         items[['id']],
...         categories.head(4),
...         cat_descr.drop_duplicates('category_code')
...     ],
...     on=['id', ['category', 'category_code']]
... )

The merging is performed pairwise, in the specified order. I.e. for a given list [df_A, df_B, df_C], the result is merge(merge(df_A, df_B), df_C)

Handling of the on argument is only slightly wrapped around Pandas behavior. The on argument can be omitted to join on the columns with the same name in all DataFrames. A single string may be provided if it’s the common column to join on in all DataFrames. The general structure for a list of DataFrames [X1, X2, ...,  Xn] is [X1X2_on, X2X3_on, ..., Xn-1Xn_on], where XiXj_on can be a string (common column), a pair of strings (left_on, right_on arguments), or a pair of list of multiple columns to join on. See Examples below.

Numerical suffixes are used for identically named columns, to allow an arbitrary number of merged DataFrames.

Note

Since the merge operation is performed iteratively left-to-right, each XiXj_on specification can use any column in preceding DataFrames, not just the columns in the adjacent Xi and Xj DataFrames.

Examples

All DataFrames are to be merged on their 'id' column:

>>> grove.merge([items_df, categories_df, measurements_df], on='id')

DataFrames items_df and categories_df are merged on 'id', while cat_descr_df is merged on 'category', which matches 'category' in categories_df:

>>> grove.merge([items_df, categories_df, cat_descr_df],
...             on=['id', ['category', 'category_code']])

Multi-column join between items_df and categories_df (on columns 'id' and 'id2'), followed by joining in measurements_df on column 'id_x' matching column 'id2' in the items - categories merge.

>>> grove.merge(
...     [items_df, categories_df, measurements_df],
...      on=[[['id', 'id2'], ['id', 'id2']], ['id2', 'id_x']]
... )

For multi-column joins with common columns, make sure to put these in a nested list:

>>> grove.merge(
...     [items_df, categories_df, measurements_df],
...      on=[[['A', 'B']], ['id2', 'id_x']]
... )
Parameters:

  • df_list (list) – DataFrames to be merged, in order they appear in the list

  • on (Union[str, int, list]) – Column names to join on. Either a single string or omitted if it’s the same across all DataFrames. If column names to join on differ, these are given as a list.

Return type:

DataFrame

Returns:

Merged DataFrame

Perform sanity checks#

grove.sanity_check_df(df, id_column='')#

Check for typical desirable data properties for the given DataFrame:

  • Unique IDs (either in specified id_column or any non-float, no-N/As column)

  • No completely empty (N/A) columns

  • Warn about ‘object’ type columns since they cause merge problems

Example

>>> df = pd.DataFrame.from_records(
...     zip([10, 20, 30, 10, 40],
...         ['a', 'b', 'c', 'd', 'e'],
...         [np.nan] * 5,
...         ['10', '20', '30', '40', None]),
...     columns=['id', 'descr', 'values', 'serials'])

>>> grove.sanity_check_df(df, id_column='id')
WARN: ID column 'id' does not have unique values
WARN: The following columns are completely empty (N/A): ['values']
False

>>> grove.sanity_check_df(df, id_column='serials')
WARN: ID column 'serials' has N/A values
WARN: The following columns are completely empty (N/A): ['values']
False
Parameters:

  • df (DataFrame) – Pandas DataFrame

  • id_column (str) – If given, this column is checked for unique values

Return type:

bool

Returns:

True if all checks passed, False otherwise

Optimize size of DataFrames#

grove.reduce_mem_df(df, target_float='float32', inplace=False)#

Minimize memory usage of a DataFrame by using the smallest applicable Numpy datatypes for all integer and float columns.

For floats, the target float precision (default float32) must be decided beforehand, since this is application-specific.

Subsequent operations on the DataFrame will likely promote the dtypes to the platform default (e.g. int64) so best used with read-only data.

Good practice is usually to keep input data immutable, but due to sheer size of some DataFrames, this operation can be done in-place, by setting the inplace flag.

Example

>>> rng = np.random.default_rng(42)
>>> df = pd.DataFrame.from_records(
... zip(rng.integers(0, int(1e6), size=df_len),
...     rng.random(size=df_len) * 1e6,
...     rng.integers(0, 1, size=df_len)
...     ),
... columns=['integers', 'floats', 'binaries'])
>>> print(df.dtypes)
integers      int64
floats      float64
binaries      int64
dtype: object
>>> print(grove.reduce_mem_df(df).dtypes)
integers     uint32
floats      float32
binaries      uint8
dtype: object
Parameters:

  • df (DataFrame) – Input DataFrame, which will not be changed unless inplace is set.

  • target_float (str) – Float columns will be converted to this precision.

  • inplace – Defaults to False. If True, the input DataFrame will be changed.

Return type:

DataFrame

Returns:

A new DataFrame with reduced datatypes if inplace is not set, otherwise a reference will be returned to the changed input DataFrame.

Optimize size of Series#

grove.reduce_mem_series(values, target_float='float32')#

Minimize memory usage of a Series by using the smallest applicable Numpy datatype. Handles integers and floats only.

For floats, the target float precision (default float32) must be decided beforehand, since this is application-specific.

Subsequent operations on the Series will likely promote the dtype to the platform default (e.g. int64) so best used with read-only data.

Example

>>> rng = np.random.default_rng(42)
>>> series = pd.Series(rng.integers(0, int(1e6), size=10))
>>> print(series.dtype)
>>> print(grove.reduce_mem_series(series).dtype)
int64
uint32
Parameters:

  • values (Series) – A Pandas Series.

  • target_float (str) – A float Series will be converted to this precision.

Return type:

Series

Returns:

A copy of the input Series with optimized dtypes.