cuDF Reference

cuDF is a GPU DataFrame library that provides a pandas-like API for loading, joining, aggregating, filtering, and manipulating tabular data entirely on the GPU. It's part of the NVIDIA RAPIDS ecosystem and is built on the Apache Arrow columnar memory format.

Full documentation: https://docs.rapids.ai/api/cudf/stable/

Table of Contents

1. Installation and Setup
2. Two Usage Modes
3. cudf.pandas Accelerator Mode
4. Core API: DataFrame and Series
5. IO Operations
6. GroupBy Operations
7. String Operations
8. User Defined Functions (UDFs)
9. Missing Data Handling
10. Data Types
11. Memory Management
12. Interoperability
13. Multi-GPU with Dask-cuDF
14. Performance Optimization
15. Key Differences from pandas
16. Common Migration Patterns

---

Installation and Setup

Always use uv add (never pip install or conda install) in all install instructions, docstrings, comments, and error messages.

uv add --extra-index-url=https://pypi.nvidia.com cudf-cu12    # For CUDA 12.x

Verify:

print(cudf.Series([1, 2, 3]))  # Should print a GPU series

---

Two Usage Modes

cuDF offers two ways to accelerate pandas code:

1. cudf.pandas (Zero-Code-Change)

Drop-in replacement that automatically accelerates pandas. Falls back to CPU for unsupported operations. Best for: quick acceleration of existing code, mixed codebases, prototyping.

2. Direct cuDF API

Replace import pandas with import cudf. Maximum performance, no proxy overhead, but requires adapting code to cuDF's API (which has some behavioral differences from pandas). Best for: production pipelines, maximum performance, new GPU-first code.

---

cudf.pandas Accelerator Mode

The fastest path from pandas to GPU — no code changes required.

Activation

# Jupyter/IPython (MUST be before any pandas import)
%load_ext cudf.pandas

# Command line
# python -m cudf.pandas your_script.py
# python -m cudf.pandas --profile your_script.py  # With profiling

# Programmatic

cudf.pandas.install()

Critical: If pandas was already imported in the session, you must restart the kernel/process.

How It Works

• import pandas returns a proxy module that wraps cuDF and pandas.
• Every operation is first attempted on GPU (cuDF). If it fails, it automatically falls back to CPU (pandas).
• Data transfers between GPU and CPU happen only when necessary.
• Uses managed memory by default — can process datasets larger than GPU memory.
• Currently passes 93% of pandas' 187,000+ unit tests.

Profiling GPU vs CPU Execution

%%cudf.pandas.profile        # Shows GPU vs CPU operation breakdown per cell
%%cudf.pandas.line_profile   # Per-line GPU/CPU timing

Accessing Underlying Objects

proxy_df.as_gpu_object()  # Get the cuDF DataFrame directly
proxy_df.as_cpu_object()  # Get the pandas DataFrame directly

Note: automatic fallback stops working after you extract the underlying object.

Compatible Third-Party Libraries

cuGraph, cuML, Hvplot, Holoview, Ibis, NumPy, Matplotlib, Plotly, PyTorch, Seaborn, Scikit-Learn, SciPy, TensorFlow, XGBoost.

Not compatible: Joblib. For distributed work, use Dask-cuDF instead.

Limitations

• Join operations don't guarantee pandas' row ordering (for performance).
• Cannot use import cudf alongside cudf.pandas in the same session.
• Pickled objects are not interchangeable between regular pandas and cudf.pandas.
• Proxy arrays subclass numpy.ndarray, which can cause eager device-to-host transfers.
• To force CPU-only: set CUDF_PANDAS_FALLBACK_MODE=1.

---

Core API

Creating DataFrames and Series

# From dict
df = cudf.DataFrame({"a": [1, 2, 3], "b": [4.0, 5.0, 6.0], "c": ["x", "y", "z"]})

# From pandas

gdf = cudf.DataFrame.from_pandas(pd.DataFrame({"a": [1, 2, 3]}))
# or
gdf = cudf.DataFrame(pandas_df)

# Series
s = cudf.Series([1, 2, 3, None, 5])

# Back to pandas
pdf = gdf.to_pandas()

Common Operations (Same as pandas)

df.head(10)
df.tail(5)
df.describe()
df.info()
df.dtypes
df.columns
df.shape

# Selection
df["a"]                     # Column → Series
df[["a", "b"]]             # Multiple columns → DataFrame
df.loc[2:5, ["a", "b"]]   # Label-based indexing
df.iloc[0:3]               # Integer-based indexing

# Filtering
df[df["a"] > 2]
df.query("a > 2 and b < 6")  # Supports @var for local variables

# Sorting
df.sort_values("a", ascending=False)
df.sort_index()

# Missing data
df.fillna(0)
df.dropna()
df.isna()

# Aggregations
df["a"].sum()
df["a"].mean()
df["a"].std()
df["a"].value_counts()

# Transforms
df["a"].clip(lower=1, upper=5)
df["a"].apply(lambda x: x * 2)  # JIT-compiled

# Combining
cudf.concat([df1, df2])
df1.merge(df2, on="key")
df1.merge(df2, on="key", how="left")  # left, right, inner, outer

# Arrow interop (zero-copy)
arrow_table = df.to_arrow()
df = cudf.DataFrame.from_arrow(arrow_table)

---

IO Operations

GPU-accelerated file reading and writing — often dramatically faster than pandas for large files.

Parquet (Recommended for Performance)

# Read
df = cudf.read_parquet("data.parquet")
df = cudf.read_parquet("data.parquet", columns=["a", "b"])  # Read only specific columns

# Write
df.to_parquet("output.parquet")

# Metadata inspection (without loading data)
cudf.io.parquet.read_parquet_metadata("data.parquet")

# Incremental writing
writer = cudf.io.parquet.ParquetDatasetWriter("output_dir/", partition_cols=["year"])
writer.write_table(df)
writer.close()

CSV

df = cudf.read_csv("data.csv")
df = cudf.read_csv("data.csv", usecols=["a", "b"], dtype={"a": "int32"})
df.to_csv("output.csv", index=False)

JSON

df = cudf.read_json("data.json")
df = cudf.read_json("data.json", lines=True)  # JSON Lines format
df.to_json("output.json")

ORC

df = cudf.read_orc("data.orc")
df.to_orc("output.orc")

Other Formats

Prefer Parquet over CSV — columnar format reads faster on GPU, supports predicate pushdown, and compresses well.

---

GroupBy Operations

Basic GroupBy

df.groupby("category").sum()
df.groupby(["category", "subcategory"]).mean()
df.groupby("category").agg({"value": "sum", "count": "max"})
df.groupby("category").agg({"value": ["sum", "min", "max"], "count": "mean"})

Supported Aggregations

Universal: count, size, nunique, nth, collect, unique Numeric: sum, mean, var, std, median, idxmin, idxmax, min, max, quantile Specialized: corr, cov

GroupBy Transform

df.groupby("category").transform("max")  # Broadcasts result to match group size

GroupBy Apply

df.groupby("category").apply(lambda x: x.max() - x.min())

Warning: Apply runs the function sequentially per group — can be slow with many small groups. Use vectorized aggregations whenever possible.

JIT-Compiled GroupBy (User-Defined Aggregation)

def custom_agg(df):
    return df["value"].max() - df["value"].min() / 2

result = df.groupby("category").apply(custom_agg, engine="jit")

JIT restrictions: no nulls, only int32/64 and float32/64, cannot return new columns.

Important: Sort Behavior

cuDF uses sort=False by default (unlike pandas which sorts by default). To match pandas:

df.groupby("category", sort=True).sum()
# Or globally:
cudf.set_option("mode.pandas_compatible", True)

---

String Operations

cuDF provides GPU-accelerated string operations via the .str accessor — identical API to pandas.

s = cudf.Series(["Hello World", "foo bar", "RAPIDS GPU", None])

# Case
s.str.lower()
s.str.upper()
s.str.title()
s.str.capitalize()

# Pattern matching
s.str.contains("World")
s.str.startswith("Hello")
s.str.endswith("GPU")
s.str.match(r"^[A-Z]")

# Extraction and replacement
s.str.extract(r"(\\w+)\\s(\\w+)")
s.str.replace("World", "GPU")
s.str.slice(0, 5)

# Splitting and joining
s.str.split(" ")
s.str.cat(sep=", ")

# Info
s.str.len()
s.str.isalpha()
s.str.isdigit()

# cuDF-exclusive operations (not in pandas)
s.str.normalize_spaces()   # Collapse whitespace
s.str.tokenize()           # Tokenize strings
s.str.ngrams(2)            # Generate n-grams
s.str.edit_distance(other) # Levenshtein distance
s.str.url_encode()
s.str.url_decode()

---

User Defined Functions

Series.apply() — JIT-Compiled

s = cudf.Series([1, 2, 3, 4, 5])

def square_plus_one(x):
    return x ** 2 + 1

s.apply(square_plus_one)  # Compiled to GPU kernel via Numba

With arguments:

def add_constant(x, c):
    return x + c

s.apply(add_constant, args=(42,))

DataFrame.apply() — Row-wise (axis=1)

def row_func(row):
    return row["a"] + row["b"] * 2

df.apply(row_func, axis=1)  # Access columns by name via dict-like syntax

Null Handling in UDFs

Nulls propagate automatically:

s = cudf.Series([1, cudf.NA, 3])
def f(x):
    return x + 1
s.apply(f)  # Returns [2, , 4]

Explicit null checks:

def f(x):
    if x is cudf.NA:
        return 0
    return x + 1

String UDFs

String operations inside UDFs support: ==, !=, >=, <=, startswith(), endswith(), find(), rfind(), count(), in, strip/lstrip/rstrip(), upper/lower(), replace(), + (concatenation), len(), boolean checks.

For string UDFs creating intermediate strings, allocate heap:

from cudf.core.udf.utils import set_malloc_heap_size
set_malloc_heap_size(int(2e9))  # 2 GB

Rolling Window UDFs

s = cudf.Series([16, 25, 36, 49, 64, 81], dtype="float64")

def max_sqrt(window):
    result = 0
    for val in window:
        result = max(result, math.sqrt(val))
    return result

s.rolling(window=3, min_periods=3).apply(max_sqrt)

Limitation: Rolling UDFs do NOT support null values.

Custom Numba CUDA Kernels on cuDF Columns

For maximum control, write CUDA kernels that operate directly on cuDF columns:

from numba import cuda

@cuda.jit
def gpu_multiply(in_col, out_col, multiplier):
    i = cuda.grid(1)
    if i < in_col.size:
        out_col[i] = in_col[i] * multiplier

df["result"] = 0.0
gpu_multiply.forall(len(df))(df["a"], df["result"], 10.0)

UDF Limitations

• Only numeric non-decimal types have full support; strings have partial support.
• **kwargs not supported.
• Bitwise operations not implemented in UDFs.
• GroupBy JIT: no nulls, only int32/64 and float32/64, cannot return new columns.
• Rolling UDFs: no null support.

---

Missing Data Handling

• Missing values are `` (not NaN) — cuDF uses a separate null mask, not NaN sentinels.
• All dtypes are nullable (including integers — no float coercion for missing ints).
• np.nan inserted into integer columns becomes `` without casting to float.

s = cudf.Series([1, None, 3, None, 5])

s.isna()                # Boolean mask
s.notna()
s.fillna(0)             # Fill with scalar
s.fillna({"a": 0, "b": 1})  # Fill with dict (per-column)
s.dropna()

# Aggregations skip NA by default
s.sum()                 # skipna=True (default)
s.sum(skipna=False)     # Propagates NA

# GroupBy excludes NA groups by default
df.groupby("a", dropna=False).sum()  # Include NA groups

---

Data Types

All types are nullable. List columns have a .list accessor (get(), len(), contains(), sort_values(), unique(), concat()). Struct columns have a .struct accessor (field(), explode()).

No object dtype for arbitrary Python objects — object dtype only stores strings.

---

Memory Management

RMM (RAPIDS Memory Manager)

cuDF uses RMM for GPU memory allocation. Configure it for your workload:

# Pool allocator (recommended for production — avoids per-allocation cudaMalloc overhead)
pool = rmm.mr.PoolMemoryResource(
    rmm.mr.CudaMemoryResource(),
    initial_pool_size="1GiB",
    maximum_pool_size="4GiB"
)
rmm.mr.set_current_device_resource(pool)

# Managed memory (allows datasets larger than GPU memory)
rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())

# Managed + pool (best of both)
pool = rmm.mr.PoolMemoryResource(
    rmm.mr.ManagedMemoryResource(),
    initial_pool_size="1GiB"
)
rmm.mr.set_current_device_resource(pool)

Aligning CuPy and Numba with RMM

When using cuDF with CuPy or Numba, align all libraries on the same allocator to avoid memory fragmentation:

# CuPy
from rmm.allocators.cupy import rmm_cupy_allocator

cupy.cuda.set_allocator(rmm_cupy_allocator)

# Numba
from rmm.allocators.numba import RMMNumbaManager
from numba import cuda
cuda.set_memory_manager(RMMNumbaManager)

Copy-on-Write

cudf.set_option("copy_on_write", True)
# or: export CUDF_COPY_ON_WRITE=1

Slices, .head(), shallow copies, and view-generating methods share memory until one is modified. Reduces memory usage significantly for workflows with many derived DataFrames.

Memory Profiling

rmm.statistics.enable_statistics()
stats = rmm.statistics.get_statistics()
# Returns: current_bytes, current_count, peak_bytes, peak_count, total_bytes, total_count

---

Interoperability

CuPy (Zero-Copy)

# cuDF → CuPy
arr = df.to_cupy()             # DataFrame → 2D CuPy array
arr = cp.asarray(df["col"])    # Series → 1D CuPy array
arr = df["col"].values         # Series → 1D CuPy array

# CuPy → cuDF
df = cudf.DataFrame(cupy_2d_array)
s = cudf.Series(cupy_1d_array)

# Via DLPack
df = cudf.from_dlpack(cupy_array.__dlpack__())

Arrow (Zero-Copy)

arrow_