pattern outlier

Use Outlier Pattern for Exceptional Documents

Isolate atypical documents with large arrays to prevent them from degrading performance for typical queries. When a small subset of documents is much larger than the rest, those outliers can dominate memory, index, and query costs. Split overflow data into a separate collection and flag the document.

Problem scenario:

A typical book might have 50 customers in an embedded array, while a bestseller like Harry Potter accumulates 50,000 (~2.5MB). Queries return the full document, so the outlier dominates memory and network cost. A multikey index on that array produces 50,000 entries for a single document.

Correct (outlier pattern):

Typical documents keep their full embedded array and set hasExtras: false. Outlier documents cap the embedded array at a threshold (e.g. 50), set hasExtras: true, store a denormalized customerCount, and overflow remaining items into a separate collection in batched documents (e.g. { bookId, customers: [...], batch: 1, count: 950 }). Application code checks the hasExtras flag to decide whether to load overflow batches.

Implementation with threshold (example; tune per workload):

const CUSTOMER_THRESHOLD = 50

async function addCustomer(bookId, customerId) {
  // Try the normal case first: atomically add to the embedded array only if
  // the current customerCount is below the threshold (treat missing/null as 0).
  const result = await db.books.updateOne(
    {
      _id: bookId,
      $or: [
        { customerCount: { $lt: CUSTOMER_THRESHOLD } },
        { customerCount: { $exists: false } },
        { customerCount: null }
      ]
    },
    {
      $push: { customers: customerId },
      $inc: { customerCount: 1 }
    }
  )

  if (result.matchedCount > 0) {
    // Normal case succeeded - customer added to embedded array
    return
  }

  // Outlier case - add to overflow collection
  const lastBatchDoc = await db.book_customers_extra
    .find({ bookId: bookId })
    .sort({ batch: -1 })
    .limit(1)
    .next()

  const nextBatch = lastBatchDoc ? lastBatchDoc.batch + 1 : 1
  const targetBatch =
    lastBatchDoc && lastBatchDoc.count < 1000
      ? lastBatchDoc.batch
      : nextBatch

  // First, try to append to the intended batch, enforcing the 1000-item cap under concurrency.
  const overflowFilter = { bookId: bookId, batch: targetBatch }
  if (targetBatch !== nextBatch) {
    // Only enforce the count cap when targeting an existing batch.
    overflowFilter.count = { $lt: 1000 }
  }

  const overflowResult = await db.book_customers_extra.updateOne(
    overflowFilter, // Write to the intended batch, respecting the count cap when reusing a batch
    {
      $push: { customers: customerId },
      $inc: { count: 1 },
      $setOnInsert: { bookId: bookId, batch: targetBatch }
    },
    { upsert: targetBatch === nextBatch }
  )

  // If we failed to match when trying to reuse the previous batch (it filled concurrently),
  // fall back to writing into the next batch.
  if (overflowResult.matchedCount === 0 && targetBatch !== nextBatch) {
    await db.book_customers_extra.updateOne(
      { bookId: bookId, batch: nextBatch },
      {
        $push: { customers: customerId },
        $inc: { count: 1 },
        $setOnInsert: { bookId: bookId, batch: nextBatch }
      },
      { upsert: true }
    )
  }

  await db.books.updateOne(
    { _id: bookId },
    {
      $set: { hasExtras: true },
      $inc: { customerCount: 1 }
    }
  )
}

Index strategy:

// Index on main collection - only 50 entries per outlier doc
db.books.createIndex({ "customers": 1 })

// Index on overflow collection
db.book_customers_extra.createIndex({ bookId: 1 })
db.book_customers_extra.createIndex({ customers: 1 })

When to use outlier pattern:

When NOT to use this pattern:

• Uniform distribution: If all documents have similar array sizes, no outliers to isolate.
• Always need full data: If you always display all 50,000 customers, pattern doesn't help.
• Write-heavy outliers: Complex update logic may not be worth the read optimization.
• Small outliers: If outliers are 200 vs typical 50, just use larger threshold.

Verify with

// Find outlier documents
db.books.aggregate([
  { $project: {
    title: 1,
    customerCount: { $size: { $ifNull: ["$customers", []] } }
  }},
  { $sort: { customerCount: -1 } },
  { $limit: 20 }
])

// Calculate distribution
db.books.aggregate([
  { $project: { count: { $size: { $ifNull: ["$customers", []] } } } },
  { $bucket: {
    groupBy: "$count",
    boundaries: [0, 50, 100, 500, 1000, 10000, 100000],
    default: "100000+",
    output: { count: { $sum: 1 } }
  }}
])
// Look for a long-tail distribution where a small subset is far above median/p95

// Check index sizes
db.books.stats().indexSizes
// Large multikey index suggests outliers are bloating it

Reference: Outlier Pattern