Shallow Clones and CI Optimization

What is a Shallow Clone?

A shallow clone is a Git clone that downloads only a limited number of commits from the repository history, rather than the complete history. This is controlled by the --depth flag:

# Clone only the most recent commit
git clone --depth 1 https://github.com/example/large-repo.git

# Clone the last 5 commits
git clone --depth 5 https://github.com/example/large-repo.git

In a shallow clone, the .git/shallow file records the boundary commits — the oldest commits that were downloaded. Anything older than these commits is not present in the local repository.

Full clone:
  A - B - C - D - E - F - G - H (HEAD)
  ←───────────────────────────────→
  All commits downloaded

Shallow clone (--depth 3):
  A - B - C - D - E - F - G - H (HEAD)
                          ←───→
  Only G, H downloaded (plus E-F as boundary)

Full history (A→B→C→D→E→F→G→H)

ABCD

shallow boundary cutoff: main

Shallow --depth 3 (F→G→H only)

ABCM

BEF

Shallow --depth 1 (H only)

ABCE'F'

shallow boundary cutoff: feature

When Shallow Clones Are Useful

CI/CD Pipelines

This is the most common use case. In CI/CD, you typically only need the current commit to run tests, build artifacts, or deploy code. Downloading the full history of a large repository can add minutes to pipeline execution time.

# GitHub Actions - shallow clone (default since 2023)
- uses: actions/checkout@v4
  with:
    fetch-depth: 1  # Shallow clone

# For full history (when needed)
- uses: actions/checkout@v4
  with:
    fetch-depth: 0  # Full clone

Large Repositories

For repositories with extensive histories (e.g., the Linux kernel with over 1 million commits), a shallow clone can reduce clone time from minutes to seconds:

Quick Inspections

When you just need to look at the current state of a repository:

# Quickly check out a project to read the code
git clone --depth 1 https://github.com/example/project.git
cd project
# Read the code without waiting for full history

Single-Branch Clones

By default, git clone downloads all branches from the remote. The --single-branch flag limits the clone to only the branch you're checking out:

# Clone only the main branch
git clone --single-branch https://github.com/example/project.git

# Clone a specific branch
git clone --single-branch --branch develop https://github.com/example/project.git

Combined with shallow cloning:

# Clone only the latest commit of a specific branch
git clone --depth 1 --single-branch --branch main https://github.com/example/project.git

Limitations of Shallow Clones

Shallow clones come with several restrictions:

1. No Full History

# This will fail or give incomplete results
git log --oneline
# Only shows the shallow commits

git blame file.py
# May show "boundary" commits without details

git bisect
# Cannot bisect beyond the shallow boundary

2. Cannot Push from Shallow Clones (Historically)

Older versions of Git could not push from shallow clones. Modern Git (2.20+) allows this in many cases, but it's still limited:

# May fail with older Git versions
git push origin main

# Error: "shallow updates are not allowed"

3. No Tag Fetching by Default

Tags are not fetched in shallow clones unless explicitly requested:

# Fetch tags in a shallow clone
git fetch --tags --depth=1

4. Limited Merge and Rebase Operations

Operations that require full history may fail:

# May fail if the merge base is beyond the shallow boundary
git merge feature-branch

# May fail if rebase needs commits beyond the boundary
git rebase main

5. Cannot Create Patches Beyond Boundary

# May fail if the diff extends beyond shallow history
git format-patch HEAD~10

Unshallowing a Clone

If you started with a shallow clone and later need the full history:

# Fetch the complete history
git fetch --unshallow

# Fetch more history (increase depth)
git fetch --deepen=100    # Get 100 more commits
git fetch --deepen=50     # Get 50 more

# Fetch full history for a specific branch
git fetch --unshallow --depth=2147483647

Partial Clones

Introduced in Git 2.19, partial clones go beyond shallow clones by selectively fetching objects rather than commits. Instead of limiting the depth of history, you can exclude specific types of objects.

Blobless Clone

Downloads commit and tree objects but not blob objects (file contents). Blobs are fetched on-demand when you access a specific file:

# Clone without downloading file contents
git clone --filter=blob:none https://github.com/example/project.git

# File contents are downloaded when you access them
cat file.py          # Triggers on-demand download
git log -p file.py   # Downloads the blob for this commit

Treeless Clone

Downloads commit objects but not tree objects (directory structures):

# Clone without downloading directory structures
git clone --filter=tree:0 https://github.com/example/project.git

Combined Filters

# Combine filters (Git 2.32+)
git clone --filter=blob:none --filter=tree:0 https://github.com/example/project.git

Partial Clone vs Shallow Clone

Sparse Checkout + Shallow Clone

Sparse checkout allows you to check out only specific directories from a repository. Combined with shallow cloning, this is extremely efficient for monorepos:

Modern Sparse Checkout (Git 2.37+)

# Clone with sparse checkout enabled
git clone --depth 1 --filter=blob:none --sparse https://github.com/example/monorepo.git

cd monorepo

# Initially, only the root is checked out
ls
# README.md  (just root files)

# Add specific directories
git sparse-checkout add packages/frontend
git sparse-checkout add packages/backend

# List current sparse checkout configuration
git sparse-checkout list

# Set to specific directories
git sparse-checkout set packages/frontend docs

# Disable sparse checkout
git sparse-checkout disable

Combined Optimization

For maximum efficiency in CI/CD with a monorepo:

# Shallow + sparse + blobless
git clone --depth 1 --filter=blob:none --sparse \
    --branch main \
    https://github.com/example/monorepo.git

cd monorepo

# Only check out the directory needed for this job
git sparse-checkout set packages/frontend

# Now only frontend files are present
npm install
npm test

This combination can reduce clone size from gigabytes to megabytes for large monorepos.

CI/CD Optimization Best Practices

GitHub Actions

# Fast checkout for most jobs
- uses: actions/checkout@v4
  with:
    fetch-depth: 1

# Full history for jobs that need it (e.g., changelog generation)
- uses: actions/checkout@v4
  with:
    fetch-depth: 0

# Shallow checkout with LFS
- uses: actions/checkout@v4
  with:
    fetch-depth: 1
    lfs: true

GitLab CI

variables:
  GIT_DEPTH: 1  # Shallow clone depth for all jobs

test:
  script:
    - npm test

# Override for specific jobs
changelog:
  variables:
    GIT_DEPTH: 0
  script:
    - ./generate-changelog.sh

Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Checkout') {
            steps {
                checkout([
                    $class: 'GitSCM',
                    branches: [[name: '*/main']],
                    extensions: [[$class: 'CloneOption', depth: 1, shallow: true]],
                    userRemoteConfigs: [[url: 'https://github.com/example/project.git']]
                ])
            }
        }
    }
}

CircleCI

version: 2.1
jobs:
  build:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout:
          # CircleCI does shallow clones by default
          # Use --no-depth for full history

Bitbucket Pipelines

definitions:
  steps:
    - step:
        clone:
          depth: 1
        script:
          - npm test

Performance Comparison

Typical performance for a repository with 10,000 commits and 2 GB of history:

Note: These numbers are approximate and depend on network speed, repository structure, and server performance.

Common Pitfalls

1. Missing Tags

# Shallow clones don't fetch tags by default
git clone --depth 1 https://github.com/example/project.git
git tag
# (empty)

# Fix: fetch tags explicitly
git fetch --tags

2. Version Determination

Tools that determine version from git history (like git describe) may fail:

git describe --tags
# fatal: No tags can describe <commit-hash>

# Fix: fetch more history
git fetch --unshallow

3. Code Coverage and Diff Tools

Tools that compare against a base branch need full history:

# May fail with shallow clone
git diff origin/main...HEAD

# Fix: fetch the full branch history
git fetch origin main --unshallow

4. Submodule Interaction

Shallow clones with submodules require additional flags:

git clone --depth 1 --recurse-submodules --shallow-submodules \
    https://github.com/example/project.git

Summary

Shallow clones and partial clones are essential tools for optimizing Git operations in CI/CD pipelines and when working with large repositories. Choose the right combination of depth, filtering, and sparse checkout based on what your workflow actually needs. For most CI/CD jobs, --depth 1 provides the best balance of speed and functionality.