04. SHA-1 and Content-Addressable Storage

Capsule overview

In this capsule you'll master how Git uses SHA-1 hashing to build a content-addressable storage system. You'll learn how hashes are calculated, why content-addressing enables deduplication and integrity, and how Git is migrating to SHA-256.

SHA-1 is the foundation of Git's integrity.

What is SHA-1?

Definition:

SHA-1 = Secure Hash Algorithm 1
Input:  Any data (bytes)
Output: 40-character hexadecimal string (160 bits)

Properties:

1. Deterministic: Same input → Same output (always)
2. Fixed size: Always 40 hex chars
3. One-way: Can't reverse hash to content
4. Collision-resistant: Different input → Different output (mostly)
5. Avalanche: Small change → Completely different hash

How Git calculates SHA-1

Formula:

SHA-1("{type} {size}\0{content}")

Where:
- type: "blob", "tree", "commit", "tag"
- size: Content length in bytes
- \0: Null byte separator
- content: Actual data

Example:

# Content: "Hello Git"
# Git calculates: SHA-1("blob 9\0Hello Git")

echo -n "Hello Git" | git hash-object --stdin
# e51ca0d0b8c5b6e02473228bbf876ba000932e96

# Verify manually with openssl:
printf "blob 9\0Hello Git" | openssl sha1
# e51ca0d0b8c5b6e02473228bbf876ba000932e96 ← Same!

Content-Addressable Storage

Concept:

Traditional filesystem:
path → content
/home/user/file.txt → "Hello"

Git (content-addressable):
hash(content) → content
ce01362... → "Hello"

Benefits:

1. Deduplication:
   Same content → Same hash → Stored once
   
2. Integrity:
   hash(stored_content) must equal address
   If corrupted → hash won't match → detected
   
3. Immutability:
   Changing content changes hash
   Old version still accessible by old hash

Deduplication example:

# 100 files with same content
for i in {1..100}; do
  echo "Hello" > "file$i.txt"
done

git add .
git ls-files --stage | awk '{print $2}' | sort -u
# Only 1 unique SHA → 1 blob stored (not 100!)

Integrity verification

Git detects corruption:

# Manually corrupt an object
# (Don't do this in real repos!)

# Git checks:
git fsck
# If corrupted:
# error: sha1 mismatch for objects/ab/c123...
# error: corrupt object

SHA-1 collision concerns

Is SHA-1 broken?

2017: Google demonstrated SHA-1 collision (SHAttered)
Two different PDFs with same SHA-1

For Git:
- Collision is theoretically possible
- Practically extremely unlikely
- Git adds protections against known attacks
- Migration to SHA-256 in progress

SHA-256 migration:

# Git 2.29+ supports SHA-256
git init --object-format=sha256

# SHA-256 = 64-character hex (256 bits)
# Much stronger collision resistance

Plumbing commands for hashing

hash-object:

# Hash content (don't store)
echo "test" | git hash-object --stdin

# Hash and store
echo "test" | git hash-object -w --stdin

# Hash file
git hash-object file.txt

# Hash and store file
git hash-object -w file.txt

rev-parse (resolve refs):

git rev-parse HEAD
# Full SHA of HEAD commit

git rev-parse HEAD~3
# SHA of 3 commits back

git rev-parse main
# SHA of main branch tip

git rev-parse --short HEAD
# Short SHA (7 chars)

Exercises

Exercise 1: Manual hash calculation

# Create content
echo -n "Hello Git Internals" > test.txt

# Hash with Git
git hash-object test.txt
# Output: abc123...

# Hash manually (verify formula)
CONTENT="Hello Git Internals"
SIZE=${#CONTENT}
printf "blob ${SIZE}\0${CONTENT}" | openssl sha1
# Should match Git's hash

Exercise 2: Verify deduplication

git init dedup-test
cd dedup-test

# Create 3 files with same content
echo "Identical" > a.txt
echo "Identical" > b.txt
echo "Identical" > c.txt

git add .
git ls-files --stage
# All 3 files → same blob SHA

# Count actual objects
find .git/objects -type f | wc -l
# Only 1 blob (deduplication!)

Exercise 3: Integrity check

git init integrity-test
cd integrity-test

echo "Important data" > file.txt
git add . && git commit -m "Add data"

# Verify integrity
git fsck --full
# No errors

# Check object count
git count-objects -vH

Summary

In this capsule you learned:

✅ SHA-1 formula for Git objects
✅ Content-addressable storage concept
✅ Automatic deduplication
✅ Integrity verification via hashing
✅ SHA-256 migration future
✅ hash-object and rev-parse commands

Estimated time: 20 minutes

Next: 05. Refs, HEAD, and Branches