SYSTEM DESIGN · QUESTION 108
How would you design a reliable large-file upload service?
On this page
I would clarify file sizes, resumability, access, retention, and processing requirements. A typical design authenticates the user, creates an upload session, and issues short-lived scoped storage URLs for multipart chunks. The client retries parts independently; the service verifies checksums, assembles or finalizes the object, scans it, and publishes metadata only after validation.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A large upload can fail after sending most of its bytes because a network disconnects. Restarting from the beginning wastes time. Multipart upload divides the file into parts that can be retried independently, while a session tracks which parts have safely arrived.
A USEFUL COMPARISON
Picture it this way
A moving company labels boxes and checks each one in separately. If one box is damaged, the entire move does not restart. A file upload session similarly tracks chunks and verifies the final package before making it available.
FOLLOW THE JOURNEY
Use resumable multipart transfers and validate before publication
Show upload-session creation, chunk transfer, verification, scanning, and completion.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Unvalidated content should not appear as ready.
A client requests an upload session, sends retryable parts to object storage, then the service verifies checksums and scans the assembled object before publishing metadata.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Unvalidated content should not appear as ready.
UNDER THE HOOD
How it works, step by step
Authorize upload
Check user, file policy, size, and destination.
Why it matters: Uploads consume storage and can carry risk.
Create session
Allocate an upload ID and short-lived scoped part URLs.
Why it matters: A session supports resumability without exposing broad credentials.
Transfer and retry parts
Upload chunks with checksums and per-part status.
Why it matters: A failed part can be resent without restarting everything.
Verify and publish
Complete multipart storage, scan content, and expose metadata.
Why it matters: Unvalidated content should not appear as ready.
OUTSIDE THE TEXTBOOK
A practical example
A user uploads a large video over an unstable mobile connection. The client stores an upload-session ID and completed part numbers, retries only failed parts, and resumes after reconnect. The server verifies the final checksum, scans the file, and then marks it available for transcoding.
GO A LITTLE DEEPER
Technical details that matter
- Use signed URLs scoped to one object and a short expiry; do not expose storage credentials.
- Define cleanup for abandoned multipart sessions and quotas for temporary storage.
- Checksum verification detects transfer corruption but does not establish that the file is safe.
MAKE A GOOD CHOICE
When is this useful?
- Large media, document, or dataset uploads.
- Unreliable networks where resumability matters.
- Direct-to-object-store transfers that reduce application-server bandwidth.
For small files, a conventional API upload can be simpler. A managed upload gateway may provide resumability and scanning controls.
WATCH OUT FOR
Common misunderstandings
- Trusting client-provided MIME type or filename.
- Publishing an object before malware scanning or integrity checks.
- Leaving abandoned multipart sessions and signed URLs active indefinitely.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Multipart state and cleanup add complexity.
- Direct storage upload requires careful authorization and CORS policy.
- Scanning and processing can delay availability.
WHAT TO REMEMBER
Key takeaways
- 1Use sessions and retryable chunks for resumability.
- 2Scope upload permissions and verify integrity.
- 3Scan and validate before publishing or processing.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do you resume after a disconnect?
Persist the upload session and completed part identifiers, then request or retry only missing parts.
Why use checksums?
They detect corruption or mismatch between expected and received content; they do not prove the file is safe or authorized.
SAY IT OUT LOUD
How to explain this in an interview
“I would create an authenticated upload session and use short-lived, scoped URLs for multipart chunks. The client retries parts independently and resumes from stored progress. On completion, the service verifies checksums, scans the object, and only then publishes metadata. I would also enforce quotas, clean up abandoned sessions, and restrict file access.”
Connect this question to the roadmap topic it supports.