The Perils of Timezone — A Complete Visual Guide

The Trap

It works on your laptop. That is the whole problem.

Timezone bugs almost never show up where you write the code. They surface at 2 AM, near midnight, twice a year on a DST boundary, or the first time two services in different regions try to agree on the order of events. The instant is the same everywhere — the label is what betrays you.

One Instant · Five Clocks

14:30 IST, 09:00 UTC and 02:00 PDT are the same moment. Store the label instead of the instant and you can no longer order your own logs.

Where it bites

Naive local time

datetime.now(), new Date() without an offset, CURRENT_TIMESTAMP in a session-local TZ — each reads the machine’s clock, so the value means something different on every host.

The DST cliff

Twice a year a local wall-clock hour repeats or vanishes. Schedules double-fire or skip; “add 24 hours” lands an hour off; durations computed in local time silently gain or lose 3600 seconds.

Container defaults

A base image is UTC; one team bakes in TZ=Asia/Kolkata; another mounts the node’s zoneinfo. Now “the logs” are a mix of clocks and nobody remembers which service is in which.

Offset-less strings

2026-08-06 14:30:00 with no Z and no offset is not a timestamp — it is a riddle. Whoever parses it guesses a zone, and the guess is usually “mine.”

The midnight rollover

“Group by day” in local time puts the same UTC instant in two different buckets depending on where the report runs. Financial cut-offs and daily aggregates drift by a day near the boundary.

The classic bug. A job “runs daily at midnight” using the container clock. The container is UTC, the operator assumes IST, and the report for the 6th quietly contains 18:30 of the 5th through 18:30 of the 6th. It reconciles perfectly — against the wrong day.

# Same line of code. Two completely different values.
bad  = datetime.now()                      # naive — whatever the host clock says
good = datetime.now(timezone.utc)          # unambiguous instant, everywhere

# Serialise the instant, not the wall clock:
"2026-08-06 14:30:00"                     # a riddle
"2026-08-06T09:00:00Z"                    # ISO 8601, UTC, self-describing

The Discipline

One clock inside. Local time only at the glass.

The fix is not clever — it is boring, and boring is the point. Everything that stores, computes, or transports a time uses UTC. The instant is converted to a human’s timezone exactly once: at the moment it is rendered for that human. Local time is a presentation concern, never a storage format.

The Only Place Local Time Belongs

UTC flows up untouched. The dark box is the single conversion point — push it any lower and every layer beneath inherits a timezone it did not need.

The five rules

1. Store UTC, always. Databases, caches, message payloads, metric samples, log timestamps — all UTC. This is the one you never bend.
2. Transport UTC as ISO 8601 with a Z or explicit offset. A timestamp on the wire must describe its own zone. No bare YYYY-MM-DD HH:MM:SS.
3. Convert at the edge, once. The browser, the dashboard, the report renderer applies the viewer’s zone. Nothing upstream knows or cares that IST exists.
4. Keep the tz database fresh. Offsets and DST rules change by government decree. Ship tzdata and update it; never hardcode +05:30.
5. Compute durations on instants, not wall clocks. Subtract epochs. “Same time tomorrow” is a calendar operation done in the target zone — deliberately, not by adding 86400.

Store & compute

Instants, in UTC

An instant is a point on the universal timeline. It has no timezone — it is the same for everyone. Keep your data at this level and ordering, diffing and correlation are free.

Present

Wall-clock, in the viewer’s zone

“14:30 IST” is a human rendering of an instant for one audience. It is derived on demand and thrown away. Two users in two zones see two strings for one row — correctly.

The litmus test. Could two servers in two regions, and a browser in a third, all agree on the order and spacing of your events without a conversation? If yes, you stored instants. If no, a wall clock leaked into your data.

In the Cluster

Containers are UTC. Keep them that way.

Kubernetes hands you the correct default for free: every container starts in UTC. The tempting “fix” — inject TZ=Asia/Kolkata into every pod with a mutating policy — is the anti-pattern. It shifts log prefixes and naive code to IST while your metrics and traces stay UTC, so you have manufactured exactly the split-clock mess from Tab 1. IST goes at the edges instead.

IST at the Edges · UTC in the Core

Three edges speak IST to humans and the scheduler; the core keeps one clock. No pod’s TZ was harmed.

The tools, each at its layer

Scheduling → CronJob.spec.timeZone

Stable since Kubernetes 1.27. The job fires at IST wall-clock — DST-aware — while its container still runs in UTC. This is the right way to make “midnight IST” happen.

Dashboards → Grafana default_timezone

Grafana queries UTC and renders in IST per-org or per-user. The data behind the panel never moves; only the axis labels do.

UI → toLocaleString with an explicit zone

The browser converts on render. Always pass the zone explicitly — never trust the visitor’s machine to be in the zone your label claims.

Anti-pattern → cluster-wide TZ mutation

A Kyverno/admission policy stamping TZ=Asia/Kolkata on every pod looks tidy and quietly desynchronises logs from metrics from traces. Resist it.

# Scheduling: fire at IST wall-clock, container stays UTC (k8s ≥ 1.27)
apiVersion: batch/v1
kind: CronJob
spec:
  schedule: "0 0 * * *"        # 00:00 …
  timeZone: "Asia/Kolkata"    # … interpreted as IST, DST-aware

# Grafana: query UTC, display IST — no container clock touched
grafana.ini:
  users:
    default_timezone: ist

# Frontend: convert on render, zone always explicit
new Date(e.event_at).toLocaleString('en-IN', {
  timeZone: 'Asia/Kolkata', hour12: false,
}) + ' IST';

If you must go cluster-wide. With no policy engine installed and a modern (~1.34) cluster, the least-bad way to inject an env everywhere is a native MutatingAdmissionPolicy (CEL, no extra controller) — but understand you are overriding a correct default. Reserve it for genuinely display-only fleets, never for anything that stores or correlates time.

One-line rules of thumb

Situation	Do this
Persisting a timestamp	UTC. No exceptions.
Sending one over the wire	ISO 8601 with Z / offset.
Showing one to a person	Convert at render, zone explicit, suffix the label (IST).
Running a job at a local hour	CronJob.spec.timeZone — not a container clock.
Tempted to set container TZ	Don’t, unless the workload is display-only.
Computing a duration	Subtract instants (epochs), never wall clocks.

The whole guide in one sentence: keep one clock in the machine and let people wear their own.

Written after one too many “it was fine locally” debugging sessions. Store UTC, render local, and sleep through the DST weekend.