From Python to Go: Rewriting the Ingestion Pipeline

Our ingestion pipeline was born in Python in 2019. It started as a simple script that read from Kafka, validated JSON, and wrote to PostgreSQL. By 2024 it was a 40,000-line monolith with Celery workers, asyncio event loops, and enough metaprogramming that onboarding a new engineer took three weeks. We rewrote it in Go. The new system is 12,000 lines, handles 3x the throughput, and uses half the memory.

The decision was not ideological. We like Python for data science and scripting. But the pipeline is not a script. It is a long-running service that processes millions of events per hour, maintains strict ordering guarantees, and must recover from network partitions without data loss. Python's GIL and garbage collector were becoming architectural constraints, not implementation details.

The rewrite took six months with two engineers. We did not do a big-bang cutover. We ran both systems in parallel for three weeks, comparing output event-by-event. The Go system produced identical results but with p99 latency of 12ms instead of 180ms. The difference was memory layout. In Python we were serializing and deserializing dictionaries repeatedly. In Go we keep events as structs in a ring buffer and pass pointers. No JSON parsing on the hot path.

Error handling was the cultural adjustment. In Python we relied on exceptions bubbling up to a central handler. In Go every error is explicit. It is verbose — our error-checking density is about 15% of lines — but it forces you to think about failure at the point where it happens. We found three race conditions during the rewrite that had been latent in the Python version for years, masked by the GIL's implicit serialization.

"Go makes you write the error path. Python lets you pretend it does not exist. That pretense is expensive at scale."

The most surprising benefit was team velocity. New engineers are productive in Go within a week. The type system catches most integration errors at compile time. The standard library covers 90% of what we need — HTTP, JSON, TLS, concurrency — without version conflicts. We have not had a dependency resolution issue since the migration. In Python, pip conflicts were a monthly occurrence.

We kept Python for the ML inference layer. That is where it belongs. The Go pipeline calls the Python service over gRPC for feature extraction, then handles the rest itself. The boundary is clean and each language does what it does well.

If you are considering a similar rewrite, my advice is: do not rewrite for performance alone. Rewrite because the current system is preventing the team from moving fast. Performance is a side effect of good design.