How Should New Graduates Choose a Technical Specialization? Starting from ML Engineering Practice

A New Graduate's Real Dilemma

In a tech community, a developer with about one year of experience raised a highly representative question. His tech stack is quite solid: Python, FastAPI, Postgres, Kafka, and Kubernetes running on AWS EKS (with autoscaling), plus managed model API integration and responsibility for evaluation (eval) and monitoring.

This tech stack represents the mainstream production architecture for AI engineering today. FastAPI is one of the highest-performing async web frameworks in the Python ecosystem, built on Starlette and Pydantic with native OpenAPI documentation generation, making it particularly well-suited for building model inference API services. Kafka, as a distributed message queue, typically handles async inference request buffering, event-driven model evaluation pipelines, and similar roles in AI systems. AWS EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes service that eliminates the operational burden of the control plane, and paired with Cluster Autoscaler or Karpenter for node-level autoscaling, it's one of the standard choices for enterprise AI service deployment today.

In his own words: "I've done production-grade ML from the ops side, but I've always been a caller of models." He'd never gone deep "below that line" — no C++ experience, and aside from a local side project, no GPU programming either.

What makes this even more interesting is his candid admission that a large portion of his development work was AI-assisted, primarily using Claude. This workflow was "good enough to ship products," but it's precisely this that triggered a new anxiety: rather than continuing to expand breadth horizontally, he wants to build depth in a specific direction.

Behind this question lies the collective predicament of many young engineers today — when AI can quickly handle the "make it work" parts, where does your real moat lie?

The Gap Between "Callers" and "Builders"

This new graduate's self-assessment is extremely precise: he's a "caller of models," not a "builder" of the underlying model serving infrastructure. This dividing line is the most critical watershed in today's AI engineering landscape.

The Upper Layer: Applications & Operations (MLOps)

His current position falls within MLOps and the AI application layer. MLOps (Machine Learning Operations) as a concept took shape around 2019, with the core goal of bringing DevOps principles to the full lifecycle management of machine learning systems. According to Google's classic paper Hidden Technical Debt in Machine Learning Systems, the actual model code in a real ML system is only a small fraction — surrounding it is a massive amount of engineering work including data collection, feature engineering, configuration management, monitoring, and serving infrastructure.

The core capabilities at this layer include:

• System Integration: Connecting managed model APIs into actual business workflows
• Reliability Engineering: Kubernetes orchestration, autoscaling, service monitoring
• Evaluation Systems: Building eval and monitoring to ensure model output quality

The eval component has become particularly important in the LLM era, because large language model output quality is difficult to measure with traditional metrics like accuracy or F1-score. It requires building a complete quality assurance system incorporating automated evaluation (such as LLM-as-judge), human feedback, and A/B testing.

These capabilities are far from trivial. In fact, as large model APIs become widespread, "how to reliably run models in production" is itself a scarce skill. He has already independently shipped and operated such a system — a complete end-to-end experience that many senior engineers may not possess.

The Lower Layer: Model Serving

The "serving side" he aspires to refers to the underlying implementation of model inference services — including inference engine optimization, GPU resource scheduling, operator implementation, memory management, quantization, and batching. This layer typically requires C++, CUDA, and other hardware-adjacent skills, representing a clear gap from his existing Python + Kubernetes stack.

The key technologies at this layer deserve deeper understanding. KV Cache is a core optimization in Transformer inference — during autoregressive generation, the model needs to compute attention with all previous tokens for each new token generated. KV Cache stores previously computed Key and Value tensors to avoid redundant computation, but this creates enormous memory management challenges, as a single long-sequence request can consume several GB of GPU memory. Continuous Batching, first systematically proposed in the Orca paper, allows requests of different lengths to dynamically join and leave inference batches. Compared to traditional static batching, it can improve throughput by several times. Tensor Parallelism splits a single model layer's weight matrices across multiple GPUs for parallel computation, suitable for scenarios where model parameters exceed single-GPU memory capacity, requiring precise management of inter-GPU communication overhead.

Is Kubernetes Experience Actually an Advantage?

One core question he raised: Is Kubernetes and autoscaling experience a real springboard into model serving, or is it an entirely different skill set?

The answer: It's both.

From a systems perspective, his container orchestration, resource scheduling, and observability experience is a genuine advantage. Model serving in production equally requires deployment, scaling, and monitoring — this knowledge transfers directly. Understanding how an inference service scales horizontally in a cluster and handles traffic spikes is itself an important part of serving work. In GPU cluster scenarios specifically, scheduling complexity far exceeds CPU clusters — GPU nodes are expensive, slow to start, and different models (A100, H100, L40S) suit different workloads. Achieving efficient GPU resource multiplexing and intelligent scheduling is one of the core challenges in model serving infrastructure.

But from the inference kernel perspective, it truly is a different skill set. GPU memory management, KV Cache optimization, continuous batching, and tensor parallelism are computation-intensive low-level engineering with little relation to Kubernetes' "orchestration mindset." This requires understanding GPU SM (Streaming Multiprocessor) architecture, CUDA's thread hierarchy model, memory bandwidth bottlenecks, and other hardware-level knowledge.

In other words, his existing experience allows him to quickly ramp up on the "infrastructure side of model serving," but if the goal is "inference engine kernel development," he'll need to build up the C++/CUDA skill track from scratch.

Is C++ Really a Necessity?

Regarding "how much C++ do I actually need," the realistic answer is layered:

Most Positions: Python Is Sufficient

If the target is the application and orchestration layer of model serving, Python remains the primary language. Mainstream inference frameworks like vLLM, TGI, and Triton all provide comprehensive Python interfaces. Day-to-day work involves configuration, tuning, and integration rather than hand-writing low-level operators.

These frameworks each have distinct characteristics: vLLM is a high-performance LLM inference engine developed at UC Berkeley. Its core innovation, PagedAttention, borrows the paging concept from operating system virtual memory to manage KV Cache, improving GPU memory utilization by 2-4x and greatly alleviating memory fragmentation. TGI (Text Generation Inference) is Hugging Face's production-grade inference service, deeply integrated with the HF model ecosystem, supporting one-click deployment. NVIDIA Triton Inference Server is a general-purpose model serving platform supporting multiple backends including TensorRT, ONNX Runtime, and PyTorch, with enterprise features like dynamic batching, model ensemble, and multi-model GPU sharing. The Python interface layer of these frameworks handles request routing, scheduling policies, and metrics reporting, while the actual computation cores are implemented in C++/CUDA kernels.

A Few Core Positions: C++/CUDA Is Indispensable

Only when you truly dive into inference engine kernels, custom operators, and extreme performance optimization do C++ and CUDA become hard requirements. These positions are relatively few in number but have extremely high technical barriers, representing truly scarce depth specializations. Typical work includes: writing efficient attention operators for new model architectures (e.g., implementing FlashAttention requires mastery of CUDA shared memory management and thread synchronization), implementing new quantization schemes (e.g., CUDA kernels for GPTQ, AWQ), and performing extreme optimization for specific hardware (e.g., NVIDIA Hopper architecture's TMA units).

For this new graduate, a pragmatic path would be: first establish a solid footing in Python-dominant model serving positions, leveraging existing Kubernetes advantages to break in, then gradually penetrate deeper layers based on interest.

Why Does Deep Specialization Matter More in the Age of AI-Assisted Programming?

The most thought-provoking aspect of this question lies in his reflection on AI-assisted development. He explicitly stated that precisely because so much work can be accomplished with Claude's assistance — "good enough to ship" — he realized the need to build genuine depth in a specific direction.

AI-assisted programming is profoundly changing the production methods of software engineering. According to GitHub Copilot data, AI-suggested code accepted by developers accounts for approximately 30-40% of total code. Large models like Claude and GPT-4 are already capable enough at code generation to handle vast amounts of standardized CRUD logic, API integrations, and configuration writing. The evolution Andrej Karpathy described from "Software 1.0" (hand-written rules) to "Software 2.0" (neural networks learning rules) is further developing into a natural language-driven "Software 3.0" paradigm.

This is an extremely astute judgment. When AI can rapidly produce "working" code, those general skills easily replaceable by AI are seeing their market value diluted. What's truly difficult to replace is precisely:

• Deep domain knowledge: Understanding why a system is designed a certain way, not just making it run. For example, understanding why vLLM chose paged memory management over contiguous allocation, or why pipeline parallelism outperforms tensor parallelism in certain scenarios.
• Mastery of underlying principles: Being able to diagnose thorny problems that AI can't answer. When inference latency spikes unexpectedly, being able to systematically investigate across GPU utilization, memory bandwidth, PCIe bottlenecks, and other dimensions.
• Systemic engineering judgment: Making correct decisions amid complex trade-offs. For example, finding the optimal balance between latency, throughput, cost, and reliability.

In other words, the AI era actually amplifies the value of "specialization." Breadth can be supplemented by tools; depth requires long-term investment to build a moat. This phenomenon can be explained in economic terms as a "skill premium" — when the supply of basic skills dramatically increases due to AI assistance, the relative scarcity and market premium of differentiated deep skills actually increases.

Specialization: Planned or Serendipitous?

He also asked a question many people are curious about: "How did you end up in your field — was it planned or accidental?"

From the experience of numerous practitioners, most specialization directions are actually the product of "half-planned, half-serendipitous" circumstances. Very few people start out determined to do inference optimization or distributed training. More commonly, they encountered certain types of problems in specific projects, developed interest, and naturally went deeper. This aligns closely with the "Planned Happenstance" theory in cognitive psychology — the best career development often comes from being prepared to seize chance opportunities.

Advice for this new graduate:

1. Extend along existing strengths: Kubernetes + model serving infrastructure is a natural entry point with the lowest risk. Start by deploying and operating inference frameworks like vLLM/TGI, gradually understanding their internal mechanisms.
2. Test the waters with side projects: He already has experience with a local GPU project. He can continue using small projects to explore lower layers and determine whether he truly enjoys this type of work. For example, try implementing a simple matrix multiplication in CUDA, or contribute a performance improvement PR to an open-source inference framework.
3. Don't lock in too early: One year of experience is still very early stage. A specialization direction can gradually crystallize over the next two to three years. The technology landscape changes rapidly — in 2022, LLM inference optimization was an extremely niche direction, and today it's a hot track. Maintaining learning ability itself is the most important "meta-skill."

Conclusion: Breadth as Foundation, Depth as Identity

This new graduate's dilemma is fundamentally a question every modern engineer must face — in an era of increasingly powerful tools, how do you define your irreplaceable value?

He already possesses solid engineering breadth and complete production experience — that's a valuable foundation. What comes next isn't spreading thinner, but finding that deep direction that aligns with both his interests and his existing strengths. For him, the infrastructure side of model serving may be the most natural starting point, while whether to go all the way down into the C++/CUDA kernel world can be left for time and practice to answer.

In this era where AI can help write 80% of your code, real competitiveness lies in the remaining 20% — the knowledge and capabilities that require deep understanding, independent judgment, and can't be elicited by simple prompts. This isn't a rejection of AI tools, but a redefinition of the human engineer's value.