Oasis Capital in Conversation with Professor Tianqi Chen: From Zero to One, Believing in Interesting Things
Counselor Vitality
This time, we have Tianqi Chen, creator of TVM, MXNet, and XGBoost, and assistant professor in the Machine Learning Department and Computer Science Department at Carnegie Mellon University. His work has made it possible to natively deploy any large language model on virtually any hardware. Is compute still a problem? Enjoy.
Oasis Capital: Let's start with your recent research directions or what you're working on these days.
Prof. Chen: My research style is problem-driven. For the past five years, we've been focused on how to make machine learning accessible to more people and able to run in more device environments. Our recent research priorities are, first, machine learning systems — not just solving algorithms, but making the systems engineering itself iterate faster; and second, doing more open-source software so our work can live in the open-source community, letting people directly try our research and get feedback from industry and other fields through that process.
With the rise of Generative AI and large models, we also want to combine large models with our past accumulation to explore some new directions.
There's been a lot of progress in large model deployment lately. Machine learning compilation has gone from being relatively obscure to something PyTorch and major vendors are gradually exploring. The whole field is in an unpredictable state. It's like when deep learning was just taking off — the big data wave seemed to be ending, and no one knew what to do. Now it's another uncertain era, but that's a good thing.
Oasis Capital: Since large language models emerged, what's your latest work been focused on?
Prof. Chen: Over the past five years, one concentrated direction has been machine learning compilation. Machine learning engineering is becoming an increasingly large problem. To run on more efficient devices, we need to build reusable architectures rather than redoing systems engineering for every hardware platform.
Given the high memory consumption characteristics of large language models, a core focus of our recent work is how to use machine learning compilation techniques to speed up deployment, training, and underlying support. Based on this work, the MLC (Machine Learning Compilation) series of projects lets us deploy some language models on phones, mobile devices, or in browsers through WebAssembly and WebGPU technology, while also running faster on various GPUs including NVIDIA, AMD, and Apple.
When language models can be deployed across multiple devices, we can build more open solutions based on this technology, lowering the cost of deploying open models.
Oasis Capital: You've done a lot of interesting open-source work, from early Apache MXNet to XGBoost, TVM — how do you connect these pieces together? Over more than a decade of machine learning development, what are your reflections from both algorithm and systems perspectives?
Prof. Chen: Machine learning has changed enormously in recent years. From the early days of algorithm-centric modeling, deriving support vector machines and linear models; to after big data landed and was applied to advertising and recommendations, when people started thinking about scaling machine learning; to the beginning of the deep learning era, when data and compute entered everyone's consideration. This meant building good machine learning systems became indispensable. That's why I shifted to machine learning systems research, with the goal of comprehensively considering algorithms, compute, and data, finding the drive to solve problems in this process, and truly pushing the machine learning field forward.
Oasis Capital: NVIDIA has clearly benefited from the development of large language models. Your work will inevitably make hardware competition more diversified. How do you view the relationship between paths like NVIDIA and open-source solutions?
Prof. Chen: I don't think they're in competition — we also do a lot of optimization work for NVIDIA GPUs. Our solution can often leverage vendors' native libraries. NVIDIA is indeed leading in many aspects right now, but we're not trying to necessarily surpass NVIDIA, and NVIDIA isn't perfect in all scenarios. What we're interested in is how to make the whole field move forward faster, regardless of who's pushing it.
So this isn't a relationship where you must compare A versus B. With compute still relatively scarce, having more possibilities for everyone to move forward together is what we want to see.
Oasis Capital: Can you specifically talk about where NVIDIA is relatively leading?
Prof. Chen: Overall, it's hardware and programming models. The traditional approach of just stacking silicon is already difficult to solve without changing programming models. With new card releases, programming models can't directly migrate.
What people need now is a solution that can iterate quickly when new hardware environments and new models emerge. Why did deep learning develop so fast in the past decade? Because the barrier to deep learning modeling itself was lowered dramatically. In the next 5 to 10 years, machine learning engineering will become very important. For every possible combination of hardware, model, and data, specialized engineering solutions are needed. Our target of interest is how to make engineering iteration faster. In hardware iteration, NVIDIA is still unmatched, so how to better support new hardware is an interesting topic.
But there's one difference now — before, other vendors simply couldn't do it, which was equivalent to zero. Now through our solution, AMD went from not being able to run to being able to run, and running pretty well. From other vendors' perspectives, that's also progress. The process from 0 to 1 is invaluable.
Oasis Capital: Since deep learning barriers have become so low, can we foresee that large language model engineering barriers will also drop very low in the future?
Prof. Chen: It's like before I did XGBoost, data science was very complex, but now you basically tune a bit and get decent results. We hope to lower the barrier for machine learning engineering, and we've already achieved some results. But how low it can go, and when, is inseparable from research directions, research investment, and especially the collective efforts of the open-source community.
Oasis Capital: In this direction, collective open-source efforts might generate greater incentives, while closed-source leaders would prefer to keep their expertise "hidden away" as specialized technology?
Prof. Chen: Not talking about models, but from the machine learning engineering infrastructure perspective, both open and closed source will have promoting effects. Academia needs to keep up with the times and continue moving forward. Even for modeling, there are completely closed-source approaches, and there are at least open approaches like LLaMA. I personally believe more in open source — open-source communities iterate faster.
Oasis Capital: What challenges remain for running models on phones?
Prof. Chen: Our solution can directly run 7B models on phones. From 7B to 3B there's really no pressure — the solution will definitely mature. It also depends on whether you want to run it, or how large a model you want to run, and whether that size of model is useful — that's another question.
For 3B, our current solution can run directly. Go to MLC, we have an App Store app you can download and try. For 7B, the phone might heat up and energy consumption might be an issue, but high-end phones can run it. The next question is how to integrate with vertical applications, and whether it's necessary to run on phones at all, since tablets and laptops can too. The reason to run on phones is that people care about data not leaving the device.
Our current MLC solution lets people run on phones, tablets, Apple laptops, or in browsers, and of course on servers too — relatively flexible.
Oasis Capital: Since your solution lets people run models on phones, how does this path differ from what NVIDIA and other major companies are doing?
Prof. Chen: Why run on phones? Why run on the edge? Because I think there are different application scenarios. Suppose you're gaming — if you have to pay a dime for every sentence you say to an NPC, that's not impossible, but if you can directly complete decent tasks locally in certain scenarios, I think that would be welcome.
For example, to have a capable private assistant, you must maximally disclose personal information. Would you be willing to send this content to third parties? Is it more secure locally? And personalization — how to make language models understand you better? These directions can all evolve into different forms of applications.
When computers were first invented, there was a claim that the world only needed eight supercomputers in a few national labs to meet global computing needs, but personal computers emerged. If model barriers can be lowered further, will a Personal AI era emerge? That's also the direction we want to achieve.
Oasis Capital: If someone has tens of thousands of A100s training, from your perspective, what problems mainly require experienced infrastructure architects and systems people to solve?
Prof. Chen: Since I don't have large-scale training experience in school, my sense is that first, machine learning infrastructure itself is in its early stages, quite different from traditional data infrastructure. Getting many GPUs to work together for long periods — how to improve and make good use of utilization across different hardware — hardware optimization itself is also a cross-layer process involving hardware engineering and systems-level, and model-level coordination. There's definitely demand there. In the machine learning systems field, people are paying more and more attention, so talent in this area will also increase. I also hope to lower the barrier in this field, for example through machine learning compilation to reduce repetitive work. Engineering deployment demand has always existed; it's just that the level of attention has increased now. Our core feeling is that demand for talent and various aspects in this field is strong, which is naturally a good thing for machine learning and engineering deployment practitioners. We've also been thinking about how to achieve more with less through some automation technologies.
Oasis Capital: NVIDIA, AMD, and Apple each have their own architectures. What's the difference in how they run?
Prof. Chen: From the whole ecosystem perspective, NVIDIA is still relatively the best. AMD's main problem is software — different companies invest completely different effort in software. Getting the whole flow from hardware to model to application set up, from model to foundation to memory optimization, even the initial modeling-related infrastructure needs to be built. Through years of effort, we can now build a relatively compatible infrastructure based on compilation. We've also done AMD experiments before — for example, the flagship gaming card 7900 XT can run at roughly 80% of the 4090's performance. We hope to solve more software problems through automation. We can now run multi-GPU, for example two gaming cards together can run 70B, equivalent to the maximum being able to run LLaMA. Being able to play with all kinds of hardware is pretty exciting.
Apple is architecturally leading, especially the Ultra with its large memory. The simplest way to run LLaMA models directly on a machine is to buy Apple's latest M2 laptop. Our solution can also work on Apple.
Software problems require everyone's effort — as long as there's foundational architecture, problems can be solved. Of course, it still needs engineering. Our solution isn't limited to the inference side; it just requires opening up some thinking.
Oasis Capital: Some of your work, like XGBoost, has had enormous influence. How do you choose research problems?
Prof. Chen: What we focus on is how to integrate all the work together. The challenge of machine learning engineering isn't solving one specific problem, but rather having 10 solutions — this one for transformation, another for sparsity, another for batching, and possibly other solutions... This is why deep learning developed so fast, because its software engineering modularity is quite high.
To do ResNet, you don't need to care about detection, because you can naturally connect the detection head to ResNet's backbone. People doing optimization don't need to care what ResNet is — just write the optimizer well, take ResNet and connect it. Machine learning modeling is now modular and highly reusable, but machine learning engineering hasn't quite reached that level yet. So our recent focus is designing an infrastructure to solve various problems.
Oasis Capital: Some say you're essentially a creator of an entirely new discipline of machine learning compilation. What do you think of that?
Prof. Chen: It's not really an entirely new discipline — we have a lot of accumulation and investment in compilation itself. My personal research approach is problem-driven. The thinking at the time was exploring what methods to use to solve machine learning engineering problems, with so many hardware backends — how to leverage minimal capability to move this field? We felt automation was the necessary path, and compilation engineering was one of those paths. The definition of compilation itself is also continuously evolving. Our latest solution can integrate manual and automatic approaches to accelerate engineering iteration.
Oasis Capital: What's the outlook for tree-based models?
Prof. Chen: Tree-based models are still essential technical solutions in many industries. Among data scientists' top five tools, XGBoost still ranks among them. Every direction in the future will need tree models, and in the foreseeable future they're still very important, especially for tabular data, finance, and other fields — they're still used extensively.
Oasis Capital: Have you encountered any particular challenges in your research process?
Prof. Chen: Challenges always exist — research has no standard answer. Whether trying, failing, in industry, or in academia, there are all kinds of forward goals. It's all pretty intense (laughs)! But we very much enjoy being able to participate in pushing this field forward.
Oasis Capital: How do you judge whether your research direction is a relatively correct path?
Prof. Chen: You can't judge — you can only believe. Steve Jobs had a saying: The journey is the reward. Often the goal isn't the final trophy, but the process of walking the path.
As for whether something is correct or not — for example, before 2012 I started working on deep learning, which was a very correct yet wrong decision. The goal at the time was to use deep learning algorithms to solve ImageNet, but after two and a half years, there were no results. However, the accumulated experience could be carried over to other things to do in the future. The key is still "do what you find interesting."
Oasis Capital: Do you think you're a very intense person?
Prof. Chen: Doing interesting things means it doesn't feel intense — mainly enjoying the process (laughs).
Sustaining Vitality
What do you think is technological vitality?
Exploration in uncertainty — in the process of moving forward, you discover many interesting surprises. This is the proof that technology has vitality. —Prof. Tianqi Chen Machine Learning Department and Computer Science Department Carnegie Mellon University
