NVIDIA’s Marc Spieler: AI, Data Centres, and Energy

The podcast opens with updates on the closure of the Strait of Hormuz, a German state-owned energy company contracting for Canadian West Coast LNG, and the Pope’s theological document warning about AI.

Next, Peter and Jackie introduce this week’s guest, Marc Spieler, Senior Managing Director for the Global Energy Industry at NVIDIA, joining from Houston, Texas, to discuss the latest developments at the intersection of AI and energy.

Energy and AI are deeply interlinked. Energy companies are using AI to improve efficiency across oil and gas, renewables, and emerging sources such as next-generation fission and fusion. At the same time, AI’s explosive growth is driving significant new electricity demand, requiring a build-out of both generation and grid infrastructure.

Predicting future power demand from AI remains uncertain; it depends on the pace of adoption and whether GPUs, along with other delivery components of the digital infrastructure stack, will become more efficient over time. Marc highlights that data centres are becoming more flexible, with the ability to reduce consumption during periods of grid stress. This would allow new data centre capacity to be added without straining the grid, while also lowering costs for all power consumers by improving system utilization during off-peak periods.

Content referenced in this podcast:

Please review our disclaimer at: https://www.arcenergyinstitute.com/disclaimer/

Check us out on social media:

X (Twitter): @arcenergyinst
LinkedIn: @ARC Energy Research Institute

Subscribe to ARC Energy Ideas Podcast
Apple Podcasts
Amazon Music
Spotify

Episode 328 transcript

Disclosure:

The information and opinions presented in this ARC Energy Ideas podcast are provided for informational purposes only and are subject to the disclaimer link in the show notes.

Announcer:

This is the ARC Energy Ideas podcast, with Peter Tertzakian and Jackie Forrest, exploring trends that influence the energy business.

Jackie Forrest:

Welcome to the ARC Energy Ideas Podcast. I’m Jackie Forrest.

Peter Tertzakian:

And I’m Peter Tertzakian. Welcome back. Well, Jackie, I’m getting fatigued of doing this timestamp stuff. I wish the world would just slow down a bit, but I guess we have to do it given the news of the day is always moving markets and doing things, and we’ve got the price of oil with the West Texas Intermediate down to $87. It’s Friday, May 29th, and we’re awaiting more news on a potential ceasefire out of Iran, maybe, maybe not. But what is the situation that’s currently going on on the other side of the world?

Jackie Forrest:

Yeah, I thought we’ll talk about Iran quickly and a few other news items here from Canada. Well, of course there’s mixed signals here. There’s reports that maybe there will be a deal, and if so, it would be a 60 day opening of the Strait of Hormuz and Iran would clear the mines, but it’s mixed signals because at the same time we’re hearing that Iran hasn’t signed off on this, Donald Trump hasn’t signed off on this. So I hope it does get resolved, obviously. Inventory levels are getting quite low and I do think if this goes on for another month, maybe we’ll start to see actual shutdown of industry and things like that. So I’m hopeful that we see a change. So that’s the Iran situation. I did want to also talk about the news that a German state owned energy company is buying gas from the proposed Ksi Lisims LNG project on the west coast of Canada.

Peter Tertzakian:

So the proposed Ksi Lisims is 12 million tons per annum. The LNG one was 14. The German off take agreement I think is one million tons, which was not large, but it’s I think a good signal again that there’s a lot of international interest in coming to Canada. It seems like a long way for an LNG tanker to go from the west coast of BC all the way around. I guess it would go through the Panama Canal and through the Atlantic to Northern Germany through the… I guess yeah. I don’t know.

Jackie Forrest:

Smaller ships could go through the Panama Canal. I will say that these cargoes can be swapped.

Peter Tertzakian:

Yeah, they get traded. They get traded.

Jackie Forrest:

It never rests. But to me, if I went back pre-2022, I would’ve said the chances of a German company wanting to buy gas off the west coast of Canada, zero. But I think this shows the advantage of Canada’s geopolitical and geographic location at this time where people are looking for diversity of supply. And I did want to have a quote from our energy minister, Tim Hodgson. “We’re in a world where our allies are begging us, they are begging us to produce our resources.” So this is Canada’s time. This is our time to lead.

Peter Tertzakian:

Yeah, it is Canada’s time, and it’s good to see the German interests. The Germans did come to Canada a couple years ago and said, “We want your LNG.” And we offered them hydrogen even though we didn’t have hydrogen, nor did we have LNG. So anyway, here we are. It’s all seemingly coming to pass. It’s all about energy.

Now, speaking of energy, I was at an AI conference last week, I think I mentioned that, or I guess it’s two weeks ago, I’m losing track of time, and made the statement that AI is an energy play. Certainly from the perspective of energy analysts, we view it as an energy play because 100% of the electrical power that goes into a data center is converted into heat basically. And then you have to cool it on top of that. I mean, this is not really novel in terms of the thinking certainly within the AI world or the energy world, but for a lot of people, it puts it into perspective that, okay, yeah, from an AI practitioner’s perspective, it’s all about information processing and accelerating towards, say, artificial general intelligence and all that. But from an energy perspective, it’s all about electrons and molecules getting into the data centers.

And I think everyone in the conference, and certainly myself as a AI practitioner, sees that there is a massive step change that has occurred even in the last two, three months in terms of the capabilities of these systems, these platforms such as Claude and OpenAI, et cetera, Gemini, in vibe coding and agents spinning out agents that run overnight. And so what has gone from being simple prompts to things that are running and consuming energy. So as an energy play, let’s talk about it.

Jackie Forrest:

Well, hey, by the way, it’s so mainstream that not only we’re talking about it, but the Pope is talking about it.

Peter Tertzakian:

The Pope is talking about it. So we’ve got to talk about the Pope and his views, which are quite fascinating actually, at the end of this podcast. So spoiler alert, we’re going to talk politics, religion, and AI, but I think it’s time that we brought on someone who really knows about the inner workings of these data centers. We have had data center constructors and developers before, but we’ve never had someone who actually knows about the inner workings of semiconductors, graphic processing units, tensor processing units, et cetera, et cetera. So we are delighted to have with us from Houston, Texas, Marc Spieler, senior managing director of the Energy Industries from NVIDIA. Welcome, Marc.

Marc Spieler:

Hi, great to be here. Thanks for having me.

Jackie Forrest:

Okay, Marc. Well, we’re really excited about this conversation because we can’t get enough of data centers in terms of questions around energy use. So it’s great to have you here. Tell us a bit about your background and your role at NVIDIA.

Marc Spieler:

Sure. So I joined NVIDIA about seven years ago to head up their global energy business, which was not a new business for NVIDIA. NVIDIA has been in the energy business since the early 2000s. I spent 13 years at a large energy services company before that, and then prior to that was at a technology company covering the energy industry. So my entire career has been at the intersection of technology and energy, primarily oil and gas up until NVIDIA. And now I’ve become deeply engaged in the power space, not only from a power generation perspective, but also transmission and distribution. So it’s becoming quite evident, I’m sure, to everybody that the electrification of the world is a real thing. And I think as we try to do that, not only in the AI world, but also electric vehicles and home appliances and everything else, that the need for energy is increasing tremendously.

And at NVIDIA, we go to market by industry, and so we try to figure out the world’s most challenging problems in those industries and how accelerated computing can solve those. So that’s what I think.

Peter Tertzakian:

Yeah. No, that’s amazing. And NVIDIA is huge. I mean, it’s like four or five trillion dollars. I lose track of market capitalization. And so when you say you sell to industry, certainly for me, it conjures up, as I said, these very advanced, sophisticated semiconductors. What else does NVIDIA sell? Does it sell packaged systems and advisory services? Or what is the overall set of offerings?

Marc Spieler:

Sure. So at our core we’re an engineering company. We don’t actually manufacture anything. We’ve got partners that manufacture our silicon chips. We’ve got manufacturers that turn those into GPUs or boards or other types of processing units. And then we’ve got partners like cloud providers or Dell HP and other OEMs that turn them into servers. NVIDIA designs and engineers infrastructure, but the biggest thing we are is a software company. And we produce more software, more open source software than anybody else. And the reason we do that is because that’s really what makes a significant difference in how data is processed. And we do it in a parallel way, and in order to the outcomes that we do, it’s important that that software stack exists.

Peter Tertzakian:

Right. So let’s talk about the software and actually let’s just talk about the basics. AI is really, this term, artificial intelligence, is nothing new. The term was coined in 1955 by a fellow named John McCarthy at Dartmouth College. I mean, maybe Jackie, you don’t know, but I mean, I started out as computer science when I went to school and then flipped into the energy business by taking a few geology courses. But in the 1980s, the term was popularized under what was called the expert systems. That’s what it used to be called, and was championed by the likes of the Digital Equipment Corporation, which was a huge company back then for those of us who remember. So given that AI has been evolving and people have been talking about it in the background, what is it in the last three years that has really transformed and made this term and everything underneath it leap to the next level? What is a systematic change?

Marc Spieler:

Well, I believe that it’s now commercially viable. I think we saw this a couple years ago with the introduction of OpenAI is ChatGPT, where you could actually take the hardware infrastructure, the software stacks, and the power that’s required because it’s a manufacturing process where you take electrons and data, and you convert them into tokens or knowledge. And now for the first time, I think we’ve been able to do that in such an efficient manner that it’s commercially viable and you can use this intelligence to basically help augment humans in the way that they work at an affordable price point.

Jackie Forrest:

Mm-hmm. Well, and the chat was so accessible. You just go in and ask it things, and anyone can get started. I don’t know if I shared this yet, Peter, but I just got Claude installed a couple days ago, and so far my productivity has not been better. It’s like, I need an MD file and I need to set this up and then I’m using the wrong thing and then I put it to the wrong sandbox file. But eventually there’s more of a learning curve, but I do think eventually I’ll get more productive because of it. It’s not as easy to use, but now we’re starting to get these tools that people like myself who don’t have any background in software engineering can apply. So I’ll agree with that. But when you think about where this is all going, Marc, and we’ll use that baseball question, like what inning are we in terms of achieving the full potential of AI?

When I see what, even what I’ve experienced in the last year from starting with just like entering questions into a chat to now this more advanced thing, where are we?

Marc Spieler:

I’d say we’re just at the beginning of the game. We probably just passed the national anthem. We’re building out more stadiums, we’re setting up teams, we’re recruiting players, and I think those organizations that are building out those teams are going to have an incredible opportunity to advance the fields in which they work. The capabilities of AI is going to continue to accelerate. We started off with just the ability to interpret pictures and words and those things, and as you indicated, simple prompts. Now we’ve got agentic workflows. We’re going to be able to look at different multimodal applications like pictures and videos and different sensors, and be able to integrate those all into outcomes that couldn’t be done in the past at the speed which the world’s moving so early in our journey.

Peter Tertzakian:

Yeah. And is that a consequence of the scale of the data centers that are being busing? And maybe you can talk about the difference between the learning and training in these large language models, which in my mind it’s these gigawatt hours of energy coming in and training basically on the entire knowledge that’s embedded in the internet to create models that we can interact with, and then the inference. Can you just talk about those two terms like training versus inference and where things are at?

Marc Spieler:

Sure. So training is basically taking all of this data and converting it into models that can then later be leveraged to ask questions to or to be able to run data through and come back with outcomes. So think of it like training is teaching kids in college. So you’re going to spend four years training a student and when they come out, they should be able to answer some general questions about the subjects that they studied and have good insight and everything else. And then as you continue to fine tune or to provide additional data to those students, they will get smarter and smarter, and eventually become experts in their field. And it just comes down to your capacity to continue to develop those models.

Peter Tertzakian:

So in this metaphor, I mean, are we at the cusp now of graduation and that the inference is starting to, now that the learning phase, I mean, it’s always learning, but that now we’re shifting the emphasis to understanding how to get an inference out of these models and the data centers that they are housed in? And maybe also speak to the different types of hardware and the energy consequences of inference versus learning, because I know that there’s differences in that.

Marc Spieler:

Sure. So I would say that we’ve got some great models out there in the market today, and we continue to get bigger and bigger models that take advantage of all the new data that’s constantly being created. So data and electrons in, models out, and then inferencing is asking those questions. And what I would say is between the infrastructure that’s been developed as well as the software, over the last 10 years, we’ve seen about a 3,000X improvement in training and about a 45,000X improvement in inference performance per watt. So think about what it would take 10 years ago to run some of the models. You couldn’t afford it, you couldn’t build it. It wasn’t existing.

And when I talk about commercial viability and capabilities, Jensen often describes it as this five layer cake, right? You have energy, then you have the chips, you have the overall infrastructure of the data centers, then you have these great models that you’re developing. And lastly, the applications that are built on top of it. And without optimizing all five layers, you can’t get the performance per watt or the performance per dollar that is going to be expected moving forward. This efficiency is critical for the speed and the energy efficiency in which we roll AI out around the world.

Jackie Forrest:

Okay. Well, we’re going to talk a little bit more about the energy use and efficiency improvements, but before we go to that, let’s talk about how it’s being used today, even if it is the anthem being sung as opposed to even the first inning. I know you had some experience in the oil and gas business before coming to NVIDIA. So in the ’80s, oil and gas companies were big users of mainframe computers, and when you think about seismic data and some of that, those were very computationally intense work that was done. What today is some examples of how the upstream oil and gas producers are using this?

Marc Spieler:

Sure. So you’re absolutely right. The amount of data that upstream oil and gas companies acquire just to understand the subsurface, you shoot seismic data and you eventually try to look for oil and gas reserves. So they continue to use huge supercomputers and accelerated computing to do that, and that’s general high performance computing. What’s changed now is they’re now using AI and different AI models to interpret that data and be able to look for patterns or find the subsurface outcomes that typically require hours and hours, months of humans looking through this data in order to find these patterns, whether they’re geophysicists or geologists or others, now they’re using AI to augment that. So the standard boring work that these folks did in the past to look for these things now can be automated through AI. And these are some of the projects we’re working on with SLB and other ecosystem partners.

Peter Tertzakian:

Like Schlumberger. That’s Schlumberger?

Marc Spieler:

Schlumberger, yeah. And so some of the big seismic companies are going down this path as well as the large operators and even smaller operators.

Jackie Forrest:

I actually attended CERAWeek in Houston. I don’t know if you had a chance to attend that this year, but there was actually talk about use of it in operations. Is that pretty widespread? And what have you seen there?

Marc Spieler:

Yeah. So more and more, we’re looking at things on the surface as well, especially in areas where safety is paramount. How do we use AI to understand what’s going on in those environments, help people to make better decisions and more informed decisions by looking at previous outcomes and data. So one of the examples we use is we’ve created what’s called PetroNeMo with Petrobras down in Brazil. And basically they’ve created a model that allows them to look at previous maintenance data on their rigs, and basically help the employees that are working on those rigs to have a better understanding of what’s going on on that rig and previous outcomes and next best actions. So once again, it’s starting to tune those domain specific models to get better outcomes for the employees working in those operational environments.

Peter Tertzakian:

Yeah. Now in the 1980s, we talked about mainframe computers and computational intensity, and it was really the domain of the large multinationals like Shell and Exxon and Chevron, where I worked, and smaller companies didn’t have the means to buy big mainframes and run them. So there was definitely a capability division. And then in the ’90s came workstations and PCs and things and it equalized. Is it… You’re talking about the big Schlumbergers and the companies that are able to do now really the next level of advanced geological interpretation with these systems. Talk about the divergence in the size of energy companies that are able to really take advantage of these systems. Is there a divide now that may ultimately be lessened? And then there’s, of course, the adoption divide. Some adopting this tech and others are laggard. So maybe talk about overall adoption or even ability to adopt based on financial means.

Marc Spieler:

Sure. So I think that’s evolved quite a bit. If you look at some of the large seismic companies that existed and still exist, the Veridians, former CGG, TGS, Schlumberger, the initial cloud providers, if you think about it, when oil and gas companies needed large volumes of data processed, only a few of them could afford those large supercomputers as you said, everybody else outsourced their data processing to these companies. Today you have the major cloud providers. And so when you want to process large sums of data, you can actually go to a cloud provider and you can process that and rent the system for the time you need. And so I think they’ve done a great job of commoditizing the ability for companies of all sizes to access that. And what you would see is that more and more software companies are evolving and creating these AI capabilities so that they can provide it as a service. So no matter how small you are, you should be able to access this.

Peter Tertzakian:

I mean, there’s a lot of talk right now about the constraints and compute because so many people are using this, and the cost of computation. In other words, the cost of tokens are going to go up. Is that going to be also a problem for those who can afford it versus those who can’t? How much inflation do you see in the cost of token and the cost of usage in this stuff?

Marc Spieler:

I anticipate that the cost of tokens will continue to get better and better. Every generation of our infrastructure continues to get less expensive per unit of outcome. I think what you’re seeing here is what’s called Jevons Law. The more affordable things become or the lower the cost, the more people use. And so what we found is that a lot of times when costs were higher, you would find certain use cases that justified that cost. As the cost continues to drop, you’re going to find more and more use cases.

Peter Tertzakian:

So maybe there’s a near term surge because everybody is rushing to use this stuff so the price goes up, but longer term you would expect it to go down.

Marc Spieler:

Yeah. I think once again, the constraints on the system is our ability to build these AI factories and provide the power that’s required. And so you see significant amounts of new infrastructure being built and the need for power and the interconnection in order to hook up to the grid and get the power that you need is taking a lot longer than what we’d hoped. And so we’re working with a lot of these energy companies now to solve that problem using AI. How do we do the studies? How do we do the manual processes that typically goes into assessing the grid and being able to get these large loads interconnected much faster?

Jackie Forrest:

Okay. Well, let’s come to some of the power constraints in a bit, but I did want to come back to adoption and talk to you a little bit about the companies. There’s something called technology debt. In other words, companies have legacy information and systems and that prevents them from really adopting AI fully. For instance, if you have all your files and stuff in a way that AI can’t access them, to what extent do you think companies need to rethink their whole way of organizing data to get the most out of AI? Is that the first step in terms of becoming really efficient?

Marc Spieler:

Yeah. I think that there’s no industry out there that I’m aware of. There’s some close ones with healthcare and others that have as much data as energy companies. We collect data tremendously and we save it all. And you’re right, sometimes it’s siloed and it’s not easily available or we don’t have the great metadata. There are AI tools to help solve that and curate that data and get them into a format that can be ingested and used to train AI models or to inference by leveraging databases of that data and access it. What I would say is the cost is not small for companies that have tremendous amounts of data, but the value is significant. And so what we’re seeing is more and more companies starting projects that will prove out the business case to move more and more of their data into AI friendly storage, and then being able to process that data and leverage it.

I think this industry is typically slow-moving and cautious. I think we are seeing them move at a pace that they haven’t done in the past because they’re seeing the outcomes and the capabilities that are potential. Plus it’s a very cyclical market. And so a lot of the employees that have that knowledge in their heads have either retired or been caught up in downturns in the industry, and the ability to have their current employees or new employees in the future access and be able to leverage this information faster is going to result in companies getting the outcomes that they’re promising to their shareholders and being able to exceed their expectations with the employee base that they have.

Jackie Forrest:

Okay. Well, that comes to another issue though, just there’s already been obviously a lot of headcount reductions in the energy industry, especially oil and gas, and there’s a lot of concerns around how AI is going to replace more people. Are you seeing job losses in the companies that you’re working with because of the efficiencies they’re gaining?

Marc Spieler:

No, not at all. Not due to AI. Unfortunately, I think a lot of companies right now are struggling to find the domain expertise that they may have let go during former downturns, and then their ability to leverage AI to become more efficient. There’s really two ways to look at AI as a leader in these companies. One is to reduce costs. Can I do more with less people? And the other is, can I get greater outcomes with the people that I have? And so I think those companies, and we see a lot of them, those are the early innovators in this early part of the game that are looking at it and not saying, “How do I reduce headcount?” But, “If I’m a 30,000 person company, how do I deliver outcomes of 50,000 versus how do I deliver the same with 20,000?” There’s one is a cost savings measure and one is an outcomes or a revenue or profit focus.

And those companies who are going to win in the industry are really focused at how do I achieve greater outcomes and do more with my existing staff than reduce the staff.

Peter Tertzakian:

Right. Well, that’s oil and gas, and this was, I think you said it, it’s one of the most data intensive industries in the world. In fact, as I recall, the only institutions that had more computational and data intensity was the weather office and obviously the military, but AI has obvious applications in renewable energy as well. All the energy systems are data intense. So what are you seeing, whether it’s the renewable primary energy sources like wind and solar all the way down to batteries, geothermal, hydro, grid management and so on?

Marc Spieler:

All of the above. Power generation is a huge area. And a lot of our work started outside of oil and gas with areas like wind and how do we do wind wake optimization or to optimize entire wind farms, and traditionally those are high performance computing workloads, but after you’ve done a lot of those simulations, you can actually create surrogate models, and using AI you can actually simulate what the wind will do and this has allowed us to optimize turbine placement. We’re doing the same thing with solar right now and solar prediction models to optimize next business day forecast accuracy. We combine that with Earth-2, which is our weather digital twin of the earth, and really that helps us to predict the outcomes of renewables in a way that we can plan for it, which of course reduces our dependency on fossil fuels because we can actually plan accordingly.

Peter Tertzakian:

Yeah. Wait, when you say we, I just want to understand the relationship between NVIDIA and, say, some of these companies that are using it. So you said like you’re largely a software company, but are you actually going… Is NVIDIA actually going in and working shoulder to shoulder with these companies to develop the AI agents and systems to be able to, say, predict the wind better or place the wind turbines solar panels in better places?

Marc Spieler:

Yes. Yes, we are. So the way NVIDIA works is we want to solve hard problems. And so we partner with a small amount of end customers as well as software development companies, national labs, research institutions to solve really hard problems. So yes, when I say we, it means we get in there and engage and we prove out the concept. If it makes sense, we publish libraries and open source math libraries or solvers and others for people to adopt. In most cases, those solutions are developed and deployed through software companies or the end customers. But our goal is always to demonstrate the art of the possible and prove out a concept with what we call lighthouse accounts. And these are accounts that want to engage with us at an engineering level to solve a really hard problem. And then what we find is that the industry quickly follows that because it’s been proven out that AI or accelerated computing can solve these problems in a much more efficient manner.

Jackie Forrest:

Okay. And you talked about Earth-2. So is that a software offering that you have and that helps you predict the weather and therefore the day ahead? You can predict how much wind and solar you’re going to have. Maybe just explain what that is.

Marc Spieler:

Correct. So it is a climate and weather platform and so it’s used by weather services and others to build on top of in a way for them to provide services to their customers. So our goal is not to become a weather services company. What we want to do is provide the underlying tools to accelerate the work in which they’re doing, and companies then purchase data and other capabilities from the weather services and others, but then can use that within an AI model to converge on outcomes that potentially get higher resolution, more specific areas, and answer questions that they’re looking to solve. If you’re drilling in a certain geographic location and you know that storms are coming, how do you look at a higher resolution outcome to understand when you should start evacuating people from that facility and the downtime, but make sure that you’re able to do it safely for the people that are working.

Same thing for energy trading. What is the weather to be expected in this area? Where should you adjust your supply chain? And things like that. But we’re not a weather service. We don’t want to compete with our partners. We want to be able to make sure that they’re able to deliver a premium product to the industry.

Peter Tertzakian:

Right. You know what I’m excited about and I think this is a rhetorical or obvious question, but you must be involved in the exotic end of the energy frontier I’ll call it, with simulating new types of, say, nuclear reactions and nuclear fusion and these sort of frontier areas of energy now to be able to use AI to accelerate these potential energy revolutions.

Marc Spieler:

Absolutely. Nuclear is going to make a major comeback. Nuclear fission technology has been around for quite some time. Some countries have gotten bigger into it while others haven’t. I think the ability to apply AI and simulation to that front to increase the speed at which we can do permitting and licensing, analyze safety, run the operations, will help fission technology and the next generation, the Gen Four reactors, the smaller modular reactors, come online faster. And then fusion, which is, I think, the holy grail of energy. If that’s possible, it’s safe, it’s efficient, it holds so much potential. And so we’re actually creating the first digital twins with some of our strategic partners and those early fusion reactors are going to be available shortly and then the goal is to make them as efficient as possible.

Jackie Forrest:

Well, that is the hope that we’re going to get fusion energy because we may need it for all the AI we’re going to be consuming as a civilization.

Marc Spieler:

Absolutely.

Jackie Forrest:

Let’s get to today’s energy intensity of the data centers because that is a bit of a hot topic, how much energy these things are using. And those were fabulous numbers you threw out there about the increase in efficiency you’ve seen already. Can you tell us a little bit more about how you’re trying to make these systems even more efficient and what your expectations are in terms of reducing data center energy consumption could be if we’re doing the same work going forward? Have you already got most of your efficiency gains or is there still a lot out there?

Marc Spieler:

We continue to work on it energy efficiency gains generation after generation. So with the announcement of our Vera Rubin platform earlier this year at GTC, we talked about 10x performance over our current platforms. And so from an infrastructure perspective within NVIDIA, we continue to make strides on the chip and GPU and networking framework. But as I mentioned earlier, it’s really about this entire stack. So it’s how do we get the most efficient energy, make the chips as efficient as possible, the infrastructure, which is done through partnerships with Schneiders and Siemens Energy and Siemens and GE Vernova and Hitachi and all of those folks who are making the infrastructure for data centers, and then the model builders and then the applications.

And we’ve got to look at that entire stack. And every time we can increase the efficiency… And I forgot companies like Vertiv and Motivair who are doing the cooling and that, because I know at the beginning we mentioned cooling and the heat as a big offtake. But once again, we’re using closed systems now. We’re removing the need for significant water. And once again, every electron that we can save that doesn’t go towards running infrastructure but instead goes towards generating tokens makes the cost per outcome less and less. And so-

Peter Tertzakian:

Right.

Marc Spieler:

… we created this reference design called DSX, and that really is a reference design that we promote and we give away to our partners so that they can take advantage of the engineering we’ve put into this entire ecosystem and leverage it to get the most performance per watt, the entire system.

Peter Tertzakian:

You talk about the stack, and I learned this at a recent conference I went to and there was a presentation by a fellow from NVIDIA, and that the stack actually has more granularity within the stack, that there’s high performance chips and lesser chips that don’t need. I mean there’s in my mind the example of, well, what is the weather going to be in Houston today? Like you don’t need to go to the most computationally advanced chips within the data center to answer that question. It actually, you’ll get shunted to a lesser system that does not require that. So there’s energy efficiency for the outcome as you call it, but there’s also within the architecture ensuring that the prompt goes to the right type of processor for the right type of complexity.

Marc Spieler:

Yeah, that’s exactly right. So there’s lots of ways in which we can create energy efficiency and flexibility, and part of it is moving it to the right set of infrastructure. There’s also distributed inference, so you might not need to go as far to a big data center where you could go to local or modular data center. More and more we’re trying to create efficiencies and flexibility because we also see these big AI factories as assets to the grids in which they sit because that way we can turn up or down that work and give power back to the grid for peak requirements.

Peter Tertzakian:

And can you talk about… There’s these AI servers, we bought one for our office, and to me this is like the early signals of a trend when we went from mainframe computers to workstations to PCs. You don’t need a mainframe and a massive data center again to answer the question about what’s the weather going to be. You can just use a smaller distributed system. So do you see more and more proliferation? I mean, we hear Apple has got a huge backlog for its Mac minis and it’s Mac Studios because people are buying these things and want them. Can you talk about that trend as well? I know you make some of these as well.

Marc Spieler:

Yeah, absolutely. I think people want to be able to work on their desktops and the cost efficiency of having the right infrastructure at your fingertips on prem makes a lot of sense as you develop models or do inquiries. And then, depending on the type of company, you might choose to scale up. A lot of the token consumption by end customers is going to come as a service and what we’re going to find is they’re going to be integrated into different software that they’re using, like Copilot or anything else. We’re going to find more and more of the software stacks being developed will have agentic uses, and then you’ll have your own local. But yeah, we do this today in autonomous vehicles and other edge devices where you might not always have connectivity back to a cloud. You can’t turn off those capabilities. You need to process the sensors and the LiDAR data and be able to make real-time decisions without connectivity back to a cloud.

Jackie Forrest:

Okay. So it is confusing because you hear, “Well, we need five gigawatt hyperscale campuses,” on one hand. And then you’re reading something else which like, “We’re just going to put these little micro things in your house.” And then there was this article I was reading where I could just volunteer to host a mini data center in my house. Can you do the same things? Or do we need all of these things depending on what we’re doing? Or could the training happen on 100 homes mini data centers?

Marc Spieler:

Not likely. What I would say is it’s all of the above, and it comes down to efficiency and scalability. If you’re training large foundational models like your big ChatGPTs or Gemma or Anthropic models, you need lots of GPUs and you need the network connectivity to tie them all together in a way that you’re going to be able to train them efficiently. If you’re inferencing and asking questions against those models, you may only be running on a single GPU or two, and those can be distributed depending on the size of the model that you’re inferencing against. I know it’s a vague answer, but it’s really all of the above. And as we start to use more and more inferencing for drones or robots in our homes and humanoids or EVs or other devices that will become smarter and smarter, the distributed inference it makes a lot of sense.

It’s probably not as cost effective in large scales, but for the types of workloads that get pushed to it, it’s very cost effective and it allows you to move fast because the power’s available in small increments today around the grid, but to try to go get a gigawatt or five gigawatts is very challenging in the same location.

Jackie Forrest:

Okay. Well, let’s come to that because one of the concerns with AI data centers, especially when we start talking about these… It’s funny, even a year ago when I heard gigawatt, I thought that was crazy and now we’re talking about five gigawatt campuses, is that these campuses have no flexibility, that 24/7 they need the five nines and they’re adding a lot to peak power requirements and therefore we need to put all this generation and transmission distribution to support them. Can you just tell us, is that the case for all uses? And I know NVIDIA is partnering with Emerald AI, and I will put a link to that in the show notes, looking at how we can make a, they’re calling it a power flexible AI factory. So talk to us about the current situation about flexibility and if that’s going to change.

Marc Spieler:

Yeah. So first, I think understanding the difference between a traditional data center and an AI factory is really important. A traditional data center is heterogeneous and it does a lot of different things. It can run web services, it can run email servers, ERP systems, all different kinds of things, and some of those are business critical and you need five nine availability. AI factories is exactly what I just said. It’s a factory. It’s got one job, produce tokens, and therefore you look at that and you say, “Okay, takes electrons in and data and creates tokens.” And that’s got more flexibility based on what you’re trying to achieve because you can move workloads accordingly and in real time or close to real time, and this is what we’re doing with Emerald AI is it allows you to flex the work that you’re doing.

And so we’ve been working closely with EPRI who has their DC Flex initiative and they’ve got lots of utilities involved in technology companies to demonstrate how are these AI factories flexible resources to the grid, where if you’re interconnected there and you need additional capacity for the grid because it’s a warm day or a peak day, you can actually ask the data center or in this case the AI factory to actually give me back some power.

So if it’s a gigawatt data center and you say, “Hey, I can only give you 800 megawatts for the next three hours because of temperature.” We’ve got the capabilities within DSX Flex to either ramp down the GPU performance by capping the power requirements. We can redirect the inference to another data center or another AI factory. We can in theory slow down the computer, turn different jobs off depending on the SLA that they have with their customers, but it provides much more flexibility than the traditional five nine data center.

Jackie Forrest:

And is that in the future or is that happening today?

Marc Spieler:

No, we’ve done quite a few proof of concepts already. I think we’ve done five or seven of them, and we’re in the process of building out our Manassas data center, which will be a working version of that that will be online I think later this year, that will demonstrate that running in operations. But this is not new, it’s just a different method of providing flexibility. But you could ask today, if you think about demand response, on really hot days in certain areas, they might send an email out to all of their customers saying, “Hey, can you turn down your AC or you see the Nest thermostats that can be adjusted?” Think of this as just an agreement with an AI factory to do that at a much larger scale.

Jackie Forrest:

I’ll just have one follow up. In your press release, you talk about this could unlock a hundred gigawatts of grid capacity and supercharge the AI revolution because now we don’t need that generation. If you’re not contributing to peak load, it certainly changes the infrastructure needed. That is quite different in terms of when we think about the constraints to AI, if there’s a hundred gigawatts of power we’re not using.

Marc Spieler:

Well, if you think about the grid, the grid has become so complex. It’s bidirectional, right? We’ve got lots of people with solar panels feeding back into the grid, and we have renewables spread out across the grid that are sending things on, and the complexity is high, but the grid is built for a peak load. So if you know that in Texas, on a warm summer day, you’re going to have to have so much power because of all the AC running, you need a grid that’s capable of delivering that power, but probably 99.5, 99.9% of the time, you’re probably running at about 60% of that power, but you built for peak. And so the question is, can we bring in AI factories as assets that can use that 40% delta, and then when you hit those peak loads, you can actually turn down the AI factory and be able to provide that peak, but when you’re not using it, then you can go back to running business as usual.

This will do two things. It’ll leverage what I would call stranded capacity on the infrastructure, and it should make power considerably more affordable because today you’ve built the infrastructure for peak, but you’re only charging customers at 60%. So if these AI factories can increase that utilization and you’ll hear us talk about the denominator, if we can increase the denominator, all of a sudden now it makes the overall cost to everyone much more affordable.

Peter Tertzakian:

Yeah. So we’re in this conversation about efficiency and optimization and so on. Let’s take a slightly different tack. So I said at the outset from an energy lens, this is 100% an energy play because 100% of the electricity that goes in is converted into heat. And then there’s this abstraction that comes out, which is tokens or knowledge, units of knowledge. And so I don’t know, I haven’t done the calculation. I’m going to do it for next time, five gigawatts of electricity converted to heat, how many homes that would heat, but are there some of these data centers that are thinking about district heating? It just seems that would make so much sense. You put this beside a town and you can heat the whole town, especially if it’s in a northern latitude.

Marc Spieler:

Yeah, absolutely. We’ve got organizations in California and other places and across Europe that are looking into this intensely with the ability to take the heat that’s coming out of the data center and use it to heat homes or secondary value in that heat. We’ve continued to make our chips as efficient as possible so we can actually run them at a higher temperature so we’re not having to extract as much heat off of them, but I think it’s a great idea. And we look to some of the startups that are in this area that are really focused on how do they in reality.

Peter Tertzakian:

Well, it might even help with the adoption. There’s a lot of pushback in communities with data centers, but tell the community that, “Hey, we’ll heat your homes and at a reasonable cost compared to what you would otherwise do it,” then there might be more receptivity to that kind of thing. Well, there’s so much more we can talk about. We didn’t even talk about the inefficiencies of the stuff that’s coming out of the data centers, the vibe coding phenomena is now people talking about vibe slop, which is just code that does not necessarily need to be generated. But I mean, this is what happened with the email slop that we’ve had over the course of the last since email was invented. And anyway, I mean, there’s so much more we can talk about, but Marc, you’ve been so generous with your time in terms of educating us on what NVIDIA does, educating us on what you’re doing in the world of energy, oil, gas, renewables, electrical power, efficiency, and so on. So thanks very much for joining us.

Marc Spieler:

I appreciate it. I enjoyed the conversation and I look forward to another one in the future.

Peter Tertzakian:

Good. Well, Jackie, the only thing left is quickly to talk about the Pope and his encyclical. Now, I found it quite fascinating and I actually listened to some podcasts and did some reading on this encyclical. And the Pope decided to make his first encyclical, which was basically a papal teaching. It’s about the size of half a book. I can’t admit that I’ve read it, but I think that it’s really important that whatever you think about the Pope and religion and what have you, that this is a global institution that leads people. And I think basically what the Pope is saying is that we need to think about what this technology is doing to humanity. He’s not outright saying he’s against it or for or whatever. He’s just saying, “Let’s just pause for a thought and what it means to humanity, what it means all the way from what it does to societal jobs to what it means to artificial cognition and things like that.”

So actually I think it’s an important piece of work and I am going to read it here if I ever get a minute to myself, but I don’t know what you did. Did you read it?

Jackie Forrest:

Yeah. Oh no, I didn’t read it. And then I did read some things that said, “Be careful what articles you read because everyone’s taking their own take on it. Read it yourself.”

Peter Tertzakian:

There’s a selection bias. Yeah.

Jackie Forrest:

But it is the size of a small book. But no, I think there’s some warnings there in terms of regulation and not leaving this to companies, and I think there’s some good advice there. Well, maybe on a future podcast, you can tell us after you’ve read it.

Peter Tertzakian:

Yeah. Well, I think we are in the frontiers of technology. I’m super excited by this all. I’m also somewhat, depending on the day, I wake up super scared in terms of what it means, but on balance, I’m excited, but we do need to think about the impacts because it’s philosophical, honestly, this revolution that’s going on. Thanks again, Marc. Jackie, I think that’s it.

Jackie Forrest:

Yeah. And thanks to our listeners. If you enjoyed this podcast, please rate us on the app that you listen to and tell someone else about us.

Announcer:

For more ideas and insights, visit arcenergyinstitute.com.

NVIDIA’s Marc Spieler: AI, Data Centres, and Energy

Episode 328 transcript

You might also like

Contact Information

Join Our Newsletter

Recent Podcasts