Andrew Lyden: Having an open-source model would have made market reform discussions less polarised – and we’ll need one in the future
Sub head
Andrew Lyden is a Lecturer within the Institute for Energy Systems. His recent work includes PyPSA-GB, a model of the UK energy grid capable of simulating both historical data and future scenarios. He is also involved in the INTEGRATE project, which focuses on integrating multiple energy sources with seasonal thermal energy storage to support the decarbonisation of heat. In addition to his engineering work, Andrew lectures on Energy System Economics and Management at the University of Edinburgh.
Media
Image
Content
The UK government has recently reaffirmed its commitment to reaching net zero by 2050, and there are multiple possible pathways to get there. You wrote a paper in 2024 in which you mentioned National Grid’s future energy scenarios, which vary assumptions about hydrogen consumer behaviour and carbon capture. Last year, you released open-source software that allows scientists to explore the UK's energy futures in a more transparent, peer-reviewed way. How does your model compare to more established ones? And does it reveal anything that they might overlook?
The main thrust of my model is about having something open source that everyone can use. One of the major drawbacks in how future scenarios are currently developed is that not all data is made public. A lot relies on behind-the-scenes proprietary tools, owned and controlled by different companies. These tools allow for nice-looking, complex modelling – something that PyPSA now offers as well. The National Energy System Operator (NESO), for instance, uses a programme called PLEXOS to investigate what the investment and the operation of the system might look like in the future. NESO uses this model for its Future Energy Scenarios projections. PyPSA can do very similar things. The main advantage of PyPSA isn’t so much about methodological breakthroughs, but that models like PLEXOS sell for up to $100,000 for a single user, whereas anyone can use PyPSA for free – though some Python knowledge is needed to get started. When future scenarios are developed using proprietary tools, it’s not really possible to release all the underlying data. So people can’t really fully understand things like the future hourly dispatch, and how storage, renewables, and the network work together to meet future demand. While NESO already models similar things, the idea was to have a model that can uncover the details behind their analyses.
What does PyPSA actually show? And what kind of timescales are we talking about?
Using PyPSA, we can look at every hour in the future – like 2050 – and see how much energy is generated by wind, solar, and hydro. We can also see how storage technologies respond, including when they discharge and when there’s excess renewables generation. The model also shows how new technologies contribute. Hydrogen power plants and carbon capture and storage power plants will come into the system, but they are very expensive to operate. Often, you see that they have quite low capacity factors over the year – meaning they’re not used very frequently. No baseline model that anyone can explore has been available to the academic community or industry because of limitations in the data released by NESO (formerly National Grid – ed.). When there’s a well-established future scenario, you can use PyPSA to see what would happen if you change some of the scenario parameters. For example, seasonal thermal storage doesn’t really have a role in future energy scenarios. Using the scenarios as a baseline, you can ask: what if we had more seasonal thermal storage distributed across the country? What would the benefits be?
For people like me who have not done much electricity grid modelling, can you explain how someone would actually use your model? What do you take as given and what parameters can you change?
In PyPSA-GB, there’s already a lot of data built in from NESO’s Future Energy Scenarios. Within those, there are four different pathways – in three of them, we reach net zero. So a user can come along and say: OK, I want to work with Leading the way, one of the future pathways. And I want to model the year 2050, maybe a specific summer week. That’sthe user input – you select the period you want to look at and then you run the model. It does an optimisation to see how the system can be operated in the least cost way. It considers generation, demand and the network. You then get the operation of the system over the time period you selected. The main thing we’re looking at is the integration of renewables. The future scenarios assume very high capacities of offshore wind and solar. The model really tests how flexible the grid needs to be to handle these, so options like battery storage, EVs, other demand-side solutions, and thermal storage are included to support the integration of more renewables. And you can only really see that on the kind of hourly dispatch detail with PyPSA-GB. That kind of data isn’t released from the Future Energy Scenarios, even though NESO carries out similar analysis internally. And you can go beyond this baseline. What if there’s more renewable generation? Or fewer renewables, but more dispatchable capacity with carbon capture and storage? Would that reduce the need for electricity storage? Could you run the system at lower cost? You can start to test those kinds of questions and explore how different assumptions impact system performance.
‘What if there are long periods of low renewable generation? What if the interconnectors between Scotland and the rest of the UK fail? Or if hydrogen and carbon capture and storage don’t develop as expected? What we found, overall, was that because of the massive projected increase in generation in Scotland, none of those risks make too much of a difference.’
How has the model been used until now? What kinds of questions are people asking with it?
It’s been used mainly by many PhD and master’s students at various universities. I know this because there have been some bugs, and people email me with questions about them. But I haven’t heard from everyone who hasn’t run into any issues. The people I’ve heard from have mostly been looking at flexibility options – things like what happens if we have more electric vehicles in the system that can interact with the grid with both directions. Others have looked at reducing the amount of wind curtailment or incorporating thermal storage, including in buildings. I’ve also taken the model in a few directions myself. You know about my work on thermal storage, but we also did a study through ClimateXChange for Scottish government, where we looked at the security of supply in the future energy system. They were particularly interested about how Scotland’s electricity system would operate if it became more isolated from the rest of the UK. Generally, we saw a massive increase in projected generation in Scotland, compared to only a modest increase in demand. That leads to a significant boost in security of supply – Scotland is expected to become a major energy exporter to England and to Europe. But we also stress tested the system. What if there are long periods of low renewable generation? What if the interconnectors between Scotland and the rest of the UK fail? Or if hydrogen and carbon capture and storage don’t develop as expected? What we found, overall, was that because of the massive projected increase in generation in Scotland, none of those risks make too much of a difference.
You’ve developed PyPSA-GB, but PyPSA itself has been used across Europe, in India… Similar tools are mainly used to chart futures across larger geographic areas and long periods into the future. But many challenges are more local. Have you looked at more regional issues as well, maybe using slightly different models, perhaps in your work on thermal storage?
Yeah – PyPSA generally looks at the high-voltage transmission level. As you mentioned, the European model is probably the most widely used example of PyPSA. It can model the entire European network. When people use it, they often explore what simplifications are needed to run it in the first place, but also investigate what kind of transmission build-out is needed for electricity – and potentially for cross-European hydrogen networks too. So yeah, those are very large-scale themes, even bigger than what PyPSA-GB looks at. But a lot of the challenges we’re going to see are at the smaller end of the system – in the lower voltage parts. During my PhD, I became interested in what small district heating systems could offer to the wider energy system. If we build large hot water tanks alongside small or large district heating systems, could they help soak up some of the wind that might otherwise be curtailed in the future – or even now? This could be another source of flexibility. To explore that, we used a techno-economic model and focused mainly on thermal storage combined with heat pumps – which let you use electricity efficiently to meet local heat demand. The challenge is that you need a lot more detail in the modelling. When you use PyPSA to model the whole country, you don’t need to worry too much about the intricacies of hot water. But if you’re modelling a district heating network which only has 500 buildings, you really need to think about the thermal flows, temperature changes through that tank, and create detailed models. The same level of detail is needed when modelling heat pumps.
Heating is one of the biggest challenges for the UK on road to net zero. Emissions from heating and transport are higher than those from electricity generation. Your background, apart from engineering, is in physics and energy system economics. Do you think solutions like widespread heat pump adoption are technically and economically feasible?
That’s a big question, isn’t it? From a technical perspective, heat pump rollout in the UK is quite slow – a lot slower than we need it to be. Heat pumps are probably going to be the main technology that's going to decarbonise domestic heat here. I don’t think we really have any alternative. There was a time when we considered hydrogen boilers in people's homes, but that’s generally been dismissed now because producing hydrogen for heating is inefficient compared to using heat pumps. But heat pumps come with many challenges themselves. Technically, they work very well if installed properly. And that “installed properly” issue has been a big part of the debate for the last 10 to 20 years. Initially, a lot of heat pumps weren’t installed well – sometimes, they were oversized or the controls weren’t properly set up. Lots of people ended up with poorly performing heat pumps. That’s when you get those sensational headlines in papers like The Telegraph – most of the time, the problem is just poor installation. And while people sometimes jump to the conclusion that heat pumps are a bad technology, they’ve actually been around for a long time and are well established. We can simply look across to the North Sea to Norway or Sweden, countries that heavily rely on heat pumps, even though their climates are much colder than ours. Recently, there’s been a lot more expertise developing around installation, and growing awareness of the need for high standards in heat pump design. Gas boilers are different – you can pretty much put any size boiler into a flat or a house and it will work fine. The performance might vary a bit, but it generally doesn’t bother the user. With heat pumps, things like size and correct installation matter more.
‘On the economic side, heat pumps are quite expensive and that’s why the government has the Boiler Upgrade Scheme, offering grants for those who want to install them. But many people still say it’s not as if you can just go and get your heat pump with a government voucher and you then save loads of money over the years. In the best case scenario, it will be similar to the gas boiler.’
What about the economic side of heat pumps?
On the economic side, heat pumps are quite expensive and that’s why the government has the Boiler Upgrade Scheme, offering grants for those who want to install them. But many people still say it’s not as if you can just go and get your heat pump with a government voucher and you then save loads of money over the years. In the best case scenario, it will be similar to the gas boiler. That’s because there is still quite a large difference in price between electricity and gas – gas is generally three to four times cheaper. If you have a very well installed heat pump, operating costs are probably similar to a gas boiler. Installing a heat pump also takes much longer than replacing a gas boiler with another one. It’s often not a straightforward swap – you might also need to replace radiators and sort out other issues. These factors make a big difference in people’s choices. Those who care about the environmental impact will probably get the heat pump, but others might go for the option with fewer barriers and just get a gas boiler.
Is it possible to reach net zero without heat pumps and similar thermal solutions, or are they a requirement?
There’s also district heating. With that, you’d have a heat exchange unit in your flat, supplied by a centralised heating network. But laying that pipework can be very disruptive. While it’s likely something we’ll need to do in high density areas, it’s not very economical in less populated places. That’s where we’ll need those individual heating solutions – and heat pumps are one of the few low-carbon options we have moving forward.
You have multidisciplinary expertise in engineering, physics and energy economics. How did that come about?
Physics is a really useful degree because it teaches you how to problem-solve. It gives you the confidence that you can take on any challenge. That’s why there are so many physicists in academia – they tend to think they can do anything. Physics teaches you a lot about electricity, thermodynamics, and other fundamental ways of thinking. In engineering, things get much more applied. In physics, you get to the equation and that’s your answer. In engineering, you go from that final equation to how it’s actually implemented. In a lot of what I do, I use equations from physics to build detailed models. Of course, they’re not completely accurate – and that’s where the engineering perspective helps. You need to know where you can simplify things without losing the insights you need to answer the research question. Then there’s economics, which brings another way of thinking. It’s about how people behave in the face of scarcity. And energy is, at its core, a scarce resource – if we had all the energy in the world, we wouldn’t be talking about it. It’s difficult to produce but very useful if we can get it to the right places. So from the economic side, you look at how markets shape the way people behave within the energy system. It’s not just a physical system – it’s also about market structures and regulations.
Before the interview, you told me a story about your colleague moving to Germany and you stepping in on a short notice to teach a course in energy systems economics. Was it perhaps the physics that gave you the confidence to do that? It was just some simple economics maths…
Maybe a little bit, yeah (laughs). But economics is a proper discipline in its own right, and there are a lot of intricate concepts that can be tricky to get your head around. I’ve always been one to go into different areas that I’m not familiar with. So when I had that opportunity to take that course, I saw it as something I didn’t actually have too much experience in – and that probably led me to want to do it even more. Learning from different perspectives helps you understand the bigger picture of the energy transition.
‘In this climate crisis timeline, it’s not about doing the best science or getting the perfect model. It’s about how we transition as quickly as we can. We’re not doing normal science the way people did 50 years ago – where you could sit with a problem for years, not really talk to anyone, and eventually come up with a fantastic answer.’
Do you personally believe that the UK will reach net zero by 2050, and what do you think is the most important mindset or type of thinking that we need to achieve this goal?
2050 is a bit too far away to predict anything. Technically, we can project pathways, but there are so many uncertainties that the main value of the target is giving us something to aim towards – to actually start and doing things now and make progress. Targets like net zero hep bring industry, policymakers and other actors together and they drive decisions. But whether we’ll actually reach net zero or not – there’s so much turmoil in the world that it’s hard to say how everything will unfold. In terms of getting there, a lot of my thinking is about how we can tackle these challenges in a much more joined-up, collaborative way. That’s why I really believe in the open-source nature of the models I develop. I imagine most universities in the UK will have a PyPSA-GB-type model – but they’re not made available. In this climate crisis timeline, it’s not about doing the best science or getting the perfect model. It’s about how we transition as quickly as we can. We’re not doing normal science the way people did 50 years ago – where you could sit with a problem for years, not really talk to anyone, and eventually come up with a fantastic answer. We really models that everyone can get behind – where everyone in industry and academia can clearly understand the justifications, and people in policy and industry can use them to make decisions. A good way to net zero means it doesn't cost people too much, it doesn’t harm the economy, and it doesn’t damage the environment as much.
Speaking of policy decisions – we are currently expecting a decision on whether the UK will retain a national wholesale energy market, which is part of the REMA reform, or whether it will move to a zonal market arrangement. What are your thoughts on this reform?
The debate between national and zonal market has become very contentious. One thing I want to say is that having more open modelling around it would’ve helped. Right now, the debate is driven by consultancy reports – commissioned by Ofgem, Octopus and others – that rely on in-house models. These models involve a lot of assumptions and proprietary data. We’re seeing differences in the projected benefits that come from the zonal pricing option. Specifically, the FTI Consulting report shows very high benefits, compared to the AFRY Management Consulting model, which suggests a much lower benefit. People have been asking: how would the outcomes of the models change if we changed the underlying assumptions? The classic example is the cost of capital. In a zonal system, you might get lower liquidity in each zone because you’re splitting up the market. That can lead to more uncertainty and price variability, increasing the perceived risk. That risk can make it more expensive for investors to borrow money – the cost of capital goes up, and that can reduce the overall benefits. So one of the big questions becomes: if the cost of capital increases, how much does that reduce the overall benefits? Essentially, we’ve got into a debate where, according to the FTI model, there will still be sizeable benefits – which is not the case according to the AFRY model. It all comes down to the inputs, assumptions, and how the models are operated. And unfortunately, neither of them is open-source, so you can’t use them to replicate and compare their results. I really think that there was a place for an open-source model that would come from academia — more of an independent viewpoint — to act as a way of comparing to these different models. In the next 25 years, there will be many changes and debates, and I really see a role for an open-source model that sits in the middle and can be understood by different stakeholders, consultancies, and industry representatives who decide on possible policies with the help of consultancy reports.
Would this be your PyPSA-GB model or something else? The PyPSA-GB model enables a degree of zonal price analysis…
There probably needs to be a next-level model. PyPSA-GB mainly uses the Future Energy Scenarios as inputs and is now mainly focused on the operation of the system. There probably needs to be a model that also computes the investment side, including how increases in generation and network changes are economically driven. You’d probably also want to incorporate a bit more detail on the market frameworks – how the zones are set up, and what the electricity flow between the zones is. PyPSA-GB can be a starting point fot modelling these things – as you say, the model has that detailed network, so you can produce locational prices for different areas of the country, and that’s really useful. But if you want to replicate what these consultancy models are doing, you really need to add that investment side.
How did you get into applying your modelling skills to the current national electricity market questions?
I was originally driven by the REMA process and the questions it was raising, mainly around decoupling the price of gas and electricity. Gas really sets most of the price in UK electricity markets, and that became a huge problem when gas prices increased after the war in Ukraine started. This decoupling was the original motivation for the reform, but it soon became evident that it’s a very difficult thing to do. Our system relies heavily on gas, so it’s natural that gas sets the price most of the time. We can’t magically change that, although there are various proposed mechanisms. You can set it up so that renewables set the price for some parts of the market, with different prices for different generators. However, it’s generally being found that there’s no easy way to do this. This is partly why zonal pricing came out among the preferred options – because in areas with excess renewables, the gas price can be decoupled from the electricity price. For example, in Scotland, wholesale prices could go to zero or even negative, depending on how renewables are financed.
I’ve always wanted to model these kinds of dynamics. What I found quite quickly is that the existing model I’d developed for seasonal storage across the whole system – as you mentioned – already had quite a simple zonal structure built in. What really interested me was how this connected with my previous work, which had mainly been on heating and cross-sector interactions. How does electricity link with heat? How does it link with hydrogen and electrolysers? That’s what I would like to focus on in the future. It’s also connected to the market reform. If there’s zonal pricing, does that mean there will be many electrolysers in Scotland? Or it’s probably not that simple, is it? If electricity prices for consumers are cheaper in Scotland, how does that affect heat pumps? In all my research, I want to understand more and more about how those different things connect together.
You are part of the Electricity Markets Research Hub. Is this effort to understand how different things work together what motivates you to connect with researchers from the School of Social and Political Science and the School of Mathematics?
Yeah, exactly. That’s why the Electricity Markets Research Hub is so useful. Chris and Lars from the School of Mathematics are interested in building models – maybe more focused on detailed aspects like optimisation or speeding up algorithms – but their research is still very relevant to what I’m doing. And I also really like what Ronan from the School of Social and Political Science is working on – looking at how we govern energy systems and how regulations affect them. That obviously ties into electricity markets and their modelling. We should be accounting for regulatory-level decisions in the models, too. And even beyond regulation: how people behave. There’s going to be a lot more demand-side flexibility, but in our models, we often assume that there is a central planner, and that people will use things like EVs the way the planner wants them to be used. But that’s not reality. If we want to improve our models, we need to account for how people actually behave – and that’s a whole other field, isn’t it? Others have far more expertise in that than me. So for me, it’s about how we can bring those different levels of expertise into the modelling world, to make the models better and more representative.
Interviewer: Jan Žižka