Analysis of Short-term Bidding Strategies in Power Markets

Transcription

1 193 1 Analysis of Short-term Bidding Strategies in Power Markets Pablo Frezzi, Student Member, IEEE, Francisco Garcés, and Hans-Jürgen Haubrich, Member, IEEE Equation Chapter 1 Section 1 Abstract Markets with signs of concentrations, interactions, barriers to entry/exit and coordination among participants are particularly prone to evidence tacit collusion. The liberalized power markets fulfill largely these conditions and they are therefore susceptible to suffer tacit collusion. An approach capable of analyzing such strategic behavior in power markets and quantifying it economically appears to be necessary. In this article, an agent-based model is proposed to analyze how market participants can learn tacitly collusive behavior. The competition among market participants is modeled as a repeated game with imperfect public information. Reinforcement learning is applied to model the flexible and adaptable behavior of the market participants. A test system with different levels of market concentration is used to quantify economically the relation between the market concentration and the exercise of tacit collusion. The effect of transmission constraints on the incentives to exercise tacit collusion is also analyzed. Index Terms Agent-based modeling, liberalized power markets, market power, reinforcement learning, tacit collusion. I I. INTRODUCTION N the last decade power markets of many countries were liberalized in order to reduce electricity prices by means of growing competition. This liberalization, which took place especially in the generation sector, was intended to replace the existing monopolistic structures by competitive power markets. Nevertheless, due to concentration, mergers and transmission constraints, the present liberalized power markets are more akin to oligopolies than perfectly competitive markets. In perfectly competitive markets, a supplier maximizes its profits by bidding its actual marginal costs. By contrast, in oligopolistic markets, the participants are not mere passive price takers, but both the price and the market s dynamic depend on the strategies developed by them. Therefore, the market participants are capable of exercising Manuscript received November 30, 06. This work was supported by the German Academic Exchange Service (DAAD). P. Frezzi is with the Institute of Power Systems and Power Economics (IAEW), RWTH Aachen University, Aachen, D-556, Germany (Tel.: ; Fax: ; Pablo.Frezzi@iaew.rwthaachen.de). F. Garcés is with the Instituto de Energía Eléctrica (IEE), Universidad Nacional de San Juan (UNSJ), San Juan, J50ARL, Argentina ( garces@iee.unsj.edu.ar). H.-J. Haubrich is with the Institute of Power Systems and Power Economics (IAEW), RWTH Aachen University, Aachen, D-556, Germany ( Haubrich@iaew.rwth-aachen.de). market power to make profits well above the ones they would make if the market were perfectly competitive. Under these circumstances, conventional pricing modeling based on the marginal-cost principle cannot be applied. Considering the dynamic of the present power markets, there is a growing need of simulation models capable of analyzing the short-term bidding strategies developed by the market participants. Game theory has traditionally been developed as a theory of strategic interaction among perfectly rational players who exhibit equilibrium behavior [1]-[3]. Nevertheless, the assumption that market participants show perfectly rational behavior is a simplification, which cannot be always verified in the real world. Furthermore, the consequences of repetitive processes in the behavior of market participants and in the planning of strategies remain usually unconsidered. One of the most observed strategic behaviors in power markets, which derived from the repetitive nature of them, is the tacit collusion, which involves collusion among entities without any explicit contact. Markets with signs of concentrations, interactions, barriers to entry/exit and coordination among participants are particularly prone to suffer tacit collusion [4]. The interaction among market participants in liberalized power markets constitutes a repeated and coordinated game, whereby a process of experimentation and learning changes the behavior of the participants. A computational approach that can reflect these learning processes and model the structure and marketclearing mechanism appears therefore to be necessary. Agent-based Modeling (ABM) seems to be a promising technique to model power markets. ABM considers the autonomous learning abilities of individual entities and is able to capture the behavioral aspect of repeated auctioning processes and interactions among market participants. Some previous researches were carried out in power markets applying ABM. In [5], an agent-based model is proposed to evaluate generation expansions. In [6], the introduction of a bilateral market in conjunction with a balancing market in England & Wales is analyzed. In [7], investigations about the efficiency of discriminatory auctions and the exercise of market power are presented. In spite of the relevance of the tacit collusion and the suitable characteristics of ABM to analyze it, there is not much research on this field, especially about the causes of the tacit collusion and how the market concentration influences the bidding strategies of the market participants. Furthermore, although transmission constraints

2 193 2 may influence the exercise of market power, they are usually disregarded. In this research, an agent-based model is proposed to analyze the bidding strategies of the generation agents under dynamic conditions. The model is especially intended to detect and quantify economically tacit collusion considering repetitive auctioning processes in spot markets. Different market concentration levels are considered in order to analyze and quantify the influence of the market concentration on the exercise of market power and specifically tacit collusion. Furthermore, transmission constraints are modeled in order to analyze their impact on the bidding strategies and the market prices. The competition among market participants is modeled as a repeated game with imperfect public information, in which each market participant explores iteratively the solution space until an equilibrium point is reached without knowing the payoffs of its competitors. Reinforcement Learning is used to model the adaptability and the evolution of the behavior of the market participants in the course of the time. The generation agents, whose targets are the maximization of their daily profits through the submission of energy bids, are considered independent entities by means of ABM. II. MODEL OVERVIEW Applying ABM, a spot power market, in which the generation agents carry out their transactions, is modeled. Each generation agent represents a firm with a certain portfolio of generating units, which produces electricity and sells it in the spot market obtaining a profit. The fuel costs, consumption functions and hourly demands are considered exogenous variables of the model. The spot market is operated by a market agent, whose functions are to receive energy bids, clear the market and inform the generation agents about the acceptance or rejection of their bids. The model is based on a two-stage optimization process. In the former, each generation agent develops an optimal bidding strategy to maximize its daily profits without knowing the assets and payoffs of its competitors. In the latter, the market agent solves an Optimal Power Flow (OPF) to determine dispatches and spot prices maximizing the social welfare. Lagrange relaxation and linear programming are applied to solve the OPF. After a certain number of iterations, in which the generation agents explore and develop different bidding strategies, an equilibrium point in their behavior and in the market prices is reached. Piecewise linear functions are used in the model to describe the supply functions. The exercise of market power by a generation agent is represented as a percentage mark-up (ϑ), which is added to the generation marginal cost (MC(q)). A generation agent behaves as price taker only when it chooses a mark-up equal to zero. The supply function of a generation agent, which defines the price (p(q)) the agent offers at a certain time step as a function of the quantity (q), is presented in (1). ( q) = MC( q) ( 1 + ϑ) p (1) A. Agent-based Modeling Some shortcomings of traditional methodologies led to the development of new methods for exploring and analyzing the complexities of social dynamics observed in markets. One of these emerging developments is the Agent-based Modeling (ABM). Traditional social science generally assumes that social facts such as markets or cooperative behavior exist and they produce various forms of social organization and structure. By contrast, ABM assumes that both social structure and such social facts are created from the bottom up via the interactions of individual agents. Rather than examining how social structure shapes behavior, ABM focuses on how interactions among agents create larger social structures and patterns of behavior [8]. Owing to its bottom up characteristics, ABM enables to recognize strategic behavior in power markets such as tacit collusion through the analysis of interactions among agents with different characteristics and attributes. The ABM defines an agent as an autonomous computational entity with flexible behavior capable of interacting with its environment to achieve its design targets. Intelligent agents have the capability to evaluate the environment and their position in it and take own decisions based on their learning abilities [9]. Applying ABM, a spot power market, in which generation agents carry out their transactions, is modeled. A diagram of the proposed agentbased model is presented in Fig. 1. Fuel costs Consumption functions Demand Generation Agent Target: Maximization of profits Bidding Fig. 1. Proposed agent-based model. Market Agent Settlement Target: Maximization of the social welfare Bidding strategies Market clearing prices B. Reinforcement Learning In order to reproduce their bounded rational and adaptable behavior, the generation agents are endowed with learning abilities by means of Reinforcement Learning (RL), which implies the learning of how to map situations to actions in order to maximize a numerical reward signal [10]. The agent, who is the learner and decision-maker, is not advised which actions to take, but instead must discover which ones yield the highest reward by experimenting them. The agent and its environment interact at each of a sequence of discrete time steps t. At each time step, the agent receives certain representation of the environment s state s t S, and on that basis selects an action a t A(s t ). After the time step has been completed, the agent receives a numerical reward r t+1 R

3 193 3 proportional to its realized profits and finds itself in a new state s t+1. Q t (s t,a t ) is defined as the value function for the action a t given the state s t. Specifically, Q-Learning, which is an algorithm based on RL, is applied in this research. The updating rule of the value function is given in (2). Q t+ 1 α ( st+ 1, at+ 1 ) = Qt ( st, at ) + r + ( ) ( ) t+ 1 δ max Q st+ 1, at+ 1 Qt st, at a α is a learning factor and δ a discount parameter. When the number of iterations t, then Q(s,a) Q*(s,a), where Q*(s,a) represents the actual value of the action a for the state s. For practical reasons, the number of iterations is limited, but kept large enough to ensure that the reached equilibrium lies in the proximity of the theoretical solution. A look-up table is used to record the values of Q(s t,a t ). One important challenge that arises by applying RL is the trade-off between the exploration of new strategies and the exploitation of opportunities. Softmax Action Selection based on a Boltzmann distribution function is applied as policy function to balance exploration and exploitation. III. CASE STUDIES The test system consists of 9 thermal generation technologies and 100 generating units accounting for MW. Each generation agent owns a certain group of generating units. The applied consumption functions are typical of the corresponding generation technologies. The fuel costs match those observed in the German power market in 06 considering the cost of an emissions certificate equal to 12 /EUA. The Herfindahl-Hirschmann Index (HHI) is applied to quantify the market concentration of each simulated market structure. The HHI of a market is defined as the sum of the squares of the market shares of all generation agents multiplied by The U.S. Department of Justice divides the spectrum of market concentration as measured by the HHI into three regions that can be broadly characterized as unconcentrated (HHI < 1000), moderately concentrated (1000 HHI 10), and highly concentrated (HHI > 10) [11]. The characteristics of the modeled market structures are resumed in the Table I. Market Structure TABLE I CHARACTERISTICS OF THE MARKET STRUCTURES Generation Generating agents units HHI Level of concentration Learning abilities PCM unconcentrated no 100 GA unconcentrated yes 10 GA moderate yes 5 GA high yes (2) In the PCM market structure, the generation agents are not endowed with learning abilities, and thus, they offer their marginal generation costs. In the others, the generation agents develop their own bidding strategies maximizing their daily profits by means of their learning abilities. The demand scenarios are based on the information provided by the four largest German transmission system operators (RWE, E.ON, EnBW and Vattenfall). In order to obtain the demand expressed on a common year basis, the available time series were detrended by using the growth rate of monthly energy consumption. The year 05 was selected as base year. The demand series were scaled appropriately considering the total installed generation capacity. In order to analyze the influence of the transmission constraints on the bidding strategies of the generation agents, two cases are simulated. Whereas in Case A the transmission constraints are disregarded, in Case B a 5-node system with transmission constraints is modeled. IV. SIMULATION RESULTS A. Case A: transmission constraints disregarded The hourly prices for the four market structures are presented in Fig. 2. Price ( /MWh) Price ( /MWh) (a) January Working day Saturday Sunday Hour (h) (b) July Working day Saturday Sunday Hour (h) Fig 2. Simulated hourly prices.

4 193 4 Each diagram depicts 24 hourly prices for a typical working day, a Saturday and a Sunday respectively. Two different load scenarios are considered, (a) January and (b) July. The simulated prices result to be well above the perfectly competitive values, especially with the 10 GA and 5 GA market structures. In comparison with the PCM, the prices of the 100 GA market structure are in average 1.93% higher in January and 2.18% higher in July. The figures soar in the other modeled scenarios. With the 10 GA, the prices are 44.53% and 41.03% higher respectively, and in the case of the 5 GA, they are 72.37% and 67.04% higher respectively. Clearly, the more concentrated the market is, the higher the prices are. In Fig. 3, the total monthly generation costs, producer surpluses and total revenues of the generation agents are depicted for the four simulated market structures. Additionally, the percentage variation of the total monthly revenues in comparison to those of the PCM market structure is presented (a) January % % % % Generation Costs (b) July % % Producer Surplus Fig 3. Case A: monthly revenues, generation costs and producer surplus. Since the dispatch of the generating units is approximately equal, the generation costs result to be equal for all market structures. By contrast, the producer surpluses are different due to the exercise of market power, namely tacit collusion, by the generation agents. Whereas by the 100 GA market structure, the total revenues are faintly higher than those of the PCM, the figures for the other cases rise sharply. The revenues in January for the 10 GA market structure are slightly lower than 1300 million Euros and for the 5 GA they reach 15 million Euros. The variation pattern in July is similar to that observed in January but the figures result to be lower due to the lower demand level. Assuming that the behavior of the generation agents relies only on a pure learning process to coordinate actions among them, the results verify the ability to achieve tacitly collusive agreements bidding prices higher than the marginal generation costs in order to obtain higher revenues. This outcome can also be observed in real power markets. In [12], applying benchmarking analysis, the author shows that the prices of the German spot market (EEX) between September 01 and June 03 were nearly 50% above the estimated generation costs. In [13], it is also reported that in 05 the EEX prices were in average % higher than the corresponding generation costs. In both researches, the authors attribute these differences between the generation costs and the prices to market power and to the learning ability of the market participants, i.e. tacit collusion. Additionally, the simulation results show clearly that the more concentrated the market is, the higher the bidding prices are, and consequently the higher the market prices are. Theoretical analyses, which relate the exercise of market power with the market concentration applying oligopoly models, support these results [14]. However, sparse research on this field has been carried out applying simulation models. B. Case B: transmission constraints considered In order to analyze the influence of the transmission network on the exercise of tacit collusion, in Case B, a 5-node system with transmission constraints is modeled. The configuration of the network is depicted in Fig. 4. The installed generation capacities available in each node as well as the maximal transmission capacities of the lines expressed in MVA are also indicated in Fig. 4. D4 D1 N4 G4=13003 MW L1=1700 MVA N1 L2=1700 MVA G1= 8217 MW Fig. 4. Modeled transmission network. G3=8295 MW N3 N2 N5 D3 D2 D5 L3=10 MVA G2=12485 MW L4=1700 MVA G5=2421 MW In Fig. 5, the monthly generation costs, producer surpluses and total revenues of the generation agents are depicted for the

5 193 5 four simulated market structures. Additionally, the percentage variation of the total monthly revenues in comparison to those of the PCM market structure is presented. In comparison with Case A, the revenues of Case B considering a 5-node system with transmission restrictions are slightly higher for the 100 GA market structure. This increase indicates that certain generating units can obtain higher prices exploiting the transmission constraints of the system % (a) January % % % Generation Costs (b) July + 47.% % Producer Surplus Fig. 5. Case B: monthly revenues, generation costs and producer surplus. Whereas for the 10 GA market structure the revenues increase faintly with respect of those of the Case A, the revenues for the 5 GA reduce appreciably. These figures suggest that, under the considered circumstances, the transmission constraints do not facilitate the exercise of tacit collusion. Considering that the generating units of each generation agent are connected to the same node, the figures suggest that the generator agents cannot increase the exercise of tacit collusion when the concentration of the marker rises because they receive different price signals. The fact that there are different node prices hampers their abilities to assess the payoff functions and behaviors of the other market agents, and consequently, to collude tacitly. V. CONCLUSION In this article, an agent-based model is proposed to analyze how market participants can learn tacitly collusive behavior. The competition among market participants is modeled as a repeated game with imperfect public information. Reinforcement learning is applied to model the flexible and adaptable behavior of the market participants. The research focuses on the economic quantification of the tacit collusion in a test system by comparing the perfectly competitive prices and revenues with those obtained considering the learning abilities of the generation agents to develop bidding strategies. Furthermore, the relation between market concentration and market power is proved and quantified economically. The main contribution of this research is the simulative analysis of the tacit collusion considering the learning abilities of the generating agents. Another contribution is the investigation of the influence of the market concentration and the transmission constraints on the bidding strategies of the market participants. The presented results show that the proposed model is able to reproduce the learning process observed in repeated auctions. The results also demonstrate that the generation agents are capable to develop bidding strategies colluding tacitly to maximize their profits above the competitive levels. REFERENCES [1] F. Wen, and A. David, "Oligopoly electricity market production under incomplete information", IEEE Power Eng. Rev., vol. 21, pp , Apr. 01. [2] R. W. Ferrero, J. F. Rivera, and S. M. Shahidehpour, "Application of games with incomplete information for pricing electricity in deregulated power pools", IEEE Trans. on Power Systems, vol. 13, pp , Feb [3] J.-B. Park, B. Kim, J.-H. Kim, M-H. Jung, and J.-K. Park, "A continuous strategy game for power transactions analysis in competitive electricity markets", IEEE Trans. on Power Systems, vol. 16, pp , Nov. 01. [4] The European Commission. (01). Assessment criteria for distinguishing between competitive and dominant oligopolies in merger control. Enterprise Papers [Online]. N 6. Available: [5] E. Gnansounou, J Dong, S. Pierre, and A. Quintero, "Market oriented planning of power generation expansion using agent-based model", in Proc. 04 IEEE PES Power Systems Conference and Exposition, vol. 3, pp , Oct. 04. [6] D. Bunn, and F. Oliveira, "Agent-based simulation-an application to the New Electricity Trading Arrangements of England and Wales", in Trans. on Evolutionary Computation, vol. 5, pp , Oct. 01. [7] J. Nicolaisen, V. Petrov, and L. Tesfatsion, "Market power and efficiency in a computational electricity market with discriminatory double-auction pricing", in Trans. on Evolutionary Computation, vol. 5, pp , Oct. 01. [8] B. Berry, D. Kiel, and E. Elliot, Adaptive agents, intelligence, and emergent human organization: capturing complexity through agentbased modeling, in Proc. of the National Academy of Science, vol. 99, pp , May 02. [9] G. Weiß, Multi-agent systems: a modern approach to distributed artificial intelligence. Cambridge: The MIT Press, 00. [10] R. Sutton, and A. Barto, Reinforcement learning: an introduction. Cambridge: The MIT Press, [11] Horizontal Merger Guidelines, U.S. Department of Justice and the Federal Trade Commission (1992). [Online]. Available:

6 193 6 [12] F. Müsgens, "Market power in the German wholesale electricity market", Energiewirtschaftliches Institut (EWI), Working Paper WP 04.03, May 02. [Online]. Available: [13] C. Lang, and H.-G. Schwarz, "Quantifizierung von Marktmacht am deutschen Stromerzeugungsmarkt", Energiewirtschaftliche Tagesfragen, vol. 56, Dec. 06, pp [14] J. Tirole, Industrieökonomik. München: Oldenbourg-Verlag, 1999, pp Pablo Frezzi (S 04) was born in Córdoba, Argentina on May 18, He graduated as electrical engineer from the Universidad Tecnológica Nacional, Argentina in 02. He worked in the engineering division of Arcor, Argentina. Since March 03, he works in the Instituto de Energía Eléctrica of the National University of San Juan, Argentina, where he is pursuing his PhD. in Electrical Engineering. Since October 05, he is a visiting scholar at the Institute of Power Systems and Power Economics of the RWTH Aachen University, Germany. His research interests are liberalized electricity markets, and the assessment of oligopolistic behaviors applying game theory and agent-based modeling. Francisco Garcés was born in San Juan, Argentina, on April 26, He graduated as an electromechanical engineer from the University of San Juan and obtained his Ph.D. at the Aachen University of Technology, Germany. Dr. Garcés is presently professor for electrical engineering at Instituto de Energía Eléctrica, University of San Juan, and researcher of the Argentinean Research Council CONICET. His research interests include Power System Adequacy and Reliability, Risk Management, Dynamic of Generation and Transmission Investments and Regulatory Issues in Liberalized Electricity Markets. Hans-Jürgen Haubrich (M 00) was born in Montabaur, Germany, on March 1, He studied Electrical Engineering at Darmstadt University of Technology where he graduated in 1965 (Dipl.-Ing.). Thereafter, he was member of the scientific staff at the Institute of Electrical Energy Supply of Darmstadt University of Technology where he graduated in 1971 (Dr.-Ing.) he was freelance for Brown Bovery AG, Mannheim, Germany and he was member of the staff of Vereinigte Energiewerke Westfalen AG, finally as head of the Central Planning Department. In 1985 he was appointed as honorary professor at the University Bochum. Since 1990 he is Professor and head of the Institute of Power Systems and Power Economics at the RWTH Aachen University. Since 1997 he is additionally member of the Academy of Science of the federal state North-Rhine Westphalia. Since 03 he is director of the Forschungsgemeinschaft für elektrische Anlagen und Stromwirtschaft e.v. (FGH), Mannheim, Germany.