Grid load balancing using intelligent agents
2005, Future Generation Computer Systems
https://doi.org/10.1016/J.FUTURE.2004.09.032Abstract
Workload and resource management are essential functionalities in the software infrastructure for grid computing. The management and scheduling of dynamic grid resources in a scalable way requires new technologies to implement a next generation intelligent grid environment. This work demonstrates that AI technologies can be utilised to achieve effective workload and resource management. A combination of intelligent agents and multi-agent approaches is applied for both local grid resource scheduling and global grid load balancing. Each agent is a representative of a local grid resource and utilises predictive application performance data and iterative heuristic algorithms to engineer local load balancing across multiple hosts. At a higher level of the global grid, agents cooperate with each other to balance workload using a peer-to-peer service advertisement and discovery mechanism. A case study is included with corresponding experimental results to demonstrate that intelligent agents are effective to achieve resource scheduling and load balancing, improve application execution performance and maximise resource utilisation.
References (42)
- A. Abraham, R. Buyya, and B. Nath, Nature's heuristics for scheduling jobs on computational grids, in: Proceedings of 8 th IEEE International Conference on Advanced Computing and Communications (Cochin, India, 2000).
- D. Abramson, R. Sosic, J. Giddy, and B. Hall, Nimrod: a tool for performing parameterized simulations using distributed workstations, in: Proceedings 4 th IEEE International Symposium on High Performance Distributed Computing (HPDC '95) (Pentagon City, VA, USA, 1995).
- D. Abramson, J. Giddy, and L. Kotler, High performance parametric modelling with Nimrod/G: killer application for the global grid, in: Proceedings 14 th International Parallel and Distributed Processing Symposium (Cancun, Mexico, 2000).
- K. Amold, B. O'Sullivan, R. Scheifer, J. Waldo, and A. Woolrath, The Jini™ Specification (Addison Wesley, 1999).
- F. Berman, R. Wolski, S. Figueira, J. Schopf, and G. Shao, Application-level scheduling on distributed heterogeneous networks, in: Proceedings
- Supercomputing '96 (Pittsburgh, PA, USA, 1996).
- F. Berman, A. J. G. Hey, and G. Fox, Grid Computing: Making The Global Infrastructure a Reality (John Wiley & Sons, 2003).
- C. Brooks, B. Tierney, and W. Johnston, JAVA agents for distributed system management, LBNL Report (1997).
- J. Cao, D. J. Kerbyson, E. Papaefstathiou, and G. R. Nudd, Performance modelling of parallel and distributed computing using PACE, in: Proceedings of 19 th IEEE International Performance, Computing and Communication Conference (IPCCC '00) (Phoenix, AZ, USA, 2000) pp. 485-492.
- J. Cao, D. J. Kerbyson, and G. R. Nudd, Dynamic application integration using agent-based operational administration, in: Proceedings of 5 th International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology (PAAM '00) (Manchester, UK, 2000) pp. 393-396.
- Cao, D. J. Kerbyson, and G. R. Nudd, High performance service discovery in large-scale multi-agent and mobile-agent systems, International Journal of Software Engineering and Knowledge Engineering 11(5) (2001) 621-641.
- J. Cao, D. J. Kerbyson and G. R. Nudd, Use of agent- based service discovery for resource management in metacomputing environment, in: Proceedings of 7 th International Euro-Par Conference (Manchester, UK, 2001), Lecture Notes on Computer Science Volume 2150 pp. 882-886.
- J. Cao, D. J. Kerbyson, and G. R. Nudd, Performance evaluation of an agent-based resource management infrastructure for grid computing, in: Proceedings of 1 st IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid '01) (Brisbane, Australia, 2001) pp. 311-318.
- J. Cao, D. P. Spooner, J. D. Turner, S. A. Jarvis, D. J. Kerbyson, S. Saini, and G. R. Nudd, Agent-based resource management for grid computing, in: Proceedings of 2 nd Workshop on Agent-based Cluster and Grid Computing (AgentGrid '02) (Berlin, Germany, 2002) pp. 350-351.
- J. Cao, S. A. Jarvis, S. Saini, D. J. Kerbyson, and G. R. Nudd, ARMS: an agent-based resource management system for grid computing, Scientific Programming (Special Issue on Grid Computing) 10(2) (2002) 135- 148.
- J. Cao, S. A. Jarvis, D. P. Spooner, J. D. Turner, D. J. Kerbyson, and G. R. Nudd, Performance prediction technology for agent-based resource management in grid environments, in: Proceedings of 11 th IEEE Heterogeneous Computing Workshop (HCW '02) (Fort Lauderdale, FL, USA, 2002).
- J. Cao, D. P. Spooner, S. A. Jarvis, S. Saini, and G. R. Nudd, Agent-based grid load balancing using performance-driven task scheduling, in: Proceedings of 17 th IEEE International Parallel and Distributed Processing Symposium (IPDPS '03) (Nice, France, 2003).
- H. Casanova, and J. Dongarra, Using agent-based software for scientific computing in the NetSolve system, Parallel Computing 24(12-13) (1998) 1777-1790.
- H. Casanova, and J. Dongarra, Applying NetSolve's network-enabled server, IEEE Computational Science & Engineering 5(3) (1998) 57-67.
- I. Foster and C. Kesselman, Globus: a metacomputing infrastructure toolkit, International Journal of High Performance Computing Applications 2 (1997) 115-128.
- I. Foster and C. Kesselman, The GRID: Blueprint for a New Computing Infrastructure (Morgan-Kaufmann, 1998).
- I. Foster, C. Kesselman, J. M. Nick, and S. Tuecke, Grid services for distributed system integration, IEEE Computer 35(6) (2002) 37-46.
- J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke, Condor-G: a computation management agent for multi-institutional grids, Cluster Computing 5(3) (2002) 237-246.
- A. Grimshaw, W. A. Wulf, and the Legion team, The Legion vision of a worldwide virtual computer, Communications of the ACM 40(1) (1997) 39-45.
- L. Henderson, Job scheduling under the Portable Batch System, in: Proceeding of 1 st Workshop on Job Scheduling Strategies for Parallel Processing (Santa Barbara, CA, USA, 1995), Lecture Notes in Computer Science Volume 949 pp. 279-294.
- D. Jackson, Q. Snell, and M. Clement, Core algorithms of the Maui scheduler, in: Proceedings of 7 th Workshop on Job Scheduling Strategies for Parallel Processing (Cambridge, MA, USA, 2001), Lecture Notes Computer Science Volume 2221 pp 87-102.
- N. R. Jennings and M. J. Wooldridge (eds), Agent Technology: Foundations, Applications, and Markets (Springer Verlag, 1998).
- D. Lifka, The ANL/IBM SP scheduling system, in: Proceeding of 1 st Workshop on Job Scheduling Strategies for Parallel Processing, 9 th IEEE International Parallel Processing Symposium (Santa Barbara, CA, USA, 1995), Lecture Notes in Computer Science Volume 949 pp. 187- 191.
- M. Litzkow, M. Livny, and M. Mutka, Condor -a hunter of idle workstations, in: Proceedings of 8 th International Conference on Distributed Computing Systems (ICDCS '88) (San Jose, CA, USA, 1988) pp. 104-111.
- L. Moreau, Agents for the grid: a comparison for web services (part 1: the transport layer), in: Proceedings of 2 nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid '02) (Berlin, Germany, 2002) pp. 220-228.
- H. Nakada, M. Sato, and S. Sekiguchi, Design and implementations of Ninf: towards a global computing infrastructure, Future Generation Computing Systems 5- 6 (1999) 649-658.
- E. Newcomer, Understanding Web Services: XML, WSDL, SOAP, and UDDI (Addison Wesley, 2002).
- G. R. Nudd, D. J. Kerbyson, E. Papaefstathiou, S. C. Perry, J. S. Harper, and D. V. Wilcox, PACE -a toolset for the performance prediction of parallel and distributed systems, International Journal of High Performance Computing Applications 14(3) (2000) 228-251.
- O. F. Rana, and D. W. Walker, The agent grid: agent- based resource integration in PSEs, in: Proceedings of 16 th IMACS World Congress on Scientific Computation, Applied Mathematics and Simulation (Lausanne, Switzerland, 2000).
- W. Shen, Y. Li, H. Ghenniwa, and C. Wang, Adaptive negotiation for agent-based grid computing, in: Proceedings of AAMAS Workshop on Agentcities: Challenges in Open Agent Environments (Bologna, Italy, 2002) pp. 32-36.
- D. Slama, J. Garbis, and P. Russell, Enterprise Corba (Prentice Hall,
- R. Stevens, P. Woodward, T. DeFanti, and C. Catlett, From the I-WAY to the national technology grid, Communications of the ACM 40(11) (1997) 50-60.
- C. Thompson, Characterizing the agent grid, Technical Report, Object Services and Consulting Inc. (1998) http://www.objs.com/agility/tech-reports/9812-grid.html. B. Tierney, W. Johnston, J. Lee, and M. Thompson, A data intensive distributed computing architecture for grid applications, Future Generation Computer Systems 16(5) (2000) 473-481.
- R. Wolski, N. T. Spring, and J. Hayes, The network weather service: a distributed resource performance forecasting service for metacomputing, Future Generation Computing Systems 15(5-6) (1999) 757-768.
- S. Zhou, LSF: load sharing in large-scale heterogeneous distributed systems, in: Proceedings of 1992 Workshop on Cluster Computing (1992).
- Zhuge, Semantics, resource and grid, Future Generation Computer Systems 20(1) (2004) 1-5.
- Zhuge, China's E-Science Knowledge Grid Environment, IEEE Intelligent Systems 19(1) (2004) 13- 17.