Papers by William Johnston
We propose a comprehensive strategy that makes use of the entire complex of the Office of Science... more We propose a comprehensive strategy that makes use of the entire complex of the Office of Science national laboratories and investment in university scientific and computer science communities. The proposed initial expression of the program will be in three of the multipurpose DOE laboratories: Argonne, Berkeley Lab, and Oak Ridge. Even in this initial implementation, we propose a collaboration between these three laboratories, together with the Princeton Plasma Physics Laboratory, Pacific Northwest National Laboratory, Brookhaven National Laboratory, and a collaboration of the special-purpose laboratories who lead programs in accelerator design and quantum chromodynamics calculations. The cost of scientific supercomputing is an issue of national strategic importance. The strategy we propose to implement will pursue at least three options:
Proceedings of the …, 1994
In this paper we describe several aspects of implementing a high speed network-based distributed ... more In this paper we describe several aspects of implementing a high speed network-based distributed application. We describe the design and implementation of a distributed parallel storage system that uses high speed ATM networks as a key element of the architecture. The architecture ...
Much of modern science is dependent on high performance distributed computing and data handling. ... more Much of modern science is dependent on high performance distributed computing and data handling. This distributed infrastructure, in turn, depends on high speed networks and services-especially when the science infrastructure is widely distributed geographically-to enable the science because the science is dependent on high throughput so that the distributed computing and data management systems will be able to analyze data as quickly as instruments produce it. Two network services have emerged as essential for supporting high performance distributed applications: Guaranteed bandwidth and multi-domain monitoring. Guaranteed bandwidth service-typically supplied as a virtual circuit-is essential for time critical distributed applications, as most science applications are. Detailed monitoring and active diagnosis are critical to isolating degraded network elements that inhibit "high performance use of the network." This paper discusses the design of the OSCARS virtual circuit service, its evolution as user experience showed some issues with the original design, and its deployment in a large production network. The result of the deployment is that ESnet is a fully integrated, hybrid packet-circuit network infrastructure.
— With the advent of service sensitive applications such as remote controlled experiments, time c... more — With the advent of service sensitive applications such as remote controlled experiments, time constrained massive data transfers, and video-conferencing, it has become apparent that there is a need for the setup of dynamically provisioned, quality of service enabled virtual circuits. The ESnet On-Demand Secure Circuits and Advance Reservation System (OSCARS) is a prototype service enabling advance reservation of guaranteed bandwidth secure virtual circuits. OSCARS operates within the Energy Sciences Network (ESnet), and has provisions for interoperation with other network domains. ESnet is a high-speed network serving thousands of Department of Energy scientists and collaborators worldwide. OSCARS utilizes the Web services model and standards to implement communication with the system and between domains, and for authentication, authorization, and auditing (AAA). The management and operation of end-to-end virtual circuits within the network is done at the layer 3 network level. Mu...
DOE Office of Science and ESnet-the ESnet Mission • ESnet's primary mission is to enable the larg... more DOE Office of Science and ESnet-the ESnet Mission • ESnet's primary mission is to enable the largescale science that is the mission of the Office of Science (SC) and that depends on:
Real-Time Digital Libraries Based on Widley Distributed, High Performance Management of Large Data Objects
International Journal on Digital Libraries, 1997
We describe a distributed, wide area network basedapproach to collecting, cataloguing, storing, a... more We describe a distributed, wide area network basedapproach to collecting, cataloguing, storing, and providing Webaccess for large-data-objects that originate as high-speed datastreams. Such data streams result from the operation of manytypes of on-line instruments and imaging systems, and are astaple of modern intelligence, scientific, and health careenvironments. The approach provides for real-time conversionof the data streams and large datasets to
Modern scientific computing involves organizing, moving, visualizing, and analyzing massive amoun... more Modern scientific computing involves organizing, moving, visualizing, and analyzing massive amounts of data from around the world, as well as employing large-scale computation. The distributed systems that solve large-scale problems will always involve aggregating and scheduling many resources. Data must be located and staged, cache and network capacity must be available at the same time as computing capacity, etc. Every aspect of such a system is dynamic: locating and scheduling resources, adapting running application systems to availability and congestion in the middleware and infrastructure, responding to human interaction, etc. The technologies, the middleware services, and the architectures that are used to build useful high-speed, wide area distributed systems, are now being integrated in "Grids" [1]. This paper explores some of the background, current state, and future directions of Grids.
Wiley Series in Communications Networking & Distributed Systems
Starting from Section 2, "The Grid Context," we lay out our view of a Grid architecture, and this... more Starting from Section 2, "The Grid Context," we lay out our view of a Grid architecture, and this definition provides a structure for the subsequent detailed description. In particular we identify what it is that differentiates a Grid from other structures for distributed computing, e.g. hierarchical clusters. The question of what is a minimum set of Grid services-the Grid Common Services, the neck of the hourglass model-and what needs to be added to make the Grid usable for particular communities is stated. Issues of interoperability and heterogeneity are addressed, and these are perhaps the most important distinguishing features of a Grid.
Tools for Building Virtual Laboratoriesa
Thispaper describes two pilot projects which provide testbeds for the tools. The first is a virtu... more Thispaper describes two pilot projects which provide testbeds for the tools. The first is a virtuallaboratory project providing remote access to LBNL's Advanced Light Source. The second isthe Multidimensional Applications and Gigabit Internetwork Consortium (MAGIC) testbedwhich has been established to develop a very high-speed, wide-area network to deliver realtimedata at gigabit-per-second rates.1 MotivationA collection of emerging technologies, ranging from distributed
Enabling high throughput in widely distributed data management and analysis systems: Lessons from the LHC
The effect of distributed computing technology on wide area network capacity requirements
International Journal of High Speed Computing, 1997
This paper presents arguments supporting the thesis that clusters of SMPs interconnected with gen... more This paper presents arguments supporting the thesis that clusters of SMPs interconnected with general purpose networks (perhaps several different networks to support both low-latency and bulk data transfers) are the architecture that will provide scalable high-performance computing environments. The paper presents an analysis of the high performance computing environment based on the state of technology in early 1996. The evolution of the technology base since that time has only strengthened that author's opinion that this architecture will be successful.
IEEE Intelligent Systems, 2004
This paper briefly describes Grid technology for supporting large scale science. It examines seve... more This paper briefly describes Grid technology for supporting large scale science. It examines several examples of how the process of science must evolve over the next five to ten years in order to facilitate the next steps in scientific discovery. In this context it examines the need and role of semantic description, management, and manipulation of science simulations and data. In conclusion it provides several examples of the potential (even essential) value of semantic tools in dealing with the greatly increased complexity of the multidisciplinary simulation and data environments required for next generation science.
Computer Physics Communications, 1998
Advances in data acquisition and dissemination technologies, together with several other computer... more Advances in data acquisition and dissemination technologies, together with several other computer science technologies, provide an opportunity to de ne new environments for remote access to scienti c instruments and collaboration between researchers at remote sites. These environments, called \collaboratories" 1], are intended to provide complete location-independent collaborative access to instruments, data acquisition and analysis resources, as well as to collaborating researchers. Over the past two years we have built a prototype collaboratory at the Spectro-Microscopy Facility of the Advanced Light Source (ALS) Beamline 7.0 at Lawrence Berkeley National Laboratory. The scientists who use the Beamline are based at di erent cities around the world. Before the introduction of the collaboratory, experiments at the Beamline required that multiple researchers travel to Berkeley, CA. With the collaboratory, scientists can now discuss experiments with other scientists around the world and at the Beamline, \walk" around the Beamline using remotely controlled cameras, access data as it is taken, monitor instrument parameters, and do limited control of the experiment from their o ces. This paper presents the design of the prototype Spectro-Microscopy Collaboratory, and the lessons learned in providing remote access to the facility. It also presents the software architecture and components that are being developed to provide a toolkit for bringing future experimental equipment on-line.
Most distributed applications today manage to utilize only a small percentage of the needed and a... more Most distributed applications today manage to utilize only a small percentage of the needed and available network bandwidth. Often application developers are not aware of the potential bandwidth of the network, and therefore do not know what to expect. Even when application developers are aware of the specifications of the machines and network links, they have few resources that can help determine why the expected performance was not achieved. What is needed is a ubiquitous and easy-to-use service that provides reliable, accurate, secure, and timely estimates of dynamic network properties. This service will help advise applications on how to make use of the network's increasing bandwidth and capabilities for traffic shaping and engineering. When fully implemented, this service will make building currently unrealizable levels of network awareness into distributed applications a relatively mundane task. For example, a remote data visualization application could choose between sending a wireframe, a pre-rendered image, or a 3-D representation, based on forecasts of CPU availability and power, compression options, and available bandwidth. The same service will provide on-demand performance information so that applications can compare predicted with actual results, and allow detailed queries about the end-to-end path for application and network tuning and debugging
Computing in High Energy Physics '95, 1996
There is increasing interest in makhg unique research facilities accessible on the Internet. Comp... more There is increasing interest in makhg unique research facilities accessible on the Internet. Computer systems, scientific databases and experimental apparatus can be used by international collaborations of scientists using high-speed networks and advanced software toofs to support collaboration. We arc building toots includhrg video conferencing and electronic whiteboards that are being used to create examples of virtual laboratories. This paper describes two pilot projects which provide testbeds for the toots. The first is a virtual laboratory project providing remote access to LBNL's Advanced Light Source. The second is the Multidimensional Applications and Gigabit Intemetwork Consortium (MAGIC) testbed which bas been established to develop a very high-speed, wide-area network to deliver real. time data at gigabit-per-second rates. 1 Motivation A collection of emerging technologies, ranging from distributed data handling and distributed computing to the underlying multicast network infrastructure, have the potential to make it possible to create distributed scientific and industrial laboratory environments that provide complete location independent access to instruments, data handling and analysis resources, and enable remote human collaboration. These technologies will produce not so much an incremental change of today's routine use of PCs and LANs in laboratory environments, but rather the introduction of a new paradigm more akin, perhaps, to virtual reality. The vision is for a scientist thousands of miles away to get the same (or even better) sense of presence and control as if he or she were at the experiment site. The expected value of these geographically distributed environments includes substantially increased effectiveness in doing science by more closely integrating university and federal researchers, and an enabling capability for analytical and high-value production use of unique facilities by industry.
Belle-II Experiment Network Requirements
Real-Time Generation and Cataloguing of Large Data-Objects in Widely Distributed Environments
Uploads
Papers by William Johnston