Autonomous Model Learning for Reinforcement Learning

• Abstract:
  Stochastic modeling is an excellent way of capturing system dynamics so that alternative control strategies can be evaluated and compared. I will discuss attributes that make some problems amenable to autonomous learning of system dynamics. I will then present recent advances in my lab concerning the design of learning algorithms with formal learning-time guarantees in the "KWIK" (knows what it knows) formalism along with their implementation on robotic and software control problems.
• Short bio:
  Reinforcement Learning (RL3) and his research in machine learning examines algorithms for decision making under uncertainty. After earning his Ph.D. from Brown University in 1996, Littman worked as an assistant professor at Duke University, a member of technical staff in AT&T's Artificial Intelligence Principles Research Department, and is now a professor of computer science at Rutgers. Both Duke and Rutgers honored him with university teaching awards and his research has been recognized with three best-paper awards on topics of computer crossword solving, complexity analysis of planning under uncertainty, and algorithms for efficient reinforcement learning. He has served as associate editor for three of the major journals in his field.

Composing Web Services in an open world: QoS issues

• Abstract:
  Orchestrating Web services has become the method of choice for building new services on top of existing ones, e.g., for business processes. Languages and methods have been developed and are now getting widely used, BPEL being the typical instance. When exposing the profile of a Web service, QoS parameters are typically specified. Besides security aspects, QoS involves a variety of parameters related to performance as well as quality of the returned data. How QoS should be handled in this context is the subject of my talk. A number of novel and not so well identified issues occur that make this topic deviating from QoS for networks in a substantial way. Firstly, since Web services aim at hiding details for the external world, no information regarding the infrastructure or resources supporting a Web service are exposed. This prevents from using classical resource based performance models; so contracts are preferred instead. A second important feature is that, unlike in networks, the control in orchestrations may depend on the carried data. Consequently performance and data interfere. These and other features make the subject of QoS for composite Web services a novel area offering plenty of non standard issues that I shall discuss.
• Short bio:
  Albert Benveniste graduated from the Ecole des Mines in Paris and completed his These d'Etat in Mathematics, specializing in probability theory. From 1976 to 1979 he was associate professor in mathematics at the Universite de Rennes I. From 1979 to date he has worked as Director of Research at INRIA. In 1985, he co-invented the synchronous language ‘Signal for reactive systems design in computer science’ with Paul Le Guernic; he is now a recognized contributor to the topic of formal methods for heterogeneous distributed reactive systems in computer engineering. From 1986 to 1990 he was vice-chairman of the IFAC committee on Theory and was chairman of this committee for 1991-1993. He has been or he is Associate Editor (at Large) for IEEE Transactions on Automatic Control, Associate Editor for Int. J. of Adaptive Control and Signal Processing, and Int. J. of Discrete Event Dynamical Systems. He was also member of the Editorial Board of the Proceedings of the IEEE. In 1980 Albert Benveniste was joint-winner of the IEEE Trans. on Automatic Control Best Transaction Paper Award for his work on blind deconvolution in data communications. In 1990 he received the CNRS silver medal and in 1991 he was elected as an IEEE fellow.

Linear Programming, Lyapunov Functions, and Performance Analysis

• Abstract:
  Many of the stochastic models that are used in the performance engineering context can be viewed as Markov processes, evolving in either discrete time or continuous time. In this talk, we will discuss the use of Lyapunov functions in computing steady-state performance bounds for such Markov processes. We will further discuss how such bounds can be used to develop linear programming-based algorithms that are capable of accurately computing system performance for infinite state models, in which the Markov state descriptor is either a discrete or continuous variable. We will illustrate these linear programming ideas by discussing their application to numerical computation of stationary distributions of reflected Brownian motion (RBM); such RBMs arise as "heavy-traffic" limits of conventional queueing networks. This work is joint with Denis Saure and Assaf Zeevi.
• Short bio:
  IPeter Glynn received his Ph.D in Operations Research from Stanford University in 1982. He then joined the faculty of the University of Wisconsin at Madison, where he held a joint appointment between the Industrial Engineering Department and Mathematics Research Center, and courtesy appointments in Computer Science and Mathematics. In 1987, he returned to Stanford, where he is now the Thomas Ford Professor of Engineering in the Department of Management Science and Engineering. He also has a courtesy appointment in the Department of Electrical Engineering. He is a Fellow of the Institute of Mathematical Statistics and has research interests in computational probability, queueing theory, statistical inference for stochastic processes, and stochastic modeling.