Two Sigma Investments, LP
100 Avenue of the Americas
New York, NY 10013
Javier Diaz-Montes is currently Software Engineer at Two Sigma. He received his PhD degree in Computer Science from the Universidad de Castilla-La Mancha (UCLM), Spain ("Doctor Europeus", Feb. 2010). Before joining Two Sigma, he was Assistant Research Professor at Rutgers University and a member of the Rutgers Discovery Informatics Institute (RDI2). He was also Postdoctoral Fellow at Indiana University.
His research interests are in the area of parallel and distributed computing and include Internet of Things, edge computing, cloud computing, streaming, virtualization, middleware, and scheduling.
I led the research effort around the CometCloud Project. CometCloud is an autonomic framework for enabling real-world applications on dynamically federated, hybrid infrastructures integrating (public & private) clouds, HPC data-centers and Grids.
The research in this project was focused on providing novel ways of orchestrating geographically distributed resources - including supercomputers, storage systems, IoT devices, clouds, and data analytic platforms - to enable the execution of large-scale scientific and business workflows. Specifically, we looked at how to combine cloud abstractions with software-defined environments techniques to create nimble computational environments that can dynamically and opportunistically federate distributed resources to satisfy changing user requirements, application needs, and resource availabilities. This included understanding how to leverage resources and services located at the edge of the infrastructure as well as through the network to enable the processing of data-in-motion that can achieve the desired trade-offs in terms of response time, quality of service, cost and execution efficiency.
I worked in the FutureGrid Project leading the development of FutureGrid Rain. Rain is a tool that will allow users to place customized environments like virtual clusters or IaaS frameworks onto resources. The process of raining goes beyond the services offered by existing scheduling tools due to its higher-level toolset targeting virtualized and non-virtualized resources. Rain will be able to move resources from one infrastructure to another, compare the execution of an experiment in the different supported infrastructures, and easily deploy full environments like Hadoop on different infrastructures using user's customized images.
Rain is supported by a flexible image management framework, which defines the full life cycle of the images in FutureGrid. It involves the process of creating, customizing, storing, sharing and registering images for different FutureGrid environments. To this end, we have develop several components to perform the different tasks involved. First, we have an Image Generation tool that creates and customizes images according to user requirements. The second component is the Image Repository, which is in charge of storing, cataloging and sharing images. The last component is an Image Registration tool, which prepares, uploads and registers images for specific environments, like HPC or cloud frameworks. It also decides if an image is secure enough to be registered or if it needs additional security tests.