The programme is available here:
You can participate through Twitter #cloudwg and crowdsourced write-up via Etherpad
In addition to our main annual workshop in February next year, we’re also running a smaller pre-meeting this coming month in central London. The goal of this event is to provide a space specifically for developers, researchers and devops to take a deep dive into technologies for cloud, share from their own experience and learning from each other. We’ve deliberately avoided setting a fixed timetable so that we can source topics from attendees on the day. More details and booking information for the day here:
Make sure to bring your laptop 🙂
We will be holding our 4th annual workshop early next year on the 12th February 2019. We’re pleased to be back at our familiar venue the Francis Crick Institute in central London. Please save the date!
In past years we’ve had a great set of speakers from public cloud companies and major research institutes to individual researchers reporting on how they are exploiting cloud computing to meet their research goals. More details to follow soon.
The workshop is now just under a few days away. You can see the programme for the day below. We have a broad range of contributions from across the research community and also good representation from public cloud providers. This year we are focussing on international collaborations for our plenary session. Other sessions focus on mix of application use – from where cloud adoption has reached a mature state – to others where we are examining specific technical and policy related challenges to be addressed.
8th January, Francis Crick Institute London, 1 Midland Road, London, NW1 1AT
|09:00||Arrivals, registration, refreshments (Gallery Area)|
|09:45||Session 1 – International Collaborations
|Future Science on Future OpenStack: developing next generation infrastructure at CERN and SKA – Stig Telfer, StackHPC|
|EOSC-hub: overview and cloud federation activities – Enol Fernández, EGI|
|Public Clouds, OpenStack and Federation – Ildikó Vancsa, OpenStack Foundation|
|10:45||Break (Gallery area)|
|11:15||Session 2a – Technical Challenges – Containers, portability of compute, data movement
|Session 2b – Practical challenges
|Running a Container service with OpenStack/Magnum – Spiros Trigazis, CERN||Aerospace and Cloud – Leigh Lapworth, Rolls Royce|
|Large scale Genomics with Nextflow and AWS Batch – Paolo Di Tommaso, Centre for Genomic Regulation; Brendan Bouffler, AWS||Processing patient identifiable data in the cloud – what you need to consider technically and process wise to keep your data safe – Peter Rossi, UKCloud|
|Best practice in porting applications to Cloud – Dario Vianello, EMBL-EBI||Jisc ExpressRoute Circuit Service, David Salmon and Gary Blake, Jisc|
|Demystifying Hybrid Cloud with Microsoft Azure – Mike Kiernan, Microsoft||The Janet End-to-End Performance Initiative – Duncan Rand, Jisc|
|Question time||Question time|
|12:30||Lunch (Gallery area)|
|13:30||Session 3a Innovative applications, usability and training
|Session 3b – Virtual Laboratories and Research Environments
|Visualizing Urban IoT data using Cloud Supercomputing – Nick Holliman, Newcastle University||CLIMB – Thomas Connor, Cardiff University / Nick Loman, Birmingham University||ResOps training – Erik van den Bergh, EMBL-EBI|
|Accelerate time-to-insight with a serverless big data platform – Hatem Nawar, Google Cloud||CyVerse UK: a Cloud Cyberinfrastructure for life science – Alice Minotto, Earlham Institute|
|Azure at the Turing – Martin O’Reilly, Turing Institute||EBI Cloud Portal – Jose Dianes, EMBL-EBI|
|HPC – There’s plenty of room at the bottom – Mike Croucher, University of Sheffield||Data Labs: A Collaborative Analysis Platform for Environmental Research – Nick Cook / Josh Foster, Tessella|
|Question time||Question time|
|14:45||Break (Gallery area)|
|15:15||Session 4a – Technical Challenges – batch compute on cloud
|Session 4b – Technical Challenges – Storage
|Matching to cloud technologies to Theoretical Astrophysics and Particle Physics applications – Jeremy Yates, UCL||Semantic Storage of Climate Data on Object Store – Neil Massey, NCAS / Centre for Environmental Data Analysis, STFC|
|Hybrid HPC – on-premise and cloud – Wil Mayers, Alces Flight||Accessing S3 from FUSE – Jacob Tomlinson, Informatics Lab|
|Running HPC Workloads on AWS using Alces Flight – Igor Kozin, ICR||OpenStack Manila – John Garbutt, StackHPC|
|OpenFOAM batch compute on AWS – James Shaw, Reading University||Providing Lustre access from OpenStack – Thomas Stewart / Francesco Gianoccaro, Public Health England|
|Implementing medical image processing platform using OpenStack and Lustre – Wojciek Turek, Cambridge University|
|Question time||Question time|
|Feedback, next steps, cloud strategy for research community, sum-up|
|17:00||Reception (Gallery area)|
We are hosting our next workshop in the New year. We are pleased to be returning to the Francis Crick Institute who hosted at last year’s event. You can read more about the meeting themes and registration details here:
We welcome submissions for talks, posters or proposals for breakout sessions.
It’s been a few months since our November workshop so there’s been some time to digest and reflect on some of the common themes emerging. Having attended a couple of other conferences and workshops from my community (AGU and AMS) it’s interesting to compare.
Firstly, it was great to see such a variety of application areas represented. For this our second annual workshop, we opened it for the submission of abstracts and this made such a difference. There was a great response, Life sciences having the margin on other domain areas. We had 160 register and 120 on the day. It was fantastic to have the Crick as a venue. It worked really well.
The first session looked at applications of hybrid and public cloud. Two really interesting use cases (Edinburgh and NCAS, NERC) looked at trying out HPC workloads on public cloud. This raised issues around comparative performance and costs between public cloud and on-prem HPC facilities.
On AWS, Placement Groups allow instances to put close to one another to improve inter-node communication for MPI-based workloads. This showed comparable performance with Archer (UK national supercomputer) for smaller workloads but clearly there was some limit as this tailed off as the number of nodes increased whereas Archer performance continued linearly with increase in scale. This tallies with what I’ve seen anecdotally at the AMS conference where there seemed to be on the one hand increasing uptake of public cloud for use with Numerical Weather Prediction jobs (which need MPI). However, this seemed to be being done for smaller scale workloads where they can stay within the envelope of the node affinity features available.
Another theme was portability – what kind of approaches can be used engineer workloads so that they can be easily moved between providers. Andrew Lahiff from STFC, presented a very different use case showing how container technologies can be used to run for Particle Physics, cases where there the focus is high-throughput rather than HPC requirements and so much more amenable to cloud. This work has been done a part of a pilot for the Cloud Working Group to specifically investigate how containers and container orchestration technology can be used to provide an abstraction layer for cloud interoperability. A really nice slide showed Kubernetes clusters running on Azure and Google cloud managed from the same command line console app. Dario Vianello’s talk (EMBL-EBI) showed how an alternative approach using a combination of Ansible and Terraform can be used to deploy workloads across multiple clouds.
It was great to have talks from hyper-scale cloud providers AWS, Azure and Google. The scale in hyper-scale is as ever impressive as is the pace of change in technology: very interesting to see Deep Learning applications driving the development of custom hardware – TPUs and FPGAs. Plans underway to host data centres in the UK will ease uptake. OpenStack Foundation and Andy McNab‘s talks showed examples of federation across OpenStack clouds.
In the private cloud session, Stig Tefler gave a nice illustration of network performance for VMs showing how a number of aspects of virtualisation can be changed or removed to progressively improve network performance towards line rate. Alongside talks on private cloud, the parallel session looked at Legal, Policy and Regulatory Issues a critical issue for adoption of public cloud. Steven Newhouse gave some practical insights from a recent cloud tender for EMBL. There is clearly need for further work around these issues so that the community can better informed about choices. This is something that the working group will be taking forward.
For this workshop, we experimented with an interactive session – bringing together a group of around 20 delegates to work together on some technical themes agreed ahead of time including bulk data movement and cloud-hosting of Jupyter notebooks. There was plenty of useful interaction and discussion but we will need to look at the networking provision for the next time to ensure groups can get on with technical work on the day.
We discussed next steps in the final session. There is clearly interest in taking particular areas forward from the meeting: focus groups on technical areas like HTC and use of parallel file systems with cloud or organised around specific domains within the research community. Training figured also, in the form of a cloud carpentry course so that researchers can more readily get up and running using cloud. Looking forward, in each of these case we’re looking for discrete activities with an agreed set of goals and something to deliver at the end. Where possible we’re seeking to support relevant work that is already underway and initiate new work where there are perceived gaps. We will be looking at running smaller workshops targeted at specific themes in the coming months as a means to engage and disseminate some of this work.
Phil Kershaw, STFC & Cloud-WG chair