The workshop was held on March 29th at the Francis Crick Institute in central London.

Registration is now closed.

Programme now includes links to slides – updated 27/04/2022

Timing	Plenary / Strand A Sessions Auditorium 2	Strand B Sessions Auditorium 1
09:15	Registration + teas/coffee
10:15	Introduction
10:25	Opening Plenary Chair: Cristin Merritt
	Reaching the cloud – Steven Chapman, University of Bath
	Question time
10:55	Break – teas/coffee
11:15	Session 1a – User Experiences Chair: Jay DesLauriers	Session 1b – User Experiences Chair: Cristin Merritt
	Transitioning research computing workloads to the cloud: A thematic approach at Cardiff University – Tom Green, Cardiff University	Global Symmetry is important for the detection of abnormality in mammograms – Cameron Kyle-Davidson, University of York
	On the creation of a secure ‘serverless’ workflow between a Mapbox frontend and a SalesForce backend for the Tekkatho Foundation – Mike Jones, Independent Researcher	Capturing a Moment in the Cloud Workshop: How our perception and use of cloud computing has changed over time – Wil Mayers, Alces Flight
	Reducing time-to-science with self-service HPC and AI platforms in the Azimuth portal – Matt Pryor, Stack HPC	Crawlers, Bots, Flows, Lambdas, Glues and Autopilots: Applying AI and ML to Radiological Sensor Networks for Safety and Security – Peter Martin, University of Bristol
	Question time	Question time
12:15	Lunch – buffet lunch
13:30	Plenary (invited speakers) Chair: Cristin Merritt
	Developing and using the UK Biobank Research Analysis Platform, a large-scale Trusted Research Environment – Oliver Gray and Przemyslaw Stempor, UK BioBank
	CLIMB-COVID: Cloud Infrastructure to Support the UK’s Covid-19 Response – Radoslaw Poplawski and Nick Loman, University of Birmingham
	Question time
14:15	Session 2a – Trusted Research Environments Chair: David Fergusson	Session 2b – HPC in the Cloud – Applications Chair: Stig Telfer
	TREEHOOSE: Trusted Research Environment and Enclave Hosting Open Original Scientific Exploration – Simon Li, University of Dundee	Twins in the Cloud: Simplifying the Deployment of Digital Twins for Manufacturing-as-a-Service – Jay DesLauriers, University of Westminster
	Data Safe Haven Classification and Trusted environments in the cloud: extending a Turing Django based application across multiple institutions – Rebecca Osselton, Newcastle University	An Introduction to DosNA: Distributed Numpy Arrays for High-performance cloud computing – Gabryel Mason-Williams, Rosalind Franklin Institute
	The Genes And Health TRE in the Google cloud – Vivek Iyer, Wellcome Sanger Institute	Maintaining versioned 3D digital designs using a hybrid and multi-cloud solution – Niall Kennedy, YellowDog
	Question time	Question time
15:15	Break – teas/coffee
15:35	Session 3a HPC in Cloud Chair: Alex Dibbo	Session 3b Covid & Cloud Chair: Stephanie Thompson
	The PITHIA-NRF e-Science Centre – towards a Cloud-based Platform to support Ionosphere, Thermosphere, and Plasmasphere Research – Tamas Kiss, University of Westminster	Supporting UK Covid-19 surveillance with AWS Step Functions and Fargate at Wellcome Sanger Institute – Sam Proctor, Wellcome Sanger Institute
	King’s CREATE: a new research computing ecosystem and the journey so far – Matt Penn, King’s College London	Laying the groundwork for discovering the next novel coronavirus – Brendan Bouffler, Amazon Web Services
	Ensuring fairer access and reducing obstacles to research in fixed capacity clouds – Paul Browne, University of Cambridge & Pierre Riteau, Stack HPC	Cloud-based nonequilibrium simulations to investigate regulation and environmental effects in the SARS-CoV-2 spike protein – Sofia Oliveira, University of Bristol
	Question time	Question time
16:35	Reconvene in Auditorium 2 …
16:45	Final Plenary Chair: James Grant
	DRI Update – Justin O’Byrne, UKRI Digital Research Infrastructure
17:05	Sum-up, feedback, next steps
17:10	Reception – drinks, light refreshments
18:00	Close

Our next workshop is on March 29^th 2022! 

We will be back at The Francis Crick Institute in central London. As with previous events we are looking forward to a varied programme from the UK research community. We expect the workshop to be a mix of technical talks with researchers reporting on their use of cloud technologies. 

Abstract Submission

We’re looking for talk submissions covering all aspects of research computing using cloud, both public and private. We wish to have a diverse set of viewpoints represented at this workshop and encourage individuals and institutions of all backgrounds (for example academic, technical, business, or user experience) to apply. 

We’ve provided some suggested topics and themes below, but submissions outside of these areas are also welcome. Talk sessions are typically 20 minutes and should include time for questions. We’d also like to record and share the videos and slides afterwards, so please make it clear in your submission if this is likely to be an issue. To submit an abstract complete this form:

https://forms.gle/u7XEPpB79o2Y4rM27

The deadline for submissions has been extended to Monday 24^th January (EOB). The Cloud Working Group will review the submissions and we’ll let successful submitters know by 31^st January.

Workshop Presentation Format

Due to the ongoing situation, we may need to limit attendance. As such, this year we will be providing an in-person and live streaming experience and we can now accommodate virtual speakers for this event.  

Please note that all plans being made are subject to change based on the current health and safety guidance set by the UK Government.

Proposed Workshop Themes

High Performance Computing (HPC)

We would like to hear from those who have user stories involving running HPC-class workloads in public cloud. Stories can also include utilising cloud-native methods to create software-defined HPC infrastructure; hybrid solutions that extend on-premise compute infrastructure with cloud bursting, or adapting HPC workflows to exploit cloud-native technologies, for example.

Cloud Pilots and User Experiences

We would like to hear from operators and users about their experiences of running scientific workloads in private and public cloud environments. How does this compare with traditional HTC and HPC facilities? Have you found any advantages and/or disadvantages that we should know about? Do you use any abstraction layers to make them more usable?

Hybrid Cloud 

We are looking for examples of deployments which bridge the gap between on-premise infrastructure and public cloud, or between cloud providers. This could include; efforts to make workloads portable between clouds, creating cloud services or cloud access to enhance the current solution offered, or technologies to support migration and bursting, for example. In addition, topics covering data movement/migration and data collaboration/sharing have been of a particular interest to our community in the past and would fit within this theme.

UKRI and Trusted Environments

Following the recent UKRI and DARE UK call to inform design of cross-council digital research environments, we are keen to hear from projects that are using cloud to enable research and collaboration with sensitive data.  We are particularly keen to hear from groups that are awarded funding – to provide an early platform for the projects to share previous experience that helped them secure funding and hear what they aim to achieve, as well as how they plan to share their solutions with the wider community.

COVID and Cloud

The global COVID pandemic has presented unprecedented challenges in health and economics and research has been at the forefront of addressing these, from modelling transmission to simulating viral proteins and treatments. We are keen to showcase stories from the research community where cloud has enabled projects; to share practices for operating under demanding conditions and time constraints, but also celebrate the work that is helping to ease us out of the pandemic restrictions.

Performance/ResOps

The recent COP26 conference has fired the starting gun on reducing greenhouse gas emissions to Net Zero. It is no longer an option to simply write code/applications and workflows without ensuring these have been performance optimised within reason. Neither cloud resource/technology/providers, application providers or cloud users can leave it to each other to ensure that workflows have as small a carbon footprint as possible. We are keen to hear from the cloud communities about work that has increased/maintained performance while reducing energy use. In particular we would like to hear about the role of the ResOps professional in ensuring that workflows/applications interact with cloud/cloud technologies in a power efficient manner. We hope this will share best practice and perhaps lead to further workshops and work in this challenging area.

Proposed talks are not limited to these themes but can also be in other areas of interest. Past themes have included storage, governance, IOT/data analytics and challenges faced and overcome when implementing a cloud solution at both business and technical levels.

The Program Committee listed here will make the final decision on the inclusion of any presentations to the meeting.

It’s a couple of months since the workshop and plenty of time to let the dust settle and reflect on the content. You can find most of the presentations from the workshop if you look follow the links from programme.

As I mentioned in my introduction at the meeting, I’ve noticed a transition over the past year in the adoption and application of cloud and this evident in the abstracts submitted for this meeting.  There are signs of a maturing – in the first couple of annual workshops we held, cloud usage was very much at the experimental stage with early forays into private cloud deployment and first pilots testing out public capability.   This year there were good examples of sophisticated application of cloud technology whether cloud-native applications like Chris Woods’ – use of serverless to dynamically trigger provision of clusters for batch computing – or in-depth demos of DevOps tooling from StackHPC and others.  

Late last year, the Cloud WG ran a smaller technical meeting with no formal agenda – in ‘unconference’ style.  This gave us an opportunity to do more of a deep dive with DevOps technologies.  The positive feedback we received reflected the value in networking and learning together with peers.  There was something of this continued at this year’s workshop with the afternoon demo session.  It was great to have this in-depth technical input alongside higher level presentations, whether overviews of projects or talks around challenge areas such as policy.  João Fernandes shared about the OCRE project in his presentation.  This builds on the work of the GÉANT IaaS Framework, important for the establishment of agreement with public cloud providers for access to their resources for the research community.  
On the topic of policy, the debate continued around the relative merits of public cloud versus on-premise hosting.   Cliff Addison (University of Liverpool) highlighted the tensions between quantifying benefits, budgeting at scale and maintaining portability between cloud vendors.   Owen Thomas (Red Oak Consulting) challenged assumptions with traditional HPC provision and made the case for assessment of overall value not just cost when making comparisons with public cloud.  Andrew Jones (NAG) argued against absolutes when considering the complexities in making choices for hosting for any given application.   Migration to cloud can present enormous challenges as Tony Wildish’s presentation illustrated.  He provided a walkthrough of different approaches for migration legacy code developed for on-premise to operate efficiently on cloud drawn from EMBL-EBI’s experiences.  Elsewhere in the meeting HEPCloud and UKAEA presentations show how hybrid models can be built up to select the required computing resources from on-premise and public cloud resources.  HEPCloud in particular, illustrating the benefit of public cloud to overspill from research infrastructure in order to meet peaks in demand.  

CRC Canada is an example of a complete public cloud solution architected from the ground up.   What is interesting here is the organisational and culture shifts needed to support that model.  In particular, the set up of dedicated effort for auditing and accounting when moving to a consumption based approach to billing.  Pangeo – presented by the Met Office Informatics Lab – demonstrates another cloud enabled solution but what is of interest is the formation of a collaboration bringing together open source solutions to make a platform that is cloud-ready.  At its core is a virtual research environment built largely on Jupyter and Dask together with use of Kubernetes and deployment glue to make it cloud-agnostic.  This kind of solution fits for data analytics where typically datasets have been imported into a cloud environment and manipulated into a form that is analysis-ready.  Use of BinderHub – shown with Pangeo and Sarah Gibson’s demo (Turing) – allows infrastructure to be dynamically provisioned and scientific workflows conveniently shared via Jupyter Notebooks.    

In general though, examples of long-term hosting of large volumes of research data on public cloud however are still absent.  If there’s a pattern from the sample of submissions for the workshop, it’s one of use of public cloud for compute rather than data storage: continued use of on-premise for long-term hosting of data with some bursting to public cloud for batching computing.  Cloud is utilised as a means to obtain an ephemeral computational resource: set-up an environment, stage data, perform calculation, get results and tear down again.  Even so, there appeared to be an increased awareness of the challenges of data hosting with cloud in some of the questions and discussion in the sessions.  These included issues around hybrid and public cloud and multi-cloud scenarios.  For example, if data is hosted in one cloud, how can it be accessed readily by a client with an existing presence hosted on another cloud?   There are definite signs of progress in the community but clearly there are still big challenges for cloud to be more fully utilised for research workloads.

The programme is available here:

https://cloud.ac.uk/workshops/feb2019/

You can participate through Twitter #cloudwg and crowdsourced write-up via Etherpad

We will be holding our 4th annual workshop early next year on the 12th February 2019.  We’re pleased to be back at our familiar venue the Francis Crick Institute in central London.   Please save the date!

In past years we’ve had a great set of speakers from public cloud companies and major research institutes to individual researchers reporting on how they are exploiting cloud computing to meet their research goals.  More details to follow soon.

Recently the European Bioinformatics Institute (EMBL-EBI) (an outstation of the European Molecular Biology Laboratory) based at the Wellcome Genome Campus, has been running a course on what they are calling ‘ResOps’. By this they mean essentially the adoption of DevOps-style practices and methods to research computing, notably pushing the latter in more of an infrastructure-as-code, cloud-native direction, thereby hoping to improve reproducibility and scalability.

The course was delivered by Dario Vianello and Erik van den Bergh, from the Technology and Science Integration team at the EBI. All materials from the course including presentations and code were available online, at https://bit.ly/resops_june. The team had also provided temporary accounts for course delegates on their Embassy Cloud, on which we could run practical exercises.

Dario contrasted two possible approaches to moving a scientific pipeline to the cloud – mimicking on-site infrastructure versus a cloud-native approach. The former is clearly easier, but does mean that you will miss out on many of the benefits of running on a cloud. The latter requires a lot more effort initially and may not be feasible, depending on things like application support for object storage, but it does offer great potential advantages, not least of all being lower costs.

As well as taking us through the process of porting a pipeline to a cloud, there were three lightning talks showing concrete examples of existing and ongoing ResOps work. These comprised Marine Metagenomics, RenSeq and the Phenomenal project. One common theme here is that the porting work provided a very valuable opportunity to profile and optimise the existing pipeline, deriving detailed performance characteristics that in some cases weren’t available before. In the Metagenomics case, it was necessary to remove site-specific code and assumptions that made sense when running on local compute resources. It is also important to work on the scalability of research codes, and to test scaling up on clouds, because in some cases it may be that beyond a certain point adding more cloud instances will not increase performance. If working on a public cloud, this would mean paying more for no benefit. The PhenoMeNal project was particularly impressive in its adoption of modern technologies for the virtual research environment portal, while the RenSeq work comprised an interesting example of containerising a workload, and hosting the containers within VMs. They found that it was useful to include a specific reference genome in the Docker image, even though that meant a very large 9GB image size.

In the afternoon we worked on our own test application, using Terraform to create the cloud resources, and Ansible for configuration, all pulling from publicly available GitHub repositories.

I was very impressed with the thoroughness of the course and its realism. While recognising the advantages of cloud-native computing, they were realistic about the challenges of moving scientific pipelines in that direction, and provided valuable real-world experiences that will help making decisions about how and where to host those pipelines. The fact that the materials are completely open made it even more valuable. I thoroughly recommend taking the course to anyone interested in moving science to the cloud.

 

It’s been a few months since our November workshop so there’s been some time to digest and reflect on some of the common themes emerging. Having attended a couple of other conferences and workshops from my community (AGU and AMS) it’s interesting to compare.

Firstly, it was great to see such a variety of application areas represented. For this our second annual workshop, we opened it for the submission of abstracts and this made such a difference.  There was a great response, Life sciences having the margin on other domain areas.  We had 160 register and 120 on the day.   It was fantastic to have the Crick as a venue.  It worked really well.

The first session looked at applications of hybrid and public cloud.   Two really interesting use cases (Edinburgh and NCAS, NERC) looked at trying out HPC workloads on public cloud.  This raised issues around comparative performance and costs between public cloud and on-prem HPC facilities.

On AWS, Placement Groups allow instances to put close to one another to improve inter-node communication for MPI-based workloads.   This showed comparable performance with Archer (UK national supercomputer) for smaller workloads but clearly there was some limit as this tailed off as the number of nodes increased whereas Archer performance continued linearly with increase in scale. This tallies with what I’ve seen anecdotally at the AMS conference where there seemed to be on the one hand increasing uptake of public cloud for use with Numerical Weather Prediction jobs (which need MPI). However, this seemed to be being done for smaller scale workloads where they can stay within the envelope of the node affinity features available.

Another theme was portability – what kind of approaches can be used engineer workloads so that they can be easily moved between providers. Andrew Lahiff from STFC, presented a very different use case showing how container technologies can be used to run for Particle Physics, cases where there the focus is high-throughput rather than HPC requirements and so much more amenable to cloud.  This work has been done a part of a pilot for the Cloud Working Group to specifically investigate how containers and container orchestration technology can be used to provide an abstraction layer for cloud interoperability.  A really nice slide showed Kubernetes clusters running on Azure and Google cloud managed from the same command line console app.  Dario Vianello’s talk (EMBL-EBI) showed how an alternative approach using a combination of Ansible and Terraform can be used to deploy workloads across multiple clouds.

It was great to have talks from hyper-scale cloud providers AWS, Azure and Google.  The scale in hyper-scale is as ever impressive as is the pace of change in technology:  very interesting to see Deep Learning applications driving the development of custom hardware – TPUs and FPGAs. Plans underway to host data centres in the UK will ease uptake. OpenStack Foundation and Andy McNab‘s talks showed examples of federation across OpenStack clouds.

In the private cloud session, Stig Tefler gave a nice illustration of network performance for VMs showing how a number of aspects of virtualisation can be changed or removed to progressively improve network performance towards line rate. Alongside talks on private cloud, the parallel session looked at Legal, Policy and Regulatory Issues a critical issue for adoption of public cloud.  Steven Newhouse gave some practical insights from a recent cloud tender for EMBL. There is clearly need for further work around these issues so that the community can better informed about choices. This is something that the working group will be taking forward.

For this workshop, we experimented with an interactive session – bringing together a group of around 20 delegates to work together on some technical themes agreed ahead of time including bulk data movement and cloud-hosting of Jupyter notebooks. There was plenty of useful interaction and discussion but we will need to look at the networking provision for the next time to ensure groups can get on with technical work on the day.

We discussed next steps in the final session. There is clearly interest in taking particular areas forward from the meeting: focus groups on technical areas like HTC and use of parallel file systems with cloud or organised around specific domains within the research community. Training figured also, in the form of a cloud carpentry course so that researchers can more readily get up and running using cloud. Looking forward, in each of these case we’re looking for discrete activities with an agreed set of goals and something to deliver at the end. Where possible we’re seeking to support relevant work that is already underway and initiate new work where there are perceived gaps. We will be looking at running smaller workshops targeted at specific themes in the coming months as a means to engage and disseminate some of this work.

Phil Kershaw, STFC & Cloud-WG chair