Mark Collier talks about the future of OpenStack
At the German OpenStack Days – DOST – The Cloud Report had the opportunity to talk with Mark Collier, COO of the OpenStack Foundation and one of the founders of OpenStack.
Hi Mark, nice to meet you! Please, tell us about you and the OpenStack Foundation.
Hi, I’m Mark Collier with OpenStack Foundation. Originally, I was one of the founders of Open- Stack, the project before we had the foundation. We started this project when I was working for a company called Rackspace, which is a large hosting cloud provider in the US and UK. As a cloud provider we were trying to figure out how to build a competitive cloud and how to go up against very large tech companies. I knew that we were going to need a lot of engineering, a lot of help, and a lot of developers. And we were trying to hire a lot of developers at the time. We already had a cloud. Rackspace had one of the first clouds along with Amazon, it was actually before Microsoft or Google entered the business. But we knew that in order to get access to technology we needed to have partners and really build an ecosystem around it. So, the decision was to embrace open source fully and that’s what became OpenStack. We launched that nine years ago and 2010 we partnered with NASA, the space agency, and 25 other companies. And the idea was that for infrastructure as a service we wanted to build a large community with many companies and many individuals contributing and about two years after we started the project, we created the foundation, so, that became the OpenStack Foundation. And the reason for that was really just to continue the growth, the project needed an independent home, nonprofit, not tide to one commercial entity. So, we could be a neutral third party and really help to facilitate the communities ́ growth and bring in much bigger players like IBM, Cisco and Red Hat that we wanted to see in the foundation in order to make sure that OpenStack had a long-term future. That kind of what brought us to the foundation, and I am the COO of the foundation and since then it has really been a wild ride of trying to just manage the growth.
We have tremendous number of users in the technology landscape, it changes very quickly. We had a lot of companies come into the market to build products and a lot of developers come in to contribute to the code. In the most recent OpenStack release, Train, that was made available in October of 2019, 25,500 changes were committed by more than 1,100 developers from over 50 countries and 165 organizations. This pace of development makes OpenStack one of the three most active open source projects in the world. A lot of people don’t know that because we actually don’t use GitHub which is where a lot of the stats people are looking for. We actually outgrew GitHub pretty early on, so we use Git technology which is an open source tool for code collaboration and code repositories, so we host our own Git. We use Git but we don’t use GitHub, the commercial product. And because we use our own Git a lot of data doesn’t show up when people just do a quick study. In recent years there has been this miss that “everything happening in open source is happening in GitHub”. Linux Kernel, Chromium and OpenStack, the three most active open source projects in the world and none of them use GitHub because… GitHub is great, there is nothing wrong with GitHub, but it has come along more recently, and it got a different collaboration model. So, we use a combination of our own Git, we use a tool called Gerrit for code review which is an open source project that google released several years ago, so we are active in that and we use a tool called Zuul for our code testing and that was developed within the OpenStack community. So, between Git, Gerrit and Zuul we manage our collaboration and flow of code. We have hundred thousand members now in 180 different countries, so, managing that size com- munity with all the developer activity we needed a lot of tools for ourselves. But now we are starting to see other companies adapt those tools who want to write open source software in a similar fashion to how OpenStack was developed and some of those projects are actually now part of the OpenStack Foundation. That is one of the updates that we wanted to share while we are here, the OpenStack Foundation itself have now evolved to help host other open source projects.
The idea of OpenStack was that for infrastructure as a service we wanted to build a large community with many companies and many individuals contributing.
Kubernetes is more than a trend. The CNCF builds the state-of-the-art ecosystem for containerized workloads. How does this effect the present and future of OpenStack?
OpenStack is still the third biggest open source project in the world, but there are now four other open source projects that are part of the OpenStack Foundation and I can give you a quick rundown of them: there is Airship StarlingX, Zuul and Kata Containers. They all have some container related tiles, because containers are very useful. Since containers and docker really came along that had a very positive influence on OpenStack, so, we have used containers, they have been really useful making OpenStack more reliable because we can containerize the control plane and we can quickly spin up and manage and upgrade OpenStack itself, because the OpenStack servers are in containers. OpenStack is in some ways an application, a Linux application. It is an application that provides infrastructure as a service. So, we’ve used containers for several years to make the control plane easier to upgrade, the same reason any application developer love container. So, we’ve used it in thatway. Kubernetes and OpenStack have proven to be an extremely powerful combination. More than half of our users in our user survey tell us they are using Kubernetes on top of OpenStack. And so, I think a lot of people have a misunderstanding thinking somehow, they are competitive, or one replaces the other. In reality, Kubernetes runs on top of programmatic infrastructure whether that is AWS, Azure, Google, or an OpenStack Cloud. In addition to thousands of private clouds, Open- Stack powers dozens of public clouds around the world. This includes a large footprint in Europe with providers like City Network, OVH, and Open Telekom Cloud. Recently a new project in Europe was announced, Gaia-X, to build even more clouds for Europe driven by data sovereignty requirements which are especially strong in Germany. This will be a huge opportunity for OpenStack public clouds who are already supporting this requirement throughout Europe. It is really playing that infrastructure as a service layer role to make the bare metal infrastructure programmatic. With the last several releases of OpenStack bare metal provisioning becoming a key part of that, so you can even if you want to run Kubernetes on bare metal without virtual machines which isn’t usually a good idea for most use cases in sense of security, but you can if you have really tightly secure environment where you control or you are running only app and is not multitenant… You can do that, and you are still running OpenStack, so, sometimes people conflate OpenStack with virtual machines which is not really accurate.
So, OpenStack starts of bare metal, has an occasion with keystones and has whole series of components that make up how you control access to resources, that are programmatic, and then virtual machines become kind of an optional piece of it. Still, most people who run it, run virtual machines with OpenStack, but you can run your containerized Kubernetes cluster in virtual machines which is really how most people do containers in production the run in VMs because of the security or if you really feel bold you can run it on bare metal and you can still use OpenStack for the bare metal piece. So, containers have been a really good thing for OpenStack and I think there are a lot of people that think it is the opposite for some reason but it is just the nature of when something new comes along people think that whatever was there at the time must be in jeopardy but, the reality is that all these things
build on each other. Linux didn’t go away when OpenStack showed up, OpenStack builds on Linux. This is something that we try to clear up. We have seen some of our largest users are running Kubernetes with OpenStack. With the new projects as the Foundation has evolved and we know how we are now hosting and helping communities around additional projects outside of OpenStack, they all have some sort of container tile. So, Kata Containers specifically was the first pilot project we did that wasn’t part of OpenStack, but it was part of the OpenStack Foundation. It is a little bit confusing since they both have the word OpenStack in them. We are not trying to add hundreds of projects, we ́ve only added four, so, we are taking it slow and deliberated and only hosting new projects, which really makes sense for our community.
OpenStack is one of the most complex open source software solutions at the moment. The so called „core projects“ are stable and enterprise ready enough to build a robust infrastructure as a service platform. But the Foundation has so many small additional projects for OpenStack like Sahara to be an anything as a service platform. How do you see the future of these projects? Is Open- Stack a good choice for anything as a service or should it better be used for infrastructure as a service?
I think that the primary focus is going to be infrastructure as a service, and that one of the things that happened somewhere in the history of Open Stack is that there was a period of time where we made the mistake of thinking that if it needs to go on a cloud it should be open source. That was not a mistake, but the second piece was therefore should be OpenStack. That was the mistake. What we realized was we want everything you need to build a modern cloud to be open source, but it doesn’t mean it all needs to go in the OpenStack and that was kind of a rev- elation two or three years ago. Like: wait a minute, we should just focus OpenStack on what it’s pieces. We should work with the other communities, we should help establish new communities that are building and solving problems, that work with OpenStack. But it doesn’t need to go into OpenStack. So, if you see in our Foundation again like KataContainer, Airship, StarlingX, they are all things that our community is helping with but we are not putting them into OpenStack, they are their owncommunities in a sense and they are building their own deliverables and re- leases on their owns schedules and have their own governances. Trying to shove all that into OpenStack was probably the wrong way to go. Now we ́ve taken more realistic views. If you look in any
OpenStack cloud, it is never just OpenStack and it never really was. There is Linux and KVM as a Hypervisor, you have RabbitMQ and things written in Python, so there has always been a combination of different open source things in practice when you actually take it in production. But we are just trying to take that bigger view and go. There is so much open source now, it is a wonderful thing, but it’s also creating a new set of problems for our users. So, what can we as a Foundation do to help? A lot of that involves testings, it involves working at cross communities and sort of saying not everything should go into OpenStack, not everything should go into Kubernetes, not everything should go into Linux and if you look into AI machine learning you have things like TensorFlow which isn’t at the Foundation and that is totally fine. The Foundation side is sort of irrelevant, what really matters is how can we make all of this stuff work together? So, we are doing much more with testing, we are doing tests today within the Kubernetes com- munity, every new Kubernetes release will not come out if it breaks on top of OpenStack, because OpenStack is part of their testing, just like they are testing against Amazon, Google, Microsoft. In the same way a new release of OpenStack won’t come out if it breaks on Kubernetes. So, where these pieces need to fit together, we are trying to serve the market better by collaborating, cooperating and testing, testing, testing, and then when there are gaps, we will create new projects. But our focus is on infrastructure.
There are projects that sort of cross the boundaries but for most part they are most like installers so like Magnum is a Kubernetes project within Open Stack, the important thing it does is that it installs Kubernetes. It doesn’t replace it, it installs it in an automated way. In France CERN has a 300,000-core OpenStack Cloud and they run hundreds of clusters of Kubernetes on top and they use Magnum to do that. So, there are integration projects, but they are not trying to replace other functions outside of an infrastructure service.
Kubernetes and OpenStack have proven to be an extremely powerful combination.
Some voices in the industry say that the time of OpenStack is over, what is your statement about the general future of OpenStack as IaaS platform?
I think news like that definitely get clicks or people wouldn’t put it on their LinkedIn. They have some reasons doing that, but all we do is to talk to our users, our developers and our ecosystem every day and try to help them out. While we have been here at this event, we talked to BMW who’s growing their footprint this year by 30 or 40 %. BMW is a massive OpenStack user and they are growing, they actually can’t grow fast enough. They also use Zuul which is a CI/CD tool as a top-level project. We talk to customers or users like that and they are just growing their footprint and we have new users that come online and talk about how they are solving problems with it.
Honestly, I’ve been doing this for nine years and I don’t think there has been a single year where there wasn’t some headline like “Open Stack is dead”. And I say “well, I’ve heard this before” and then I go back to work hard to make it better.
I would say that one of the challenges is that the technology world is very obsessed with what’s new and hot and young and the new thing and what’s next. But if the world is running your soft- ware production you are not really next you are right now. Once you get adoption you are not in the future. It doesn’t mean you won ́t be there in the future, it just doesn’t fit the narrative of what a lot of people are getting excited about and want to read and tweet about, because they are just quietly solving problems. I mean Linux is now widely adopted then ever but probably getting less press than ever. But Linux is every- where. And OpenStack is similarly like that. The largest power grid in the world, state grid of China runs OpenStack. So, if you’re in China and you’re getting electricity it is coming through an OpenStack cloud. China railways, the largest rail system in the world, they run OpenStack. The billion tickets a year, they sell on their trains, run on an OpenStack platform. If you are in in the US and you have AT&T as your carrier of horizon your 4G and 5G calls are going through Open- Stack. If you look at ecommerce there are a lot of automobile companies like BMW. So, OpenStack has never been more pervasive, it more becomes of the fabric of the infrastructure you use every day. Less people may tweet about it, but our goal was always to build a useful piece of technology and get it widely adopted, so in that regard I would say we are very happy with where we are nine years into the project.
We see the same here. Three years ago, all the companies in Germany were really interested in OpenStack, like BMW and other big vendors, and then we had less requests last year, mostly the customer are talking about Kubernetes, Open Shift. And at the moment we see the second wave of this. We have so many requests for OpenStack and so many companies are building private clouds at the moment based on OpenStack.
Great! Tell them to call me! The funny thing about open source and why we love coming to these events is we always meet new users. Because there is nothing in the software that phones home and tells us, so often time I meet a user and I say “these are the reasons you should use OpenStack“ and they say “we have been running it for three years “. We literally don ́t know who runs it unless they come and reach out to us or we meet them at one of these events. You are giving me good news.
Some users tell us that “OpenStack is great to use, but it is hard to operate”. It is one of the most complex open source technologies at the moment. What do you recommend for these users? Use Vanilla OpenStack? Use vendor OpenStack like Redhat, Suse, Mirantis or Canonical?
Everything depends on the individual user, but I think that one of the things we were very deliberate about when we set up the OpenStack project and the Foundation was that we wanted there to be a strong ecosystem. Because that is part of what keeps it going a long term, as companies can help users adapt the software and those companies also employ most of the developers that write the software. So, it is a virtuous cycle. We’ve never been anti vendor by any means, we have actually been very much about being that one of thethree forces that makes a platform successful as a strong ecosystem. So, most users, I think, really should work with a vendor. It doesn’t mean you can’t do it yourself and be successful, there are many examples of people that have. But I usually tell people that it doesn’t hurt to start talking to the vendors. There are many options out there, there are a lot of major players. Every Linux distribution has an OpenStack product like Mirantis and other OpenStack distributions. A good place to start.
We should work with the other communities, we should help establish new communities that are building and solving problems, that work with OpenStack. But it doesn’t need to go into OpenStack.
Another thing that I would point to is that both Airship and StarlingX are new open source projects within the OpenStack Foundation, but they both essentially contain OpenStack. They both take the best of OpenStack and Kubernetes and bring them together for specific use cases. So, Airship use case is sort of unique in that they have hundreds of sites, they are running hundreds of OpenStack clouds and they want to run thousands, because it is pushing out to the edge of their network. They want to be able to do zero touch provisioning, upgrades in the field. These are challenges that we see in the operation of OpenStack. AT&T reached a scale where they just didn’t have a choice, just doing it in a hard way which is not working at all, they HAD to do it completely automated so that’s why they created Airship. As it turns out, not everyone wants to run a thousand clouds at the edge, Airship is actually applicable to people even though they only have one cloud. Because they create a repeatable upgradable lifecycle management system for the whole infrastructure from the bare metal up and it allows to mix and match Kubernetes and OpenStack depending on the work- load and connect it all together. So, even though there is telco and they did it for 5G is not really specific to that, it is really just about automated deployment, upgrades infrastructure including OpenStack and Kubernetes. Airship is a much newer project, so people have to evaluate it of course. It is not nine years old like OpenStack, but it is relying on a lot of proven technologies. We just talked to a company last week. One of the largest coffee chains in the world is looking at Airship as a way to get OpenStack into the edge of all their coffee shops. So Airship is really interesting.
StarlingX is also combining OpenStack and Kubernetes. It is very much designed for both edge and industrial IOT use cases. So, those two projects are somehow similar, but they are both kind of run independently, there are common components between them. StarlingX is also relatively new, but it has been originated by Wind River and Intel and a few other companies, and that was originally a commercial product that Wind River built based on OpenStack. But then they decided to open source it and came to OpenStack Foundation “We have this commercial product, we want to build a community around it, we want it to be open source. It is based on OpenStack, so bring it into your Foundation makes a lot of sense.” So, it is kind of OpenStack and a bunch of other stuff that optimize it for small use cases. Two servers type deployment where you have it at edge. Because it started like a commercial product it is already running in production with some big industrial players for example: China Union Pay that crazy big China financial services company. They do five hundred million dollars a year in transactions and it is fifty thousand transactions a day. So, they are in mobile payment in China and dominant everywhere you go in China as payment form. For five years they have been running OpenStack, but now they are interested in trying out StarlingX, because they have been using OpenStack more for the data center side, taking all the data coming in from all those transactions and crunching the numbers and storing it and everything. But what they want to do is actually push OpenStack all the way to the edge into the corner of sale where cards are being swiped. This is a radical use case for putting something out really far to the edge, so they are looking at Starling X for that, because it allows you to slim down to really just some sub components of OpenStack and/ or Kubernetes in different combinations to meet the edge user case. These new infrastructure use cases are one of the main reasons we have new project at the Foundation, because OpenStack alone can’t do it all, but as a Foundation we want to help all these people adopt open source. Our mission is to help people build and operate open infrastructure, wherever infrastructure is going and growing we want to be there, we want open source to be a viable alternative to proprietary solutions and often times that means collaboration, writing software and testing. So, that is kind of where we are going as a Foundation with our community. OpenStack is at the heart of that, but there are other cool stuff going on making sure open source is viable in all these new environments.
I’ve been doing this for nine years and I don’t think there has been a single year where there wasn’t some headline like “Open Stack is dead”. And I say “well, I’ve heard this before” and then I go back to work hard to make it better.
The foundation has cancelled the COA exam in March this year, is there a new exam planned?
When we talked about winding down the COA, a lot of people came out of nowhere to say “We love it! We need it and we want it to continue.” The outcry for support and demand for it to continue took us pleasantly by surprise. So, we said, “let’s find another option” and we did. So, Mirantis approached us about helping to continue the administration of it and keep it going and update it and upgrade it for the new versions. Due to the heightened community demand, the OpenStack Foundation partnered with Mirantis to continue delivering the COA exam starting in late 2019 (www.openstack.org/coa).
Do you want to add something?
For us it is fascinating to hear the feedback and to hear what the perceptions are out there. We have a constant struggle for setting the record straight as far as “Is OpenStack dying?” It has been going on for nine years, it is a bit repetitive but we want to know where people are confused and where we can do a better job of educating the market, because for the problem that is solves there is nothing else out there that is proven at scale. We have ten million cores of compute managed by OpenStack today. And it is probably much higher than that. That is just the people who tells us they are running it and we add up the numbers so it could be many times higher. But we have a user survey we put out every year and we are going to publish a re- port around that later this year. We just closed it for 2019. The twentieth release of OpenStack got Train that is on October 16. Those are some of the upcoming things happening. Our next Summit will be in Shanghai, first time in mainland China. That should be exciting. It is an exciting time to do business in China.
It is a crazy world, but open source can bring us together. And one of the things that I love about it is that we have members from 187 countries that are part of our Foundation which is almost every country in the world. So, we try to work without regards for borders, corporate borders or national borders. Just with the OpenStack and not getting caught up in the political drama. It is a good thing we can work on common goals and not having to worry about our politicians.
These are very nice words at the end. Thank you for your time!
Michael Schmidt from SAP talks about the newest inventions of OpenStack at the DOST 2019 in Berlin.