Sep 18
0

OpenSDS India Meetups 2019 #03 Cloud Native & Data Management Use Cases!

By | Event Reports, Events, Past Events | No Comments

A Combined meetup with CNCF, Cloud Native Storage and OpenSDS
07th September, 2019, Saturday, Microsoft Office, Belandur, Bangalore

OpenSDS (Open Data Autonomy) meetup group along with CNCF Bangalore, Cloud Native Storage Bangalore organized a meetup on “Cloud Native & Data Management Usecases” on 07th September at Microsoft Office in Bangalore, India.

The major focus of this meetup was on the various usecases and challenges in Cloud Native and Data Management areas. Speakers from Microsoft, MayaData, Huawei/OpenSDS, Cloudera, CGI and Amrita University presented. Live demos on different features and usecases were also shown. The interactive session saw ~110 members participated from different organizations and academia.

(Members from organizations like Microsoft, MayaData, Huawei, CGI, HPE, Mphasis, Intuit, GE, VMWare, Capegemini, Accenture, IBM, Cisco, Walmart, Intel, Lenovo, TCS, Cloudera, Wipro, Maple Labs, Different universities and more...)

Most of the members gave a feedback that, such events to be more frequent and can consider more technical topics. They felt, it is really useful to learn and contribute to such open source communities and projects.

Many of the members attending this meetup were particularly interested in seeking the solutions for storage, data management, and cloud native challenges.

Along with speaking sessions, there were 3 lighting talks from CGI, Cloudera and students of Amrita University.

There was a networking session during the lunch providing ample time for discussing and connecting.

Link for meetup details:

https://www.meetup.com/OpenDataAutonomy/events/264395906/

Event Details:

1. `Scaling kubernetes app using Virtual Kubelets ` by Vivek Sridhar, Technology Evangelist Microsoft

2. `OpenEBS for scalable logging on Kubernetes” by Uma Mukkara, COO MayaData.

3. ‘OpenSDS with VMWare platform for Data Management’ by Najmudheen, Himanshu , OpenSDS/Huawei. [Introduced SODA Foundation during this talk by Sanil, OpenSDS]

4. Deep Learning Model in ACI+AKS by Meena, Technology Evangelist, Architect

Lighting Talks –

  1. Ajay, DevOps Architect. CGI (Azure DevOps pipeline)

  2. Sunil, Architect, Cloudera (YuniKorn – a special kubernetes scheduler)

  3. Koushik, Student, Amirta University (Deep Learning Usecase using Drones)

Jul 11
0

The Linux Foundation Open Source Summit – Japan 2019

By | Events, Past Events


Toranomon Hills Forum

July 17-19, 2019, Tokyo, Japan.

Menu

The technical conference for professional open source.
Open Source Summit Japan is the leading conference in Japan connecting the open source ecosystem under one roof, providing a forum for technologists and open source industry leaders to collaborate and share information, learn about the latest in open source technologies and find out how to gain a competitive advantage by using innovative open solutions.

To register for the event please click on the link below.
Register Now!

Schedule

Collaborate | Engage | Contribute

Venue: Toranomon Hills Forum Tokyo, Japan

Wednesday, July 17

13:30 – 14:10

Building Telemetry and Anomaly Detection Models for Cloud Native Storage

Xing Yang, Futurewei & Seiya Takei, Yahoo! JAPAN Corporation

Friday, July 19

11:50 – 12:30

Panel Discussion
Cloud Native Data and Storage Management Challenges from End Users Perspectives

Moderated by Steven Tan, Huawei; Yuji Yazawa, Toyota; Shinya Tsunematsu, GMO Pepabo; Mitchitaka Terada, IIJ and Tomoko Kondo, Softbank

Friday, July 19

16:50 – 17:30

Building an Open Autonomous Data Platform

Xing Yang, Futurewei

Speakers

Our expert panel of speakers come from various organisations.

Xing Yang

Lead Architect of OpenSDS, Futurewei

Steven Tan

OpenSDS TSC Chair, VP & CTO Cloud Solutions, Huawei

Yuji Yazawa

Toyota, Japan

Tomoko Kondo

Softbank

Shinya Tsunematsu

GMO Pepabo, Japan

Mitchitaka Terada

IIJ

Seiya Takei

Yahoo! JAPAN

Venue

Location, Directions and Timings

Where

Toranomon Hills Forum 

1-chōme-23 Toranomon, 港区 Minato-ku,
Tōkyō-to 105-0001, Japan

When

Date: 17th – 19th July, 2019

May 20
0

OpenSDS at KubeCon EU 2019, Barcelona, Spain

By | Event Reports, Events, Past Events | No Comments

Introduction

All technology trends are data-centric and data intrinsic today. So as the challenges in managing the data across heterogeneous storage vendors, different platforms and across multiple clouds! OpenSDS, the Linux Foundation project is working to build an open data autonomous platform to address those challenges (www.opensds.io). It is an ecosystem with vendors, end-users, solution providers, technologists and developers across the globe. This Linux Foundation project has members from IBM, Fujitsu, Huawei, Intel, Western Digital, Dell-EMC, Hitachi, LINBIT, Vodafone, KPN, NTT Communications, China Unicom, Yahoo! JAPAN, Toyota Infotech, GMO, IIJ, Click2Cloud, Oregon State University and more.

KubeCon+Cloud Native Con is organized by Linux Foundation and Cloud Native Computing Foundation(CNCF) and it’s a flagship conference in open source and cloud, which gathers adopters and technologists from leading open source and cloud native communities from all over the world. It is one of the biggest of its kind in the world.

KubeCon Europe 2019 was from May 20-23 in Barcelona, Spain.

Summary

The KubeCon Europe 2019 witnessed around 8000 attendees from all over the world. The sponsor showcases were more than 150 from various organizations and startups. It includes major companies like RedHat, VMWare, IBM Cloud, Cisco, Oracle, Microsoft Azure, Google Cloud, AWS, Canonical, Huawei Cloud, SUSE, Intel, SAP, HP, Yahoo!JAPAN and many more.

OpenSDS Project sponsored this event as a silver sponsor and also organized a mini summit “Open Data Autonomy” as a co-located event. OpenSDS had a showcase booth where it provided the key features and roadmap of the project.

Multiple speakers from different companies like Intel, Google, NTT, Huawei, Yahoo!JAPAN, MayaData/OpenEBS, SNIA/NetApp, SPC/HP, Quobyte, LINBIT presented various aspects of data management. Dan Kohn, Executive Director of CNCF presented about overall cloud native ecosystem and the potential of OpenSDS. The one day summit could provide various challenges, opportunities & use cases of data management and value of OpenSDS in solving the challenges. It also covered the plan and roadmap of OpenSDS project.

OpenSDS Showcase from May 21-23 was visited by more than 400 people and got more than 275 unique leads. Multiple solution providers, end users, vendors and other open source projects showed interest in the project.

Open Data Autonomy, mini summit by OpenSDS

It was a full day mini summit discussing on data management challenges, use cases and various opportunities. Presented the key features and roadmap of OpenSDS Project.

15 different sessions and one dedicated session for demo workshop conducted. Multiple speakers from different companies like Intel, Google, NTT, Huawei, Yahoo!JAPAN, MayaData/OpenEBS, SNIA/NetApp, SPC/HP, Quobyte, LINBIT presented various aspects of data management. Dan Kohn, Executive Director of CNCF presented about overall cloud native ecosystem and the potential of OpenSDS

Also we provided a lighting talk on OpenSDS in Cloud Native Storage Summit, providing the project brief.

They key points:

  • Key features and roadmap of the project

  • Rebranding announcement of the project (Open Data Autonomy direction) and support needed from partners

  • Collaboration and integration opportunities with OpenEBS, LINBIT, Yahoo!JAPAN and NTT, SPC, SNIA are discussed

  • Partners like NTT, Yahoo!JAPAN, OpenEBS and others showed interest to do testing of Capri release

OpenSDS Showcasing Booth


OpenSDS showcasing booth from May 21-23 provided the details on the project features, roadmap and the value. It was visited around 400 visitors and got 277 unique leads with 28 hot and 40 warm leads.

Apr 15
0

The 2nd OpenSDS Japan Meetup

By | Event Reports, Past Events | No Comments

The Yahoo! JAPAN Lodge is an incubator space with a vibe that is perfect for the 2nd OpenSDS Japan Meetup.

It was great to see members of the Japan Community Committee moving tables and chairs around to set up the meetup space, and getting ready to welcome attendees before to the meetup.

Yazawa-san, EUAC representative from Toyota, and Chairman of the OpenSDS Japan Community opened the meetup giving it a touch of Japanese formality and dignity.

Next was my State of the Project talk, and I was glad to have some interesting updates including Intel, China Unicom and Sakura Internet formally joining the project.

Sakura Internet came to know about OpenSDS from Kusunoki-san or Kei from NTT Communications, and representative on the EUAC. This is a testament to the open source collaboration opportunities in OpenSDS between two potentially competing businesses.

The next topic I covered was The Open Autonomous Data Platform the community is developing, and Capri — the name of the next release targeted for June. I like to tell people the releases are named after exotic resort islands and that we will plan our future events in these places — then I wait to see their reactions, which is always a mix of ‘you must be kidding’ and ‘yeah, right’.

Capri has many features requested by the EUAC and also features towards autonomous platform. File support will finally satisfy the top request from Yahoo! JAPAN, Automation & Orchestration will please KPN, Object Lifecycle will delight Toyota, and expanded multi-cloud support for GCP and IBM gives end users more choices. Intel’s contribution to Capri is NVMe over Fabric which many end users are looking forward to. Telemetry & Anomaly Detection is the community’s first foray into telemetry data collection and AI/ML. More details and status about Capri can be found on the OpenSDS github.

Yahoo! JAPAN EUAC representative Sato-san together with Takei-san presented and demonstrated the Bali release which featured multi-cloud support for AWS S3, Azure Blob, Huawei Object, and Ceph S3. Yahoo! JAPAN has been supporting OpenSDS in many ways including testing, debugging, demonstrating and sharing about the project. One of their primary use case is storage consolidation for OpenStack and Kubernetes clusters.

Next up, Yazawa-san gave a summary of the Japan community expectations and requests for the project. The Japan community has been running its own bi-weekly community meetings since the beginning of the year, and the summary highlighted the discussions.

The last session was a panel discussion moderated by Kei of NTT Communications, with speakers Yazawa-san of Toyota, Terada-san of IIJ, Tsunematsu-san of GMO Pepabo, Hamono-san of Fujitsu and Xuchuanru of Huawei Japan. The entire session was in Japanese, so let’s just say it was quite an interesting exchange between the panelists comprising both end users and vendors.

We wrapped up the evening with what I assumed was some snacks and drinks — instead we had a feast served up by the Yahoo! JAPAN kitchen. Larry who is in charge of OpenSDS Business Development, reluctantly swiped his card and paid for it. I thought, if the presentations failed to meet expectations, at least this feast should make up enough for it. Yazawa-san gave a toast with ‘kampai’ and we got the party started. In the midst of it, some folks came up to thank me for sharing the project, for the networking opportunity, and for getting the Japan community together. I thanked them back for giving support to the project.

It was nice to see most folks staying until right at the end and when we finally had to vacate the space, they automatically (autonomously?) helped to clean up and restore it.

The OpenSDS Japan Community Committee did a marvelous job organizing the meetup. We expected about 50, ended the evening with about 80, and a community that is asking when is the next meetup. I think that’s a good sign we didn’t screw up the meetup. Thank you OpenSDS Japan Community Committee for such a successful meetup and Yahoo! JAPAN for being such a great host.

No one got drunk — did we not have enough beers?

About OpenSDS:

The Linux Foundation OpenSDS Project is an open source community working to address storage and data management challenges in the cloud native era. The community is developing the Open Autonomous Data Platform for Kubernetes and OpenStack to manage, protect, secure and optimize data on-premise, and across clouds.

For more on the project and how to participate: www.opensds.io

Jan 28
0

OpenSDS was the silver sponsor at Kubernetes Day India 2019 held at Bangalore.

By | Event Reports, Events, Past Events | No Comments

Summary

KubernetesDay India 2019 was a well organized CNCF event, first of its kind in India. Around 1000 people attended the event and major organizations joined to showcase their cloud and open source solutions. Local and international experts shared their experiences and engaged with attendees on Kubernetes and related cloud native technologies. OpenSDS joined the event as a silver sponsor and introduced the project to India community for the first time! It was well received.

Around 300 people visited OpenSDS booth and we got more than 100 people interested to know more about the project. Visitors shared their specific pain points and agreed that the features and scope of OpenSDS are in the right direction where it can help to resolve existing and future challenges in storage and data management. OpenSDS team had a constructive interaction with visitors and sponsoring organizations.

About Kubernetes Day India 2019

Kubernetes Day India was the first ever CNCF event in India and is being hosted at the Infosys premises in Bengaluru. Infosys was the venue sponsor for the event. Kubernetes Day- a one day, single track event, brings together local and international experts for developers of all levels interested in Kubernetes and related cloud-native technologies. With this event, the CNCF aims at appealing to a large number of developers who might not necessarily travel to KubeCon + CloudNativeCon events in Europe, China, and North America.
Mainly the event covered most of the basic topics around Kubernetes including the usage, contributions. Speakers from India and abroad featured in the event.

OpenSDS @ the event

OpenSDS joined the event as silver sponsor. OpenSDS booth provided all the basic introduction about the project along with live demo. The project vision, the major data management challenges, architecture and project collaboration opportunities are shared. Also Kubernetes integration and multi-cloud features (including data migration) were showcased at the booth. Many of the visitors from various companies looked very much interested in the overall goal of the project, and keen on the roadmap features. The storage and data management problems which opensds is trying solve look applicable for many.

Both individuals and organizations showed interests in the features. Some of them are keen to know more about the project and explore the possibilities of collaboration with this Linux Foundation project.

Around 300 people visited the booth and more than 100 people showed interests in knowing more about OpenSDS and collaborate. There were people working in solution, multi-cloud, storage and open source. Members from major organizations visited – Infosys, IBM, Microsoft, NetApp, Oracle, OpenEBS, AWS, ISG Research, portworx, infracloud, gopaddle, pivotal and more. Visitors shared their specific pain points and agreed that the features and scope of OpenSDS are in the right direction where it can help to resolve existing and future challenges in storage and data management. OpenSDS team had a constructive interaction with visitors and sponsoring organizations.

The OpenSDS team was represented by Steven Tan, TSC Chair & CTO Cloud Storage Solutions, Liaocheng, APAC IT Mktg, Wangxiaojun, India IT Mktg, Xing Yang, Tech Lead & Architect OpenSDS, Xulin, OpenSDS Maintainer, Sanil, Architect & Lead OpenSDS India and members of OpenSDS India team.

Feb 23
0

OpenSDS Launches ZEALAND Release, Providing Unified SDS Controller Framework and API

By | Announcement

OpenSDS Launches ZEALAND Release,

Providing Unified SDS Controller Framework and API

 

In collaboration with other global open-source communities, OpenSDS has launched the first release of its software codenamed “ZEALAND,” which provides users and developers with a service-oriented and software-defined storage (SDS) controller framework and standard APIs

SAN FRANCISCO, January 5, 2018 — The OpenSDS community has announced the release of the first OpenSDS software, codenamed “ZEALAND,” which provides a unified framework and APIs for end-to-end software-defined storage controller solutions. OpenSDS ZEALAND comprises two internal projects: Hotpot, the Controller Project, and Sushi, the Northbound Plugin Project. The Hotpot project provides users and developers with block storage service, which supports basic lifecycle management of volume/snapshot/attachment, discovery and registration of different types of storage systems, scheduling to storage backend by customer-defined profiles and more. The Sushi project provides the dynamic provisioner and FlexVolume plug-ins for Kubernetes.

“In the past, storage systems were difficult to manage, and users needed to rely on highly specialized storage administrators,” noted Steven Tan, chair of the OpenSDS TSC and Chief Architect at Huawei responsible for SDS planning and strategies. “This is finally changing, thanks to continuing efforts by storage vendors to make storage systems easier to use. However, the entire storage industry now faces new challenges brought on by cloud-enabled, service-driven development. To address these challenges, OpenSDS is improving the usability of existing storage by abstracting complex storage features so that they integrate seamlessly into cloud-native frameworks such as Kubernetes. This benefits our customers as advanced storage services can be easily enabled in a variety of cloud scenarios, regardless of whether they are using private, hybrid, or public clouds.”

What’s Next for OpenSDS

The OpenSDS community is preparing to start to work on its next version – OpenSDS ARUBA. The current release reflects the real-life requirements of the core members of the OpenSDS EUAC (End-User Advisory Committee). OpenSDS will also continue to work with other projects, such as OpenStack Cinder and Manila, as well as CNCF Container Storage Interface (CSI) to launch joint solutions. OpenSDS ARUBA is expected to be released in June, 2018. It provides a number of key storage service capabilities such as array and host-based replication, group snapshots and replication, and more plug-ins to support the northbound ecosystem.

“OpenSDS brings policy based, self-service storage provisioning and orchestration, which is ideal not only for cloud-native applications, but also fits well with traditional data centers and clouds,” said Rakesh Jain, an Architect and Researcher with IBM and vice-chair of the OpenSDS TSC. “We are working with various large users  to identify their pain points and addressing them in an open manner. The Zealand release is just a small step, we are well on our way to execute on the published roadmap to make storage as pain free as possible, in coordination with other open source projects.”

“The movement toward self-managed scalable infrastructure is firmly established in the industry and provides substantial improvement in operational efficiency for adopters. Achieving this vision requires revisiting nearly all aspects of data-center operations. The cross-vendor and cross-environment management capabilities provided by OpenSDS overcomes major barriers to adoption in existing data centers by allowing the unification of today’s virtual machine-oriented operating environments while anticipating the next generation of container-oriented operating environments” remarked Allen Samuels, R&D Engineering Fellow at Western Digital and member of OpenSDS TSC

About the OpenSDS Community

OpenSDS is the world’s first open source community centering on software-defined storage. It is dedicated to providing unified SDS controller framework and APIs for cross-cloud workloads. Currently, OpenSDS is supported by a number of leading storage vendors and carriers including Huawei, IBM, Hitachi, Dell EMC, Fujitsu, Western Digital, Vodafone, Yahoo! Japan, and NTT Communications.

The OpenSDS community welcomes anyone who is interested in helping to build the open standard for software-defined storage. It embraces suggestions and proposals from the open community members and developers, who help build comprehensive SDS solutions and API standards.

For more information, visit https://www.opensds.io/and https://github.com/opensds/.

Oct 17
0

OSS EU OpenSDS Mini Summit Agenda

By | Announcement

OpenSDS EUAC Meeting  Oct 24 1000-1200hrs (Steven, Cosimo, Yusuke, Kei) + Howard (2hrs)

  1. OpenSDS Requirements and Roadmap Discussion
  2. OpenSDS API
  3. Participation Discussions
  4. 2018 Event Planning
  5. Other topics?

Lunch 1200-1400hrs

OpenSDS Mini-Summit Oct 24 1400-1800hrs  (4hrs)

  1. Welcome and Self-Introductions – (15mins)
  2. OpenSDS Introduction – Steven (30mins)
  3. OpenSDS Projects Development and Demo Presentation – Howard (1hr)
  4. Break – 15mins
  5. OpenSDS End-User Presentations (guys, please give me the presentation titles)
  6. Vodafone – Cosimo Rossetti (30mins)
  7. Yahoo Japan – Yusuke Sato (30mins)
  8. NTT Communications – Kei Kusunoki (30mins)
  9. CSI Introduction – Yu Jie? Saad? (1hr)
  10. Open Discussions
Apr 24
0

DellEMC To Join Open Source OpenSDS Project To Advance Storage Interoperability

By | Announcement

Dell EMC to Join Open Source OpenSDS Project to Advance Storage Interoperability, Contribute Code

 

SAN FRANCISCO – December 19, 2016 – The Linux Foundation, the nonprofit advancing professional open source management for mass collaboration, today welcomed Dell EMC to the OpenSDS Project. The OpenSDS community is forming to address software-defined storage integration challenges with the goal of driving enterprise adoption of open standards.

As part of its support for the open source project, Dell EMC is also contributing its first project to OpenSDS, the CoprHD SouthBound SDK (SB SDK), to help promote storage interoperability and compatibility. The SB SDK allows developers to build drivers and other tools with the assurance that they will be compatible with a wide variety of enterprise-class storage products.

The OpenSDS Project is comprised of storage users and vendors, including Fujitsu, Hitachi Data Systems, Huawei, Oregon State University and Vodafone. The project will also seek to collaborate with other upstream open source communities such as Cloud Native Computing Foundation, Docker, OpenStack and Open Container Initiative.

“Dell EMC is a welcome addition to the OpenSDS Project and we look forward to its input,” said Cameron Bahar, Senior Vice President and Global Chief Technology Officer of storage at Huawei. “We invite other vendors and enterprise customers to follow Dell EMC’s lead, and to join us in creating an open storage controller solution across cloud, containerized, virtualized and other environments, and make storage-as-a-service a reality.”

Join Dell EMC on Tuesday, December 20 at 1 p.m. EST/10 a.m. PST/18:00 GMT for an introduction and architectural tour of the SB SDK, to be webcast live on YouTube. Dell EMC will demonstrate the current capabilities of the SB SDK and provide an overview of the project’s roadmap.

“As the first storage vendor to open source a software-defined storage controller, we’re very excited to join OpenSDS,” said John Mark Walker, Director of Product Management for Dell EMC. “We look forward to collaborating with customers, partners and other vendors to create open source tools and standards for storage interoperability.”

The OpenSDS technical community hosts discussions on a dedicated mailing list: tech-discuss@opensds.io. For more information about OpenSDS, please email info@opensds.io.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

# # #

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page:https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

Apr 24
0

Linux Foundation Creates Open Controller For Software-Defined Storage

By | Announcement

The Linux Foundation Creates Open Controller for Software-Defined Storage

 

Launches OpenSDS Project to address storage management challenges and drive enterprise adoption

SAN FRANCISCO – November 8, 2016 – The Linux Foundation, the nonprofit advancing professional open source management for mass collaboration, today is announcing it will host OpenSDS, a new open source project to address software-defined storage integration challenges and ultimately help drive enterprise adoption.

Storage management today is often overly complex and duplicative, with an assortment of plug-ins and competing software-defined storage controllers for each compute framework. The OpenSDS Project aims to radically simplify the state of storage by creating a common, open controller solution across cloud, containerized, virtualized and other environments.

The OpenSDS Project is comprised of storage users and vendors, including Fujitsu, Hitachi Data Systems, Huawei, Oregon State University, Vodafone and Western Digital. The project will also help unite open source communities of interest such as Cloud Native Computing Foundation, Docker, OpenStack, and Open Container Initiative.

An initial prototype release is expected to be available Q2 2017 with a BETA release by Q3 2017. OpenSDS will leverage open source technologies, such as Cinder and Manila from the OpenStack community, to best enable support across a wide range of storage products. More details on technical roadmap and release cadence will be available in the coming months.

Supporting Comments

Fujitsu
The OpenSDS Project will be a driving force for a revolution of software-defined storage and its enterprise adoption. Fujitsu is looking forward to industry-wide collaboration in OpenSDS.”
Yoshiya Eto, VP of Linux Development Div., Fujitsu

Hitachi Data Systems
“Hitachi has a long and productive history of supporting the open source community and believes it’s good for the storage community to have an open SDS controller to manage mixed storage environments across virtualized, containerized and bare-metal environments. We look forward to engaging with the OpenSDS initiative via our deep OpenStack Community collaboration, support and customer deployments.”
Michael Hay, VP and Chief Engineer, Hitachi Data Systems

Huawei
“This is a key milestone in the storage industry, with major vendors coming together for the common good of our collective customers. OpenSDS will make it easier to utilize storage from any vendor using the same SDS control architecture across different environments. Our goal is to work with the open source community to deliver value to customers with an open SDS controller that simplifies management, promotes interoperability, and delivers Storage-as-a-Service (STaaS).”
Steven Tan, Chief Architect, SDS Management, Huawei

Western Digital
“Western Digital is committed to engaging with other industry leaders to simplify the complexities of deploying and managing infrastructure built on software defined storage architectures. We are delighted to join the OpenSDS project to bring popular virtualized, containerized and cloud technologies together under a common storage controller to help users focus more on their business and less on overcoming the complexities of managing infrastructure.”
Dave Tang, senior vice president and general manager for Western Digital’s Data Center Systems business unit

The OpenSDS technical community will host discussions on a dedicated mailing list: tech-discuss@opensds.io. For more information about OpenSDS or to learn how to participate, please email info@opensds.io.

 

About The Linux Foundation
The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page:https://www.linuxfoundation.org/trademark-usage.

Linux is a registered trademark of Linus Torvalds.

Apr 24
0

OpenSDS CNCF Cluster Integration Testing

By | Blog

OpenSDS CNCF Cluster Integration Testing Summary

 

0. Introduction

OpenSDS Community was established around the end of year 2016 and the team has been busy developing the code from ground up ever since. We have applied a “discuss, develop, verify” approach that we have architects from member companies to discuss the high level architecture, developers to do a prototype implementation and verify the prototype through testing. Near the end of March we have finished the stage 1 of our PoC development, and we applied for a CNCF Cluster environment to test the PoC code.

1. CNCF Cluster Application:

Proposal at https://github.com/cncf/cluster/issues/30

1.1 Key Propositions:

What existing problem or community challenge does this work address? ( Please include any past experience or lessons learned )

Storage is an important part of functionality for Kubernetes, and currently there are multiple in-tree and out-of-tree volume plugins developed for Kubernetes, but it is hard to know whether the integration of storage provider is functioning as it should be without testing it. Moreover for storage that is provided from cloud provider (e.g OpenStack), there should be functional testing to prove the integration’s actually working.

OpenSDS is a fellow Linux Foundation collaborative project and aims to provide a unified storage service experience for Kubernetes. OpenSDS could provide various types of storage resource for Kubernetes (e.g OpenStack block and file service, baremetal storage device), and we want to use CNCF Cluster to :
a) Verify the integration of OpenSDS and Kubernetes
b) Verify the performance of the inegration via metrics such as IOPS, latency
c) Utilize the Intel S3610 400GB SSD and 2TB NLSAS HDD for storage provisioning

1.2 Important specs for the granted CNCF Cluster resource:

Bonding All hosts have 4x10Gb ports connected with LACP bonding
Transit network (Internet) static assignment on interface bond0.7
Transit gateway 10.2.0.1
Transit DNS servers 10.1.8.3
vlan range for internal usage usage 300 – 319 (create on bond0 interface)
untagged vlan (for custom PXE) 300
OS version ubuntu
Compute node spec 2x Intel E5-2680v3 12-core
256GB RAM
2x Intel S3610 400GB SSD
1x Intel P3700 800GB NVMe PCIe SSD
1x QP Intel X710
Storage Node spec 2x Intel E5-2680v3 12-core
128GB RAM
2x Intel S3610 400GB SSD
10x Intel 2TB NLSAS HDD
1x QP Intel X710

 

2. OpenSDS Integration Testing:

2.1 OpenSDS Architecture Introduction


OpenSDS consists of three layers:
● API Layer: Provides a general RSTful API for storage resource management/orchestration operations.
● Controller Layer: Provides scheduling/management/orchestration capabilities such as discovery, pooling, lifecycle, data protection and so forth.
● Hub Layer: Provides mechanisms to connect with various storage backends.

2.2 Basic Deployment Architecture:



For deployments, OpenSDS usually could be compiled into two separate modules: Control module and Hub Module. OpenSDS Control contains the API and Controller functionalites meanwhile OpenSDS Hub contains the southbound abstraction layer (we call it the “dock”) and all the necessary adaptors for the storage backends.

Total of 5 nodes are used in the testing, of which one serves as the master node and three others as the worker node. The master node is running on the compute node provided by the cluster, where as the worker nodes are running on the storage nodes.
OpenSDS’s Control module is deployed at the master node together with Kubernetes’ Control module, OpenSDS’s Hub module is deployed at the storage node together with Kubernetes’ kubelet module.

OpenStack Cinder (with LVM and Ceph as backends), OpenStack Manila (NFS) and CoprHD (with simulator of EMC storage device) are deployed as the storage resources for OpenSDS.

The advantage of deploy hub module in a distributed way on each storage node is that it could help OpenSDS scale without the need to have multiple controller instances everywhere which will inevitably introduce the problem of state syncing and brainsplit. Since Hub communicates with Control via gRPC, it could be scaled to a rather large scale. The states are maintained at the Control module whereas Hub module is stateless. When the compute platform (Kubernetes, Mesos, OpenStack …) queries about the state from OpenSDS, it could get the necessary information from the central controller, and the central controller sync up with all the underlying hubs via simple heartbeat implementation.

2.3 Scenario 1: Multiple backends support and providing PV for Kubernetes Pods


In this scenario, we successfully tested the OpenSDS southbound adaptors for OpenStack Cinder, OpenStack Manila and CoprHD functions properly. We could demonstrate the CRUD operations for volumes and shared volumes (files) on OpenStack, and CRUD operations for volumes on CoprHD. This verifies that by integrating with various backends such as OpenStack and CoprHD, OpenSDS could indeed provide a unified management framework.

We also successfully tested that via OpenSDS Flex Volume plugin, OpenSDS could provide Ceph volume which is managed by Cinder for Kubernetes Pods. Alongside with the multi-backends support, this verifies that by integrating OpenSDS with Kubernetes, it could provides storage resource from various backends for kubernetes through a unified management plane.

2.4 Scenario 2: OpenSDS storage support for Kubernetes auto-scaling


In this scenario, we have OpenSDS Hub deployed on three storage nodes with Cinder and Ceph as the backend. Volumes are created on the backend of all the three storage nodes.

We first have Kubernetes create Pod1 on 10.2.1.233 and attach the volume provided by OpenSDS, and then by pre-configuration of the replication controller, Kubernetes will scale out Pod1 by creating Pod2 and Pod3 with the same application container (Nginix in this case) on 10.2.1.234 and 10.2.1.235. Via the yaml config file, Kubernetes will notify OpenSDS about the attachment of volume for the newly created Pods. With the help of distributed hubs on each storage node, OpenSDS is able to fulfill such task. The testing verifies that OpenSDS could provide the storage support for the auto-scaling in Kubernetes.

2.5 Scenario 3: OpenSDS storage support for Kubernetes Failover


In this scenario, we have OpenSDS Hub deployed on two storage nodes with Cinder and Ceph as the backend. Volumes are created on the backend of all the three storage nodes.

We first have Kubernetes create Pod1 on 10.2.1.233 and attach the volume provided by OpenSDS, and then we kill Pod1. With the pre-configuration of the replication controller, Kubernetes will replicate Pod1 on 10.2.1.234. Via the yaml config file, Kubernetes will notify OpenSDS about the attachment of the original volume for the migrated Pod. With the help of distributed hubs on each storage node, as well as the replication support provided by Ceph, OpenSDS is able to fulfill such task. The testing verifies that OpenSDS could provide the storage support for the fast failover in Kubernetes.

 

3. OpenSDS Integration Testing Conclusions

3.1 Summary

With the help of the CNCF Cluster facilities, we have successfully performed the integration testing for OpenSDS and Kubernetes, and also OpenSDS with OpenStack storage module and CoprHD. The testing verifies the OpenSDS PoC prototype’s capability to provide storage support for Kubernetes for multiple storage backends, on auto-scaling and fast failover.

3.2 Lesson Learnt

3.2.1 OpenSDS – Kubernetes

1. The flex volume plugin will read the output from stdout, therefore if you add log functionality to the plugin itself to print to stdout, it will report error.
2. The message type in protobuf is int32, therefore extra care needs to be put on the int-int32 conversion when using gRPC.

3.2.2 OpenSDS – OpenStack

1. The mountpoint parameter in OpenStack API documentation is labeled as optional, however in the testing we found that Cinder requires this parameter.
2. The pool_name parameter for Ceph has to be set in the Cinder config file.
3. OpenStack’ Goclinet (which is used for OpenSDS Cinder and Manila adaptor) currently only supports keystone v2, which has been deprecated.

3.2.3 OpenSDS –CoprHD

1. CoprHD only supports OpenSuse, however our cluster host os is Ubuntu. We have to deploy it in a OpenSuse VM.
2. CoprHD’s CLI is not as friendly as its UI, therefore makes it harder to deploy.
3. CoprHD’s default backends are VMAX and VNX, although we have simulators, the volume could not be listed after creation.

3.2.4 CNCF Cluster:

Unfulfilled goals: We did not accomplished the second and the third goal of our cluster application, mainly due to the time constraint. We spent about one week to setup the environment and another week to do the integration testing. A longer testing period would be preferred in the future if possible.