Chronicles of the Pioneers – Keeper of the Cloud

In this episode of our interviews with longest serving employees, we meet Tino, our Cloud Architect, who has been with Contabo for 13 years. From his beginnings as a sys admin apprentice to his current pivotal role as a Cloud Architect, Tino shares insights into the dynamic world of hosting and the company’s commitment to innovation. We explore the transformative moments in the history of Contabo, such as the introduction of VPS, Object Storage, or 1 Gbit/s uplinks, and how these changes influenced Tino’s everyday work. Tino also shares valuable career advice, as well as technical details of maintaining our German quality at budget-friendly prices. Have fun reading our newest episode “Chronicles of the Pioneers – Keeper of the Cloud”.

Tino, it’s been 13 years since you joined the company. Will you give us an overview of your journey with Contabo: how and why did you initially join the company?  

Fresh out of school and one year of social work, I was on the hunt for a meaningful career path. This led me to a German sysadmin apprenticeship. The attraction to Contabo lay in the dynamic world of hosting, where I would be in close proximity to Linux machines, servers, networking, and different kinds of knowledge. The small, close-knit team and the potential for continuous learning made Contabo a perfect fit for me. 

Starting with tasks like cabling and customer support, I quickly transitioned to more technical roles. My apprenticeship combined practical work with specialized classes, deepening my expertise in system administration, networking, and programming. Today, my role at Contabo involves preparing and planning systems for new products, serving as a link between developers and the company’s evolving infrastructure. 

In 2010, Giga V-Servers were launched, that is, our first VPS. Were you involved in the development of this service? How did introducing VPS change your everyday work at Contabo and the customer base? 

Although I wasn’t directly involved in the project development, I witnessed the transformative impact it had on the company and its customers. This new VPS product quickly gained popularity and created a buzz among customers. I could feel the excitement and saw the rapid growth that followed the launch.

Initially, my role at Contabo was more hardware-centric, involving the assembly of the servers used for VPS hosting. Back then, the process was quite manual, with the installation of the operating system from CDs and manual configuration. Over time, my involvement shifted towards the software side of things, especially during the transition to KVM virtualization. This shift shows Contabo’s ongoing evolution and commitment to staying at the forefront of technology in the hosting industry. 

Let’s recall the transition to 1 Gbit/s uplinks back in 2011. What preparations were made to achieve this? 

This shift required careful planning and execution. It involved the evaluation of hardware, ensuring compatibility with the new uplinks, and optimizing the network infrastructure. The first crucial step involved upgrading the top-of-the-rack switches to support 1 Gigabit. These switches are vital, as they connect the servers and, in turn, link to upstream switches or routers. I had to evaluate and test hardware models to ensure they met Contabo’s needs, such as supporting essential functions and providing accessible data via APIs. Cabling was another critical aspect of this transition, and I was involved in the process of setting up new cables from the racks to the routers.  

The transition to 1 Gigabit uplinks was essential due to the ever-growing demand for more bandwidth, particularly for Contabo’s VPS products. With multiple machines on a single server this upgrade ensured that every machine received the necessary high-speed connection. There was a significant increase in bandwidth, providing customers with tenfold performance improvement from 100 megabits to 1 gigabit. This boost in speed was crucial for customers with high-traffic websites or those serving video content, ensuring consistent and rapid delivery to their visitors. 

In 2013, there were several upgrades, including the switch to KVM for virtualization and the introduction of IPv6 for all VPS or DS. What was your role in these projects? How did these changes meet the evolving needs of our customers? 

The process of adopting KVM had begun as early as late 2011, with discussions about alternatives to VMware ESXi. It took nearly a year to evaluate and make the switch, primarily due to compatibility issues with ESXi. KVM was relatively new at the time, but it was a promising solution, and I contributed to the project during my apprenticeship’s final phase, focusing on availability. The actual switch to KVM happened swiftly, driven by the need to address problems arising from new CPUs and operating systems. Within a week, we had a functional solution to deploy everything on KVM-based systems. The transition brought performance improvements, quicker installation times, and enhanced automation capabilities for host system installations. 

The introduction of IPv6 was another notable change. Contabo acquired routers that supported IPv6, making it possible to offer this advanced internet protocol to customers. While IPv6 was considered the future at the time, it didn’t gain widespread adoption in the hosting space as expected. Nevertheless, our early support for IPv6 showcased our commitment to staying at the forefront of technology, even if the wider industry was slower to embrace the change. The transition to KVM and the adoption of IPv6 met the evolving needs of customers by providing improved performance and compatibility with new operating systems. 

One of our key principles is to maintain German quality at low prices. What are the processes, strategies, or equipment behind this? 

Our approach involves optimizing our platform by carefully selecting cost-effective yet high-performance hardware. To achieve this, we go a step further by implementing low-level optimizations on our systems. For instance, we use a custom-compiled Linux kernel with a special configuration, adding our own “special sauce” to enhance speed and tailor it to our specific requirements. This approach extends to other software used by Contabo as well, where we add our own configs to improve overall performance. 

I had been in charge of our VPS platform since early 2015 until I switched my position to Cloud Architect later. I focused on automation and analytics, aiming to extract valuable performance metrics from the host systems. For example, to make this data more accessible and actionable, I created Grafana dashboards, providing a visual representation of the performance. This data-driven approach allowed us to pinpoint areas where improvements were needed. One significant advancement was the development of load-balancing mechanisms, enabling the distribution of workloads from overloaded host systems to more efficient ones. In parallel, we continuously evaluated and introduced better hardware to enhance system performance. 

Another milestone was the introduction of SSD-based VPS in 2016. These VPS offerings, completely reliant on SSDs, replaced traditional spinning disks. Prior to this transition, we experimented with SSD caching to improve I/O performance, achieving mixed results. However, the decision to shift to full SSD storage was a pivotal moment in improving overall performance.   

This commitment to customization and optimization sets Contabo apart, as we continuously strive to go beyond using off-the-shelf solutions to create a hosting environment. The result is a balance of quality and cost-effectiveness that reflects our dedication to delivering top-notch service at affordable rates. 

The introduction of VPS S in 2018 broadened our VPS offers. What was the reason behind the introduction of this product?   

The driving force behind this addition was the fact that not all customers required high-powered servers. There was a growing demand for more budget-friendly options, targeting those who needed a server with minimal resource requirements, such as for running a monitoring system, a mail server, or a small private website. 

Despite concerns about potential resource overhead when hosting many smaller instances on a single server, we conducted testing, and the results showed that the overhead was manageable, enabling Contabo to offer the VPS S product successfully.  

 What was your role in introducing Object Storage in 2022? Were there any challenges and how did you overcome them? 

My responsibilities included selecting the software and hardware for this project. The primary challenge was to implement an S3-compatible object storage system, which is the market standard for scalable storage accessible via API. After a thorough evaluation, Ceph emerged as the chosen solution due to its unrivaled compatibility and suitability for our requirements.

While planning and designing the object storage system, the team faced challenges from the limited availability of public information about building large-scale object storage clusters. However, we relied on insights from universities, research groups, and official documentation to guide our approach. The uniqueness of our use case made it necessary to adapt and refine the implementation.    

The complexity of the hardware, with numerous spinning hard drives and a large cluster, posed challenges for testing in a controlled environment. Nonetheless, the project was a success, with three object store clusters currently in operation. One of these clusters has already undergone its first storage expansion, now offering more than two petabytes of raw storage. 

During your tenure, the customer count has grown to 160k. How has this customer base expansion influenced our infrastructure planning and scaling strategy? 

This expansion has reshaped infrastructure planning and scaling strategies. In the earlier years, the focus was on single host systems and individual racks, but as customer numbers increased, we now plan in terms of entire data center locations at a time, emphasizing the need for multiple racks to meet growing demand. The hardware has also evolved, with denser host systems providing more compute power and storage capacity. For instance, Contabo Object Storage began with one petabyte of storage and is designed to scale even beyond that, reflecting the evolving infrastructure needs. 

What was your role when opening new Data Centers on different continents over the years? Tell us a bit about how smooth operation is ensured. 

While I’m no longer deeply involved in operations, I did contribute to ensuring a smooth transition to new data centers. My responsibilities included preparing systems and templates to facilitate the seamless movement of infrastructure machines. This approach reduced the need for reinstalling infrastructure machines from scratch for each new data center. I also transitioned physical servers to virtual machines and deployed them on high-availability clusters, encompassing services such as DNS resolvers, routing machines, installation systems, and monitoring tools. 

Regarding smooth operation, robust monitoring systems are important. These systems constantly assess the health of the machines, enabling swift action in response to any issues that arise. While not entirely automated, the presence of monitoring systems and responsive staff ensures that Contabo can address and resolve any potential challenges promptly. 

What career advice do you have for those considering working in the cloud hosting industry? 

For those considering a career in the cloud hosting industry, I would give the following pieces of advice: 

1) invest in technology: it’s essential, particularly in areas like Linux and hardware; familiarity with these fundamentals is crucial for a successful career in cloud hosting; 

2) programming skills: while not mandatory, learning a programming or scripting language can be highly beneficial, so I recommend starting with Bash and Python; 

3) curiosity and continuous learning: the cloud hosting industry is dynamic, and staying informed about the latest trends and developments is vital; 

4) adopt tech strategies: look beyond your immediate area of expertise and consider adopting strategies from other departments, especially software development – understanding how code is managed can provide valuable insights into system management; 

5) infrastructure as code: explore concepts like Infrastructure as code (IaC) and tools like Ansible; IaC is an essential practice, particularly as you work with larger-scale systems, it involves putting infrastructure configurations into code, enhancing scalability and manageability;

6) documentation: writing clear, comprehensive documentation is a skill that’s highly valuable in the cloud hosting industry. 

Now that we’ve celebrated our 20th anniversary, how do you see the future of Contabo in the next 20 years? 

I can imagine the continuous expansion of Contabo in the future. The company’s growth trajectory shows no signs of slowing down; if anything, it’s accelerating. I foresee the establishment of more data centers distributed across various regions, bringing Contabo services even closer to its customers. This geographical expansion reflects the company’s commitment to providing accessible and reliable hosting services on a global scale.  

In addition to expanding our physical presence, we aim to offer more features and transition towards a more cloud-native environment. This transition will empower customers to use our services with greater ease and scalability.   

Thank you for embarking on this journey with us. We hope you found this interview both informative and inspiring. You might also like the interviews with Michael or Christoph we published earlier. Stay tuned for more interesting insights from Contabo employees.