Inspektor Gadget

Inspektor Gadget is a collection of eBPF-based gadgets to debug and inspect Kubernetes apps and resources. I'm one of the main developers and the tech lead of the project.

OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. I mainly contributed to the OpenTelemetry-Python.

Polycube

Polycube is an open source software framework that provides fast and lightweight network functions such as bridges, routers, firewalls, and others implemented using eBPF and XDP Linux kernel technologies. On top of Polycube we built pcn-iptables, a much more efficient clone of iptables, and pcn-k8s, a network provider for kubernetes.

IOvisor-ovn

IOvisor-ovn was an experimental project that implemented the datapth for the Open Virtual Networking (OVN) using eBPF. We built a daemon that read info from the OVN North and SouthBound databases, and based on that a service topology that used eBPF in the datapath was deployed in the different servers to provide networking for different hypervisors.

Squadra Corse

Squadra Corse is the student racing team from the Polytechnic University of Turin. Each year a group of students for the different faculties design and build an electric car to compete against another university teams from all the world at the Formula SAE. I was part of the electronics and control system division, specifically I was the lead of the team that built a telemetry system for the car.

Collecting Low-Level Metrics with eBPF

Metrics are a fundamental piece of any modern cloud observability solution. They allow operators to visualize the system performance and to understand if something is going wrong. Some metrics are rather easy to collect as they are exposed by the different user space applications. On the other hand, collecting operating-system level metrics is challenging: many times, the metrics of interest are not exposed, or collecting them is very expensive. eBPF is a powerful and efficient technology that allows us to get deep visibility into the operating system. eBPF programs are executed in the kernel context making it possible to collect low-level metrics like packet and bytes counters, IO operation latency, system calls invocations, etc. In this presentation, Mauricio will present the fundamental concepts around metrics, eBPF and how they are related. Then, he’ll show different projects like ebpf_exporter, Tetragon and Inspektor Gadget that enable metrics collection with eBPF.

Depurando un cluster Kubernetes con herramientas basadas en eBPF

Considera el uso de herramientas basadas en eBPF. Siendo una tecnología que permite a los desarrolladores escribir programas que se ejecutan en el kernel del sistema operativo, lo que permite una mayor visibilidad y control sobre el rendimiento del sistema.

Depurarando aplicaciones en AKS con el poder de eBPF

La arquitectura distribuida intrínseca de las aplicaciones nativas de nube las hace difíciles para depurar. Cuando hay un problema, no es claro dónde empezar a buscar o incluso que herramientas usar. Muchas de las herramientas de diagnóstico tradicionales están diseñadas para ser usadas al nivel del nodo y proceso, es decir, se tienen que ejecutar en el mismo host donde la aplicación está corriendo y usualmente solo permiten filtrar por parámetros como ID del proceso o usuario.

Depurando un cluster Kubernetes con herramientas basadas en eBPF

eBPF es una potente tecnología de Linux que permite depurar aplicaciones de una manera mucho más eficiente. En esta charla mostraré cómo dicha tecnología se puede usar en Kubernetes.

Using the EBPF Superpowers To Generate Kubernetes Security Policies

Kubernetes has several security mechanisms that can be used to secure your applications: - limit network connectivity with network policies - block some system calls with seccomp profiles - restrict access to some Linux capabilities in security contexts Defining those policies is difficult.

Debug Your Clusters with eBPF-Powered Tools

The intrinsic distributed architecture of applications running in Kubernetes makes them difficult to debug. When there is a problem, it is not clear where to start looking at or even which tools to use. Many of the traditional troubleshooting tools are designed to be used at the host and process level, i.e., they have to run on the same host where the application is running, and they usually only allow to filter by things like process PID or UID.

How to measure CPU and memory usage of eBPF programs

eBPF is now a well-known technology used for networking, observability and security purposes in the cloud native landscape. There are a lot of different projects like BCC, Cilium, Falco, Pixie and Inspektor Gadget (to mention a few) that use eBPF as its core technology.

BTFGen: one Step Closer to Truly Portable eBPF Programs

Running eBPF programs in a wide variety of target machines is a big challenge as eBPF programs depend heavily on the kernel version. The CO-RE (Compile Once - Run Everywhere) technology provides a mechanism to ship compiled eBPF programs that are patched according to the target kernel. However, it can’t be used on all target machines as it requires them to expose BTF (BPF Type Format) information.

BTFGen: one Step Closer to Truly Portable eBPF Programs

Many cloud native projects started using eBPF to provide OS and application observability, networking and security. Kubernetes deployments run on many different kernel versions and currently there is a big challenge on portability, as the eBPF programs depend heavily on the kernel version.

Extending systemd Security Features with eBPF

systemd uses eBPF to implement certain functionality like IP filtering and accounting. In this lightning talk we’ll explain how two new security features we implemented in systemd work: RestrictFileSystems and RestrictNetworkInterfaces.

Tracing Distribuido con OpenTelemetry

OpenTelemetry nace de la fusión de OpenTracing y OpenCensus, dos proyectos similares que brindan un conjunto de APIs para tracing distribuido y métricas.

Extending systemd Security Features with eBPF

systemd uses eBPF to implement certain functionality like IP filtering and accounting. These features have been traditionally implemented by writing the eBPF code directly in eBPF-assembly. It’s an efficient solution but makes their development and maintainability very difficult. Systemd recently got support for libbpf, which opens the door to adding new features much more easily. In this talk Mauricio will explain how two new security features were implemented in systemd using this new integration: RestrictFileSystems and RestrictNetworkInterfaces. RestrictFileSystems allows limiting the filesystem types that processes in a systemd service have access to and RestrictNetworkInterfaces allows limiting the network interfaces that processes in a systemd can use.

Isolate the Users! Supporting User Namespaces in K8s for Increased Security

Running a process as root inside containers is a security risk: if such a process is able to break out of the container into the host, it can cause considerable damage as it will be running as a privileged user there. The good news is that Linux has a solution for this problem: user namespaces isolate user and group IDs, so a process running as root in a container runs as non-root in the host. The bad news is that Kubernetes doesn’t yet support user namespaces. So, we created a Kubernetes Enhancement Proposal (KEP-127) with a plan to bring this support to a future release. We also implemented a prototype of this idea in Kubernetes and containerd. In this talk, I’ll introduce user namespaces and how they can increase the security of a Kubernetes cluster. I’ll explain how we are working with the community to bring this support to Kubernetes, the challenges we are facing, in particular with volumes, and how different approaches like shiftfs and idmapped mounts are trying to fix them.

Challenges of Using User Namespaces at Big Scale

Running a process as root inside containers is a security risk: if such a process is able to break out of the container into the host, it can cause considerable damage as it will be running as a privileged user there. User namespaces are a solution for this problem as they isolate user and group IDs, a process running as root in a container runs as non-root in the host. The OCI specification and projects like runc, containerd and cri-o support them, but Kubernetes doesn’t.

Making Kubernetes Safer with User Namespaces

Namespaces are a fundamental building block of containers that provide isolation to avoid them interfering with each other. Linux supports different namespaces to isolate different system resources like network stack, process IDs, cgroups, etc. User namespaces provide user IDs and group IDs isolation. A process can have different user and group IDs inside and outside of a user namespace. In particular, a process can be privileged (UID 0) inside a user namespace and have an unprivileged ID outside. User namespaces work together with other namespaces to allow a process to perform privileged operations in the namespaces it's running in without affecting other namespaces. For instance, a process can configure the network interface of the network namespace it's running in but not on other namespaces. Running a process as root inside containers is a security risk, if such a process is able to break out of the container into the host, it can cause considerable damage as it'll be running as a privileged user there. User namespaces offer a solution for this problem making it possible to run processes as root in the containers while being non-root in the host. In this case, the effect of a process breaking into the host is more limited as it won't have root privileges. User namespaces are supported in some container runtimes but Kubernetes doesn't support them yet. We have been working together with different communities to fill this gap by gathering use cases. We created a Kubernetes Enhancement Proposal (https://github.com/kubernetes/enhancements/pull/2101) with a plan to bring this support in the incoming Kubernetes releases. We have also implemented a prototype of this idea in Kubernetes and the containerd/cri runtime. In this talk, I'll introduce user namespaces and how they can increase the security of a Kubernetes cluster. I'll explain how we are working with the community to bring this support to Kubernetes, the challenges we are facing to support volumes and how different approaches like shiftfs and idmapped mounts are trying to fix them.

Networking en Kubernetes e Introducción a eBPF

Enabling Observability with OpenTelemetry

OpenTelemetry is a CNCF sandbox project composed of a set of libraries, agents and other components that enable the generation and collection of telemetry data. In this talk, Mauricio will present an introduction to the project, the usage of the OpenTelemetry API to instrument an application and the usage of the automatic instrumentation agents to generate telemetry data without changing the code of the applications.

Linux Tracing with BPF, BCC and More

BPF is a virtual machine inside the Linux kernel that allows to load user defined programs that are attached to different kernel hooks (kprobes, tracepoints, uprobes, etc). One BPF’s application is to perform tracing at the kernel level as BPF programs can capture information about different kernel events. BCC (BPF Compiler Collection) is a set of libraries for different languages such as Python, C++, Lua and many ready-to-use tracing tools. This talk will give a quick introduction of BPF. It’ll present an introduction to the BCC project and its features. A demo of the more popular BCC tools will be done. It’ll also show how to create (or customize) your own tools. Finally, it’ll show how BCC is integrated with other projects to perform tracing in cloud environments like Kubernetes.

Distributed Tracing with OpenTelemetry

Creating Complex Network Services with eBPF: Experience and Lessons Learned

This paper presents the most promising characteristics of eBPF, the main encountered limitations while implementing complex virtual network functions, and we envision some solutions that can mitigate the latter. We also summarize the most important lessons learned while exploiting eBPF to create complex network functions and, finally, we provide a quantitative characterization of the most significant aspects of this technology.

Transparent Optimization of Inter-Virtual Network Function Communication in Open vSwitch

This paper proposes an architecture that can optimize inter-VM communication in an NFV environment through the creation of direct channels between virtual machines. Particularly, our prototype can transparently optimize the data transfer between virtual machines running DPDK applications by dynamically recognizing the existence of point-to-point connections in the traffic steering rules, reverting back to the traditional VM-to-switch-to-VM approach when the optimization is no longer possible.

A Transparent Highway for inter-Virtual Network Function Communication with Open vSwitch

This paper presents a software architecture that can dynamically and transparently establish direct communication paths between DPDK-based virtual network functions executed in virtual machines, by recognizing new point-to-point connections in traffic steering rules. We demonstrate the huge advantages of this architecture in terms of performance and the possibility to implement it with localized modifications in Open vSwitch and DPDK, without touching the VNFs.