Utilisation - illustration

Monitoring Kubernetes HPA Utilization

Omer Levi Hevroni
Reading Time: 2 minutes

In the past few weeks, I was working on migrating a legacy micro-service to Kubernetes platform. The migration process was relatively simple – mainly migrating the code from .NET 4.5 framework to .NET core 2.2. After making sure the service is deployed and working is expected, I started to gradually move production traffic to the new instance. The new service handle the traffic well, and I was happy – look like this task is about to complete!

After a few days of a gradual rollout, I felt good enough to move all the traffic to the new service. And then it hit me: will the new service be able to handle the load of production traffic? I mean, I configured a Horizontal Pod Autoscaler (HPA) for this service – but does it enough? Apparently – no. But before I’ll explain why, let’s do a quick recap on HPA.

Continue reading “Monitoring Kubernetes HPA Utilization” →

Keeping Prometheus in Shape

Omer Levi Hevroni
Reading Time: 4 minutes

Prometheus is a great monitoring tool. It can easily scrape all the services in your cluster dynamically, without any static configuration. For me, the move from manual metrics shipping to Prometheus was magical. But, like any other technology we’re using, Prometheus need special care an love. If not handled properly, it can easily get out of shape. Why does it happen? And how can we keep it in shape? Let’s first do a quick recap of how Prometheus works.

Prometheus Monitoring Model

Prometheus works differently from other monitoring systems – it uses pull over push model. The push model is simple: Just push metrics from your code directly to the monitoring system, for example – Graphite.

Pull model is fundamentally different – the service exposes metrics on a specific endpoint, and Prometheus scrapes them once in a while (the scrape interval – see here how to configure it). While there are reasons to prefer push over the pull model, it has its own challenges: Each metric scrape operation can take time; what happens if it the scrape take longer then the scrape interval?

For example, let’s say Prometheus is configured to scrape its targets (that’s how services are called in Prometheus language) once in 20 seconds; what will happen if one scrape takes more then 20 seconds? The result is out of order metrics: instead of having a data point every 20 seconds, it will be every time the scrape completed. What can we do?

Continue reading “Keeping Prometheus in Shape” →

Using Wiremock for Fun and Mocking

Omer Levi Hevroni
Reading Time: 3 minutes

Mocking – something we all need, and yet, usually, hate. Simply because writing mocks is a boring and cumbersome task. The good news – it doesn’t have to be like that. There are so many good alternatives that make mocking easier: From libraries that ease the process of writing the mock, though contract testing with Pact (which solve the problem in a totally different way) to automatically generating mocks using Swagger/Open API spec. All are really good solutions – but today I want to focus on WireMock: A tool that really makes mocking a fun.

Continue reading “Using Wiremock for Fun and Mocking” →

Threat Modeling as Code

Omer Levi Hevroni
Reading Time: 6 minutes

Infrastructure-as-code (and GitOps) extend the use of source control (git) and code (well, manifest files) into a new field. This changed radically how we create infrastructure in the cloud, by making the process more robust and less error prone, and also easier for developers. Can we do the same for threat modeling? How can threat-modeling-as-code change and improve the way we do threat modeling today?

Let’s start with a really short introduction to threat modeling. Threat modeling is a practice that help us take a system design and look for possible security issues, by asking these 4 questions:

  1. What are we building?
  2. What can go wrong?
  3. What are we doing about it?
  4. Are we doing a good job?

Conduction a threat model helps to find issues sooner – and in most cases, detect issues that are hard to find using other practices. This is why conducting a threat model is a critical part in building a secre software. If you’re not familiar with this practice, I’m highly recommending this post by Adam Shostack, one of the authorities in the field. OWASP Threat Modeling project (and channel) is also an excellent learning resource.

Continue reading “Threat Modeling as Code” →

What’s inside the box?

Omer Levi Hevroni
Reading Time: 4 minutes

In the last two weeks, I was working on releasing the first OSS release of Kamus, a secret management solution for Kubernetes. To release Kamus, I had to publish a few docker images to Docker Hub and the CLI – which is deployed as an NPM package. This was the first time I had to deal with deploying packages to a repository. Doing this process thought made me thinks a lot about the (in)security of this process, and I want to share these thoughts with you. It’s all sums up to one small question – what’s inside the box? Do you know what you install when you download a package from a repository?

Continue reading “What’s inside the box?” →

Hacking Juice Shop, the DevSecOps Way

Omer Levi Hevroni
Reading Time: 6 minutes

Every DevSecOps engineer will recommend you to start using security tests. I personally just blogged about it recently – if you’re not familiar with the concept, I’ll highly encourage you to read it first. It’s sound right, but there is one important question: Can we find real exploits using security tests? Can we pwn a web application by scanning it for security issues?

The best way to find out is to take a real broken web application, scan it with various security tests and look for exploits we can use. Today victim is OWASP Juice Shop, a very famous vulnerable web application, written using NodeJS and Angular. Let’s try to hack it, the DevSecOps way!

Continue reading “Hacking Juice Shop, the DevSecOps Way” →

Investigating Kubernetes Nodes Disk Usage

Omer Levi Hevroni
Reading Time: 3 minutes

Today, I looked at our production Kubernetes cluster dashboard and I noticed something weird:

disk usage is high - almost 80%!
(sum (node_filesystem_size) – sum (node_filesystem_free)) / sum (node_filesystem_size) * 100

Well, this looks pretty bad. This is the average disk usage of the nodes running in the cluster. On average, only 20% percent of the disk in each node is available. This is probably not a good sign.

Continue reading “Investigating Kubernetes Nodes Disk Usage” →

Leveraging Web Security to Harden WordPress Security

Omer Levi Hevroni
Reading Time: 6 minutes

As you might already notice, my new blog is powered by WordPress. When I chose to start a new blog and run it WordPress, I started to look on WordPress security. There are already great posts about improving the security of your WordPress site (see for example this and this guides). There are also many posts describing various security plugins you should be installing.

All this information is critical – but it’s not enough. What about elementary web security practices? For example, leveraging security headers to protect your site? Or enabling security.txt so good hackers will know how to contact you, in case they find a vulnerability in your site?

Continue reading “Leveraging Web Security to Harden WordPress Security” →

Want to Write Good Code? Start Using Security Tests

Omer Levi Hevroni
Reading Time: 12 minutes

I like to write code. I’m doing it a lot, both professionally and for fun. Still, writing a good code is a challenge. Writing a code that is working, maintainable and secure is very hard to achieve. This is why we need automation – to spot the issues we missed. Tools like unit tests, code coverage or security tests can help detect various issues and help us write a better code.

Let’s take an example. I’ve created a small sample app using .NET core, my favorite language. I also created one container so we have something to play with.

Now it’s time to ask – does this code has security issues? Can I publish it to production?

You can try to answer this question by reading the code, or read along and learn what tools you can start using today to spot these issues. Continue reading “Want to Write Good Code? Start Using Security Tests” →