On May 17, 2024, Black Cell hosted its first blue-team CTF tournament, SecOps 2024 CTF. Capture the flag (CTF) competitions have become increasingly popular in the field of information security, both for competitive and educational purposes. In these events, participants search for hidden strings (aka. the “flags”) throughout the provided environment, thereby demonstrating their technical skills to complete different challenges.

The most well-known type of CTF is the offensive (red-team) jeopardy, where participants solve various challenges in categories like digital forensics, web applications, reverse engineering, exploitation, and cryptography. These challenges are typically standalone or only minimally connected (with some exceptions) and cover similar problem sets.

In contrast, blue-team defensive jeopardy CTFs focus on real-world scenarios involving complex systems that emulate an organisation’s IT environment. For SecOps 2024 CTF, we simulated a complex, full-chain attack that included vulnerabilities ranging from exploits in a monitoring application’s authentication process to misconfigurations in the environment’s Active Directory service, which is critical to the simulated organisation’s IT infrastructure. Contestants had to identify various stages of this attack and answer questions about technical details. In the following sections, we will explore the interesting aspects of building the competition’s infrastructure.

Victim Infra

To create the challenges and infrastructure for the blue team CTF, we designed a hybrid environment that represented a comprehensive attack scenario that targeted a small business. The infrastructure was divided into three parts; the on-premise victim environment; the cloud victim environment; and the attacker’s infrastructure.

The victim’s workstations were located in an on-premise virtual environment, while their servers ran on virtual machines in Microsoft Azure. Sensitive files in Azure were stored in a Storage Account and a Key Vault, which had deliberate misconfigurations. The victim’s on-premise and cloud environments were connected via Azure VPN Gateway, allowing for direct access to cloud resources from the victim’s workstations.

All endpoints were monitored with Microsoft Defender for Endpoint (MDE), and logs were routed to Azure Sentinel for full visibility and analysis. The response capabilities of MDE were deliberately hindered using various tools, to ensure the attack chain would not be blocked. Azure Sentinel was crucial for participants to detect and analyse security events centrally, as well as for conducting retrospective investigations of attack indicators and events.

The attack started with a fictitious company’s telemetry service, accessible from its website. This service simulated a sysadmin tool that allowed Linux-specific commands to be executed on a virtual machine in Azure. These commands were intended to help investigate network anomalies. During development, a server configuration file “accidentally” retained test login credentials, allowing an attacker with low-level access to use the telemetry interface. This web application simulated a custom authentication procedure based on the RFC7519 standard, using a more secure signature algorithm not in the original standard. The program used an adapted version of the Golang cryptographic package, crypt. A flaw in the modified authentication logic allowed the integrity protection of the payload to be bypassed, enabling an attacker with knowledge of the test credentials to create a specially crafted authentication token with administrative privileges.

The attacker used the telemetry feature to execute code remotely and gain access to the application container. The breakout from the container was reported by the Docker daemon, which was accessible from the container. This allowed the attacker to gain server access by launching a new container with a suitable bind-mount.

On the server, the attacker found Exchange Online mail service credentials in a configuration file. Using these credentials, the attacker could pivot to the on-premise environment through a phishing attack. The phishing email contained malware that, when executed, connected back to the attacker’s server. The attacker’s environment consisted of a Linux-based virtual machine in Azure, separate from the victim’s infrastructure.  The malware gave the attackers SYSTEM level privileges and also allowed for the collection of user credentials through keylogging.

Using this authentication data, the attackers were able to locate highly privileged authentication material stored insecurely in a Key Vault in Azure. Using these credentials along with the VPN configuration stolen from the infected workstation, the attackers could move laterally to a jump host hosted in the cloud and then to the domain controller. The Active Directory environment was exploited to steal the entire domains authentication material and then PowerShell over WinRM was used to distribute Python-based ransomware to encrypt data on the victim’s workstations.

When preparing the CTF environment, we created this virtual environment and made sure to install the security tools that would ensure the participants would have sufficient visibility to investigate the attacks. Once the simulated organisation’s infrastructure was created, we painstakingly went through the attack chain from the attacker’s perspective, compromising the various infrastructure elements within the environment. We used a detailed prewritten handbook as a checklist to ensure all attack phases were completed in a way that simulates a real attacker’s activities. After all, the attack chain is a story of sorts. After each major section of the “story” was complete, we would run queries in the various security solutions to ensure that the IoCs were all there before continuing. In the end we had to rebuild and compromise the environment twice, before we felt the resulting logs and forensic files would hold up to the toughest scrutiny.

Out attack playbook started with a vulnerable web application accessible from the internet, moved through phishing using mail authentication credentials, compromised the on-premise infrastructure, and exploited services in Microsoft Azure. Ultimately, the attacker gained control of the Active Directory service, the core of the organization, and deployed ransomware across the company’s workstations, crippling the fictitious company’s operations.

Forensic Data Storage

To enable participants to investigate different stages of the attack, they needed extensive visibility within the compromised environment. Most activity was visible through the MDE logs, but some challenges required techniques like memory analysis or malware reverse engineering. For these advanced tasks, we provided participants with various exports, including disk images, memory dumps, and other data files.

For small files, the distribution to participants could be managed using free hosting providers. However, the total storage for the disk images of the nine virtual machines present in the simulated environment exceeded 500 GB. Compressed and encrypted, this amounted to about 100 GB. With over 300 players, this could generate up to 30 TB of network traffic. This is a significant amount for any cloud service provider’s object storage. To optimize costs, we used two different solutions during the competition.

The first solution was the well-known BitTorrent protocol. Compressed and encrypted files were placed on servers with traffic limits on optimised networks, allowing players to redistribute the files while downloading. During tests, a torrent client running on a virtual server achieved download speeds of 850-930 Mbps with multiple simultaneous downloads.

Download speeds using BitTorrent

The second solution involved Cloudflare R2, which charges only for stored data, not for egress traffic. The service also provides S3 compatibility, allowing for easy integration with many widespread tools. Using rclone, after setting the appropriate credentials, we could upload files to R2 at ideal uplink speeds within an acceptable timeframe.

Uploading the CTF files to R2 using rclone

In R2, we could enforce geographical restrictions for compliance purposes, limiting data storage to the European Union. During tests with R2 using curl, we observed download speeds between 100-200 Mbps. With the Aria2 multi-protocol download client using 8-10 threads, we successfully reached the virtual server’s uplink limit.

The files were made available to players in encrypted form, which is standard practice in major CTF tournaments. This method of preparation allowed participants to receive the data before the CTF started, ensuring everyone began the competition on a level playing field. The password to decrypt the compressed files was distributed at the start of the tournament via an email, a CTF platform notification, and a Discord announcement.

Correspondence Services

In addition to using Discord, emails were a key communication channel for the competition, as they were used to send out credentials and competition details. Managing mass or transactional emails involves many factors to ensure emails are delivered properly and don’t end up in spam folders. Some of the crucial elements include SPF, DKIM, DMARC, and IP reputation. We aimed to find a provider with comprehensive security features and easy software integration.

JSON serialized email API request

Cloudflare’s MailChannel service – accessed via serverless Workers – was chosen as it met our strict requirements. Workers are a JavaScript runtime that reside on the edge from a network perspective and has computational functionality.  The GitHub project “Cloudflare-email” provides an API for Workers to route emails to MailChannel. However, the original version lacked DKIM support and had security issues, which we addressed in a fork and submitted improvements back to the project.

The pull request related to the suggested changes

After configuring the Worker, we tested email delivery using mail-tester.com, which scored our emails out of ten. Notably, since MailChannel is a public service, some emails might come from IP addresses listed on DNSBLs. Transactional email services work to quickly remove their IPs from these lists. In our tests, 1-2 out of 60 emails were sent from banned IPs, which was within acceptable limits, so we proceeded with this setup for the competition. For future reader’s information, Cloudflare announced on June 6, 2024, that it would discontinue its free MailChannel API service through Workers as of June 30, 2024.

CTF Platform

To host the competition and verify the submitted flags, we used CTFd, a well-known open-source CTF platform. We deployed CTFd in a container on a dedicated Hetzner server, protected by Cloudflare’s DDOS protection service. The main drawback of this setup is that DNS and web server (reverse proxy) misconfigurations can allow attackers to bypass the protection and directly interact with the origin server’s applications. The official documentation provides two methods to mitigate this risk.

The first solution is IP-based filtering. Cloudflare’s IP ranges are publicly available, allowing us to filter incoming HTTPS traffic at the firewall level. Traffic routed through Cloudflare is also encrypted using a private certificate issued by the provider with the certificate’s Subject Alternative Name field specifying the permitted domain. This approach prevents automated services from identifying the origin server’s IP address by blocking direct access and protecting the origin server from internet scanning services that use client hellos to retrieve certificates.

Cloudflare IP-range list

A potential bypass involves Cloudflare’s Worker service, which also runs on these IP ranges. With a specially configured Worker, the IP restriction can be circumvented.

The second solution is “Authenticated Origin Pulls” (mTLS). This requires configuring the web server on the origin server to perform a client-authenticated TLS handshake with Cloudflare. Without a valid client certificate and with firewall protection in place, attackers cannot directly interact with the services behind the web server. Both of these solutions were implemented in our competition infrastructure to ensure 100% uptime during the competition.

Conclusion

In conclusion, the SecOps 2024 CTF was a tremendous success, thanks to the enthusiastic participation and dedication of all involved. The intricate planning and execution of the event, from creating a realistic attack simulation to ensuring participants had the necessary forensic data and communication tools, were critical to achieving an engaging and educational experience. We hope this blog post provided valuable insights into the complexities and technical nuances of organizing a blue-team CTF event. Your contributions and commitment have been instrumental in making SecOps 2024 a memorable and enriching event. Thank you for being part of this journey with us.

In conclusion, the SecOps 2024 CTF was a tremendous success, thanks to the enthusiastic participation and dedication of all involved. The meticulous planning and execution required to create this event were no small feats. From designing a complex, realistic attack simulation to ensuring participants had the necessary forensic data and communication tools, every detail was crucial in delivering an engaging and educational experience. Our team worked tirelessly to create a hybrid environment that mirrored a real-world attack scenario, incorporating both on-premise and cloud-based systems, and carefully crafting each stage of the attack to test and enhance participants’ forensics skills.

One of the most rewarding aspects of organizing the SecOps 2024 CTF was seeing how participants interacted with the infrastructure we built. The challenges, required a diverse set of skills and demonstrated the importance of a comprehensive understanding of security practices. The use of advanced tools like Azure Sentinel provided participants with the necessary visibility to thoroughly investigate and respond to the simulated attacks.

Our efforts to facilitate data distribution and ensure seamless communication were essential to the smooth running of the competition. Implementing solutions like BitTorrent and Cloudflare R2 for data sharing, alongside robust email and messaging systems, ensured that all participants had access to the information and support they needed. These technical provisions were instrumental in maintaining a fair and level playing field, allowing everyone to focus on the challenges at hand without technical hindrances.

The success of the SecOps 2024 CTF would not have been possible without the collaboration and support of the entire community. Your active participation, insightful feedback, and unwavering enthusiasm were key drivers in making this event both enjoyable and educational. We are grateful for the opportunity to share this experience with such a dedicated group of professionals and learners.

We hope this blog post has provided valuable insights into the technical and logistical efforts behind the scenes of the SecOps 2024 CTF. Our aim was to shed light on the complexities and considerations involved in organizing a blue-team event of this scale and to highlight the collaborative spirit that made it all possible. Thank you once again for your participation and for helping to make SecOps 2024 an unforgettable event. We look forward to seeing you in future competitions and continuing to push the boundaries of cybersecurity education together.