CCDC 2025

CCDC 2025 was an absolute blast; I had a lot of fun during the competition as well as the time surrounding it. Most importantly though, was one of the biggest wake-up calls I’ve ever experienced, there was plenty left I had to learn.

• If you’d like to just read the whole thing through, be my guest and read on starting at Qualifiers. I’ll still be discussing plenty of cyber-related details but it’s not everything of what you’ll see.
• If you’re only here to see some technical details, I’ll be covering those as best as I can in the Analysis section.

Qualifiers

This round of qualifiers was my second ever time participating in CCDC. This was also the case for the majority of our team as well. The layout this go around was pretty simple: eight machines, one for each team member. I was on a Linux box running Squid, a proxy/web-caching server. I assisted on a different Linux box too, I can’t remember what its purpose was at this point, unfortunately. There were only two really big security events that happened on my machine, one of which was solved by duplicating the SSH service and running that on a separate port. The other was solved in a similar manner, but we just had to do a fresh install of it instead.

One other important factor that I feel doesn’t get discussed as much are the injects. For the uninitiated, these injects are essentially simulations of business-oriented tasks that real-case blue teamers would use. Some had us submitting network diagrams, others had report templates for us to follow that essentially were incident reports. I appreciate the name “injects” for this sort of deal, as it forces you to take your mind out of completely focusing on cyber defense. Communication skills are extremely valuable with real-world work like this and I personally think it’s great practice material to have us work on, to prepare us for the real world. Of course, I say that now; during the competition it really does feel like a certified roadblock.

Towards the end, we all excitedly glanced at the scoreboard as the red team started their final efforts to make us cry. Two minutes to the hands-off-keyboard, and we’re in second place. My machine gets fork bombed and is rendered completely unusable, despite my best attempts to nip the cause in the bud. 00:00:03, 00:00:02, 00:00:01, 00:00:00 - hands off keyboard gets announced. I’m pretty sure the collective sigh of relief we let out noticeably altered the air pressure in our immediate area. Regardless, we survived and good gravy! Second place! This would later turn into fourth place due to some scoring contingencies. We were happy to survive, but the fact that we placed high enough to qualify quickly filled us with a second wind that we really could have used during the competition itself.

It is here that I would like to shout out my team captain, as he had feedback ready for all of us, for the good and bad things. We had our debrief a few days later; most of us didn’t want to think about cyber defense for a while. There was a lot of more general stuff but I’d like to highlight some of the more specific things:

• More active threat hunting was certainly needed, even just one barebones Linux box can hide a lot of valuable information, especially for indicators of compromise.
• Password changes need to be automated, each machine had quite the number of users in it and to go through each one manually would likely have taken 15-20 minutes.
• Baselines should be established. It’s hard to establish what is outside the norm if there is no norm established. Though, of course, extra consideration needs to be taken in a competition like this when a lot of the machines are pre-compromised.
• Specialized Knowledge Base per person - I feel like this could really depend on the team, the layout that I’ve noticed really working is that each team member is a subject matter expert in their own field. The team that won regionals this go-around (congrats to y’all, if you happen to be reading this) had dedicated people for Linux, Windows, firewalls, injects, basically any category. I’ll say there definitely is merit for everyone on the team knowing something about everything, but for this specific competition I’d say it’s best to just keep the focus narrow.
• Automation - As we would eventually find out, automation in general is such an important step to take, even with more than just password changes as mentioned earlier.

Between the end of the competition and regionals, we weren’t able to meet as often, but we instead supplemented with our own private practicing. Despite there being an entire month and a half between qualifiers and regionals, it really didn’t feel like that long. Before I knew it, the team and I were boarding the plane that would take us to Tampa to compete in regionals.

Regionals

As a whole, regionals was some of the most fun I’ve had in my college career. I’ll probably remember the fun stuff we did for the rest of my life: having great food as a team, participating in the competition, getting stranded on a peninsula after our last day of competing, and having some great chats with my teammates.

The competition this go-around was a completely different beast. Earlier, I mentioned that automation would be crucial. This is because of the fact that as opposed to us managing a cool, easy 8 machines, we were tasked with defending a whopping 43. Now not all of the machines were running important stuff on them, some were simply just client machines that we also had to watch now. But most machines were servers. There were a lot of ssh-enabled boxes, service-based boxes like mail servers, domain controllers (of which there were multiple now), LDAP boxes, web servers, that same squid proxy, all nice and vulnerable by design to be snacked on by red team. There were also some machines that we could program to our advantage, the biggest example that comes to mind being a box running Salt, which we would come to heavily leverage as a means of fast password changes. There were also machines that could be leveraged for monitoring services such as Splunk. Most notably were both Palo Alto and Cisco firewalls that we were able to set up; each one managed one set of infrastructure.

As the competition started, we were doing quite well, we held on to second place for a good chunk of time. However, multiple hours in, it seems like red team decided to quit playing around and put all ten of their fingers on the keyboard. Everyone got slapped around in some regard, some of us were able to avoid it while others could only delay the inevitable (spoiler alert: by the end, all of us fit into that last category.) The first day was about 8 hours of nonstop live fire, so we were all ready to take a break by the time hands-off-keyboard was announced. Directly after, there was a dinner that was held where we were able to hang out with the red team. I personally had a lot of fun here, and the two red teamers that I talked to gave me some encouragement on the state of the cybersecurity job market as of me writing this post. I was also given some pointers on my resume, for which I am super grateful. Following the dinner, the team and I decided to meet back up in one of the spots in the hotel. And there, we discovered a few things we could do:

• Stalk the red teamers’ github pages & blogs, and glean any information we could
• Come up with ideas for scripts that would be able to help us, that would be relatively short to build in a competitive environment
• Do research on existing tools to figure out how they work
• Restrategize, now that we’ve seen exactly how regionals works
• Cripple red team capability with devious little tricks like deleting binaries like sudo, wget, curl, su which would ultimately hurt our use of the machine but also REALLY make things hard for red team to do their thing. This was actually a tip we found while scouring their pages.

The meeting initially was only supposed to be an hour but it wound up being closer to two and a half from what I recall. With our minds somewhat refreshed and new plays we could spring, we went to bed in preparation for the next day…

…Where red team dialed it up to 11. The time range on this specific day was less than a third of the time we had the day before, so they were making all their big plays right then too, no holds barred. I have never seen Bonzi Buddy appear on so many machines at once. A few hours that turned out to be quite the bumpy ride on the struggle bus later, the competition finally ended and we all slumped back into their chairs. I think only one team had more than 6 boxes still running by that point, so we got completely cooked. That was the idea, though. Scores were also very pleasantly neck-to-neck, unfortunately we did not place very high, score distributions were very close. Ultimately, it was an incredible learning experience. I have more information about what I learned at the end in the Analysis section. After this, we went and had dinner as a team. It was a bad day to be the plate of shrimp & jerk chicken alfredo that I had.

After this, the team and I went through quite the struggle, but that night takes the cake as one of the most memorable nights of my life. We were effectively stranded on this small peninsula off of Florida’s State Route 60 off W Courtney Campbell Causeway. One side of the bridge leading in to this area was closed by police due to an active shooter (from what I understood at the time), and the other side they were only letting traffic go out. This other side was around 7 miles - I’ve hiked almost double that in a day before but I wasn’t in condition to try that again by this point in the night. We tried a good number of solutions: asked hotels if shuttles could take us, calling ubers and taxis, some of us tried hitchhiking, others tried asking around in the SECCDC discord server (if you’re there you can search for messages past 9 PM on 3/23/2025; they’re all there.) We did make it back to the hotel about four hours after we finished our meal. There’s a lot more that happened but that was just the sparknotes. Check out the next section to hear more in-depth details about my technical takeaways from regionals!

Analysis

I was one of the team’s Linux guys, so a lot of these points will be more centered around Linux.

What went well

• Overall defense was good
  • Watching what services were running on each box & removing any services that didn’t need to be there
  • Locking out attackers with either UFW or (eventually needed) iptables
  • Frequent and strong password changes
  • Minimizing attack vectors on existing services; typically meant disabling extra bells & whistles which could be exploited
  • Checking & fixing permissions on users
  • Running updates on both host OS and existing applications/packages
  • Checking existing system utilities for Indicators of Comrpomise (IoCs) (like checking .bashrc on any Linux system, there was stuff found there, or unknown-purpose/origin SSH keys)
  • Malwarebytes on Windows was able to help us where defender couldn’t.
    • And honestly, defender is completely fine when paired with common sense in general situations. This sure wasn’t any kind of general situation though.
  • Plenty more, I only hit the bare basics on this list; each machine was different and had different means of securing it. I only touched about 5-7 machines.
• Simulation-based training helped us a lot
  • We used existing blue team member hardware (running proxmox) to host our own Red vs. Blue trainings, which was a great simulation of what CCDC would feel like.
    • This could also be done on something like amazon EC2 instances but personal hardware is always better for the control we have over it, as well as the lack of amazon’s fees.
    • Proxmox is SUPER valuable for this sort of deal. KVM virtualization is great and we were always able to simply revert from backups once our exercises were over.
  • Existing needed skills were strengthened this way, but we also were taught brand new things that we discovered during our training as well.

What didn’t go as well

• There were some services that would have aided our defense that were made available to us that we didn’t use as much as red team would have disliked
• Putting more priority on browsing both system and service logs for IoCs would have helped, I’m sure plenty of patches could be made from information gathered from that
• We didn’t use automation for password changes until just near the end of the competition, that likely would have freed up a lot of time.
  • Something to consider is that Windows and Linux have different methods of changing passwords, and Linux is generally easier to automate. Regardless, the two would have wildly different methods of automation. We were able to utilize the provided Salt server to achieve this, however.
• Seeing as availability was also a massive part of scoring for the competition, keeping the AD Domain Controller up was crucial. Despite our strong defensive efforts, red team was not letting us keep control of that very easily.

Ideas to try

• Removing password access on root accounts, and exclusively using blue team SSH keys as entry to root/admin-enabled accounts
• Get rid or backup & remove netcat, wget, curl, sudo, su. This would heavily cripple blue team usage of machines but if red team were to get into them, they wouldn’t be able to do much either
  • Could also potentially move to /usr/local/bin, and then that way only the account this trick was used on could access these binaries - would likely be good for root user, but could affect availability in certain circumstances. Also means nothing if red team were to lock you out of your account.
• Back up important service binaries like openSSH server, apache/nginx, and whatever specific services the host is using so that if one gets nuked you can drop in another and spin it up
• Have more specialized roles per person. I noticed that in the end of the competition, the teams that specifically had a Windows guy, a Linux guy, a firewall guy, etc. placed a lot higher. We did start to do this, but some of us were a bit more jack-of-all-trades as opposed to just focusing on one subject matter.
• Restricting file execution, which could happen in multiple different ways:
  • With setting .exe files to open in notepad
  • Hardening powershell execution permissions
  • Utilizing solutions such as AppLocker

Closing Thoughts

Ultimately, I had so much fun with this competition and I’m so glad that I participated and that I competed with the people that I did. At the end of the competition, though, I realized that there is so much I need to learn to further my career. I took this knowledge and ran with it. I lined up plenty of ways that I could learn and study as I wrapped up my graduating semester and did some work with each of these.

I want to say a big thank you to those who organized and volunteered for this competition; it’s genuinely been one of the biggest learning experiences I’ve had in this field and I cannot thank you all enough for it. Fish especially, I’m very glad that you liked our meme well enough to have it framed in your workplace now. As for myself I could definitely see myself coming back to volunteer, though this would have to be after I get a job and have the knowledge that would come from that available.

With how the timing was, CCDC was a great way to close out my college career as I graduated. But that was just one adventure down, and I’m looking forward to the next adventure(s) that I go through during the course of my career.

God bless and have a great day; please reach out to me if you have any further questions.