White Wolf Security

Syscan 09 - Singapore

We've spent most of this week putting everything back together after a week in Singapore for the Syscan conference there. I didn't get to sit in on any of the talks as we were either setting up or managing the Capture the Flag competition there for the entire conference but I heard good things about a lot of the talks (Dave Aitel from Immunity stopped by to check on the competition periodically and share a few details about the recent talks).

This competition added some new challenges for our infrastructure and allowed us to try out some new things, as well. Everything went off without a hitch thanks in large part to our sponsors and the excellent work of Thomas Lim at COSEINC. Special thanks are due to the sponsors of the CTF at Syscan, iSight Partners, that provided the prizes for the event as well as some support. And thanks also to Ben Nagy and Cédric Blancher who killed a lot of time at the event with us.

The Event

The CTF event for Syscan was fairly similar to our previous exercises with a few differences that made this event unique. This event had 8 teams participating (max 3 members per team), 7 of the teams were from Singapore (most from various technical schools on the island) and one team flew in from South Korea to participate in the event. The teams were scored as usual for our events, a combination of service availability and integrity, responses to provided tasks and their ability to defend their networks against attackers.

The big difference in this competition was that each team had once computer to use as their attacking computer, so each team had to defend their network and attack the networks of the other 7 teams at the same time. Our competitions usually have the participating teams focus solely on defense (with a dedicated "Red Cell" team that attacks all the networks), but we decided to try something different for this one. Ostensibly each team had one attacker and two defenders at any given time (that was how many keyboards they had, though their resources could obviously be shifted as necessary).

The prize for this event, generously provided by iSight Partners, was $10,000S for first place.

Challenges

This was the first event of this size and complexity that we've done internationally so it presented a few new challenges for us. When we do our exercises in the US we usually set up the entire environment in our lab and test everything, then pack up the infrastructure and bring it with us to the location. This means that we've tested all the hardware and software and know that it's working (or, at least, that it was before we left). International shipping of that much gear (including routers, firewalls, servers, etc.) just wasn't practical for this event, so instead Thomas from COSEINC provided us with a set of very nice ESX Servers on site to use for our servers, as well as 24 desktops rented in Singapore for the teams to use for the exercise. This also meant that we had to be doubly sure that we had all the servers and resources that we might need. Having someone at the office try and transfer a 6GB VM over the hotel's connection would not have worked out very well.

Language and culture were also an interesting issue for this event. Our setup (and specifically our documentation) is really written with an American mindset about the rules and how they'll be interpreted (mostly because that's what we are, and I don't know how to write in other people's mindsets). For this exercise not everyone spoke English (the Korean team had a translator to relay questions between the team members and us) and the culture in Asia is such that nobody wants to break any rules, or even potentially break any rules. We had a lot of teams questioning very specifically about things they were and weren't allowed to do (specific attacks they could use, closing ports, disabling services, etc.). Even when these things were mentioned in the documents we provided, because this was a competition many players would double-check just to make sure that it was correct.

Overall

The competition went extremely smoothly and all the participants seemed to enjoy themselves (especially the kids that took home $10,000S). With the exception of some minor hiccups (hotel power is universal, if nothing else) everything went as planned. To all our participants, if you happen to be reading this, thanks for playing, and hopefully we'll see you next year!

CCDC `09 - Regional Qualifiers

We held the 2 rounds of regional qualifiers at our facility January 17th and 24th. Interest in the competition has been steadily increasing over the last few years, so we've had to break the competition down into two 1-day competitions to accommodate the additional participating universities. Each 1-day event pitted the teams of four universities against one another, with the top two teams advancing to the next round (the national qualifier held in March).

The rules of the competition are fairly simple. Each team of students inherits a network to defend. The students are given a list of the operating systems and services they'll have to administer and defend during the competition. The teams are scored based on 4 criteria:

1. the ability to defend themselves from the hackers (aka the Red Team)
2. the ability to complete business injects
3. the ability to gather information about, and respond to, known intrusions (incident response)
4. the ability to maintain the availability and integrity of their business services

These criteria are scored in various ways: some manual, some automated and some a combination of both. In the end, the two teams that do best across all 4 scoring categories are invited back to participate in the next round.

The Infrastructure

The team's infrastructure was kept very similar to the regional competition of last year. Each time (of up to 8 students) was tasked with administrating and defending:

• A Cisco firewall
• A network-enabled security camera
• An Asterisk PBX Server and 2 VoIP phones
• A server running Nagios, as their out-of-band test system
• 6 servers (3 Windows, 3 Linux) running various versions of the operating system, comprising the fictional business's core infrastructure. This includes an Active Directory server, mail server, primary and secondary DNS, a web server, database server and a Human Resource management server

The Blue Teams

Because of the short length of the exercise (the competition ran from around 9AM to around 5PM) the teams weren't given any lead time ahead of the Red Team. At 9AM the teams were allowed onto the systems and had to move quickly to identify systems and services and begin working the plans that they had come up with in the weeks leading up to the competition.

In some cases plans were executed and worked well, in other cases plans changed on the fly based on the environment that the students found, and in all cases the Red Team's hacking activities made plans obsolete quickly. It's difficult to keep up the intensity required to defend a network that is consistently under attack for 8 hours, especially when you're aware that your network has been penetrated and you're still trying to do all the other things being asked of you, but the participating teams always manage admirably.

The goal of the Mid-Atlantic CCDC is not to present a true-to-life realistic scenario for the students, but to drop them right into the middle of a worst-case scenario, and see which teams can organize under pressure and rise to the top. Given those constraints: 8 hours is a long, long day.

During the After Action Review (AAR) the teams discussed the strategies they began the day with, and which ones were effective or not. Many teams started the day with strategies of immediately applying OS patches to defend against known vulnerabilities. This strategy frequently didn't work due to shared bandwidth issues (4 teams attempting to pull hundreds of megabytes of updates through one shared Internet pipe), which meant that often teams were waiting 60 minutes or more to get patches in place, while their systems were under attack.

One team came to play with only 2 members and did extremely well. One was a Linux expert, one a Windows expert, so they split the systems in half by OS and focused only on their area of expertise. Their strategy was to completely ignore business injects (the tasks given to teams to simulate everyday business requests, such as password changes and new account creation) and focus only on keeping their required services up and running correctly. This team did amazingly well. Sadly, the weighted nature of the scoring means that abandoning one part of the competition completely makes it unlikely to move on to the next round. If they had just one more person to handle administrative tasks, though...

The Red Team

As the size and number of the blue teams increases the red team is always there to match. This year had our largest number of hackers so far (17 hackers showed up for the January 17th date). Our hackers included professional government penetration testers, university professors and students, security professionals as well as just plain hobbyists and enthusiasts. This year also had a few blue team members from previous years that have since graduated, but decided to come back and join in on the fun from the other side.

The Red Team's lair can always be identified by the low overhead lighting, disproportionate number of LCD screens to people, War Games (or Hackers) being played on an overhead projector and, of course, the periodic laughter when something goes incredibly right. There were the usual fun pranks: cancelling patch installs and system reboots and remotely initiating Windows Vista installs. But there were also some new pranks tried out, like replacing the logo on VoIP phones, and the accidental destruction of a VMWare config file (oops).

These 1-day exercises are always fast-paced and usually have a few surprises throughout them, but they're always a lot of fun. Pictures from the event can be found on our website, and further information about the competition can be found on the National CCDC homepage.

Ice II Wrap-up

Well, we finally made it back from a long week in Las Vegas. A lot of interesting things happened while we were out there but, most of all, ICE II went extraordinarily well. I don't think the exercise could have run any smoother, the participants were outstanding and everyone seemed to have a great time with it and, hopefully, got something out of it.

For those just getting information about the event now the information page is still available on our site. As soon as we can compile all the logs, reports and photos those will be posted, as well. I'm going to try and keep the writeup of events fairly brief, in an attempt to keep it interesting for those that weren't there to see it. Huge thanks to SANS, for hosting the event and giving us plenty of room and equipment to work with (especially the A/V guys, who willingly ran around finding all sorts of electronic gadgets that I forgot). Special thanks to Larry Pesce and Paul Asadoorian from the Pauldotcom podcast. Larry captained the defending teams while Paul captained the attacking teams, and both took on the unenviable job of organizing a group of participants into an effective IT team. Thanks also to Bill from i-hacked who came out to be a press guy, as well as participating as both an attacker and defender on various nights.

For those interested in a participant's perspective: Paul did a write-up from his perspective as captain of the attacking teams, and Bill did a writeup of his experiences from both sides of the exercise.

** EDIT **: Larry has also posted his experiences as captain of the defending teams.

The Setup

For the event SANS graciously gave us 4 conference rooms in Caesar's Palace for the game. 2 rooms on one side of the hallway and 2 rooms directly across from that. Caesar's Palace also helped us out by setting up VLANs for us on their switches, so we didn't have to run cables across the hallway (although I'm sure no Red Cell guys would ever attempt to mess with the Defenders physical cables in an open hallway...).

On one side of the hallway were the players. Separated by a movable wall, and a few sets of velvet ropes (to keep the spectators from wandering around, as well as to make the players feel downright classy) were the Red and Blue Cells. The Red Cell, our hackers, were stationed on the left, their room generally filled with a sense of calm determination and covered with relaxing techno-beats and beer. For hours on end they attempted to wreak havoc, cause mayhem and generally infuriate the defenders.

On the right side we had the Blue Cell, our defenders. Their room was filled with bustling energy and a lot of movement and covered with the soothing sounds of Whitney Houston (not a joke, though I believe this was psychological warfare against the Red Cell). Imagine that you're in a room with a door that has no lock, and outside are a pack of velociraptors, and you know that, given only a short time, they will learn how to operate the door handle. It was a lot like that. For hours on end they attempted to prevent the wreaking of havoc, deny the causation of mayhem and generally to resist infuriation.

Night 1

As you would expect, the first day was mostly about getting acquainted with the game network. Especially for those that ended up participating on multiple nights, it was all about learning what was in play, as well as what aspects of the networks were easy or difficult (to either attack or defend). Attackers from Core Security and Immunitysec worked with the SANS attendees to break in to machines and cause general havoc. All what you'd expect from a hacking exercise.

The one big event of Night 1 came near the end of game time. As I mentioned above the only barrier between the participants and the outside world was a set of velvet ropes. While velvet ropes are an effective visual barrier, their effectiveness as a physical barrier is wholly restricted to the honor system. Near the end of Night 1 the honor system of the velvet rope broke down into what some might call "simple assault" and I'll refer to here as a "lack of civility". Alright, so it sounds much worse than it actually is. In a fit of gloating Paul came to the Defender's side and hopped the velvet rope, the defenders then euphemistically "requested Paul leave the area", by physically carrying Paul out of the room.

Paul was not amused, and Night 2 began with a strict "No physical contact, not even thumb wrestling" rule.We'll know better for next time.

Night 2

The setup for Night 2 was almost identical to Night 1 with a couple minor additions. First was the addition of a wireless access point to defend (already encrypting traffic with WPA2) as well as the addition of spectator computers. A set of laptops was set up that spectators could use to browse the websites of the teams, as well as VoIP phones that could be used to call the teams if there were problems. These laptops were the source of fun for Night 2.

One of the spectator laptops was connected wirelessly to the defending team's access point. This was our legitimate roaming client. About halfway through the night, once the defender's began to really lock their systems down tight, we allowed the Red Cell to simulate an infected client by running a single executable on the laptop. The Red Cell turned this laptop into a pivot point, using some Immunitysec tools, and started routing attacks through the, now infected, host.

Night 3

On Night 3 we decided to replace their large standard looking access point, with a tiny access point hidden underneath the Defender's table. Let's see if they notice the difference...

Again, after the Defender's really started to lock down their networks we dropped hints to the Red Cell that there might be a hidden "rogue access point" in play. The defender's reactions to this play were especially good. The mini wireless access point has a default IP address of 192.168.1.1, so several defenders watching traffic saw this address but had no idea what it was, and just chalked it up to attackers trying to phone home to an incorrect address. There were about 45 minutes left in the game when the Red Cell got to play with the access point, and at the very end Tim called the defenders on their VoIP phone to tell them where to look for the access point.

We've got some great photos of an entire team of Red Cell members watching Blue Cell guys crawling under a table to try and find this hidden access point.

Wrap-Up

Overall the games went extremely well. We had about 100 participants over all 3 nights (with some repeat offenders) as well as a couple hundred spectators that came through just to watch parts of the game or see what was going on. At a conference with about 950 people, that's pretty impressive numbers for a hacking exercise. To those participants that might be reading this, thanks for making this an awesome exercise and having fun with it. If you attended the conference, or participated in ICE II please leave us some comments, we're always looking for ways to improve the game and the experience for those playing.

And huge thanks go to Fortinet, our IDS Sponsor, for providing IDS/IPS services for the teams, as well as showing traffic analysis and trending for the game traffic in real-time. Awesome stuff, and we hope to see even more people next year!

2008 CCDC Wrap-up

The 2008 CCDC Regional competition ended yesterday afternoon. For those just checking in for the results we'll get those out of the way up front. After the two regional qualifying events the field was narrowed to 4 teams:

• Community College of Baltimore County (CCBC)
• George Washington University (GWU)
• James Madison University (JMU)
• Towson University (TU)

And the winning team going on to represent the Mid-Atlantic region at the national competition in San Antonio is: Community College of Baltimore County.

It was a long competition and we managed to throw in a lot of different things this year that we hadn't done before. Some of the biggest changes were the size of infrastructure and VoIP. Teams at the regional competition had to try and manage 17 different assets (1 firewall, 2 Cisco IP phones, 1 Asterisk server, 1 Nagios server, 4 Desktop systems, 1 PDA and 7 servers of various operating systems running different services). 17 systems may not sound like a lot to manage, but with a maximum team size of 8 people (and Towson hanging in there with only 4) and virtualization in play, it's really a lot to keep track of and try and keep running for any team.

The teams did a really great job of maintaining a (mostly) functioning infrastructure while keeping up with the ton of business injects being sent their way. Also this year we added a new "real world" asset by bringing in a business manager to act as the team's CEO. Team leaders had to have meetings with their CEO and explain why services weren't running and what was happening in the business. Luckily this year we also got a chance to talk to all of the teams privately and discuss what they like and don't like about the competition. The feedback was overwhelmingly positive, so we were glad to hear that the players on the teams are having fun and still feel like they're getting something valuable from the experience.

For those interested, picture of the event can be found:

• My pictures of the event
• Red Cell Pics of the event

I'll update these links if I find any more pictures of the event. The red cell pics in that link were taken by the son of one of the red cell members. That set also contains an infamous picture from the event. Because of the vast amount of passwords the teams need to keep track of all the teams (that I know of) write down the passwords in a book as their reference point. As the red cell photographer was walking around taking pictures of the teams he snapped a picture of the open password book, which had enough resolution to read the passwords on the sheet. I completely blame Johnny Long's No Tech Hacking for this.

The games went off incredibly smoothly this year thanks to a lot of help from a lot of different people. So thanks to everyone that helped build the game, and to all the students who participated this year. Good luck to the CCBC team, I'm sure you'll do well (and if any of you happen to check this blog, post a comment after the competition and let us know how you did).

CCDC Mid-Atlantic Regional

Just a quick post. We're all at the shop all weekend hosting the 3rd Mid-Atlantic Regional competition. Next week sometime I'll post about all the happenings here and try and get some photos up as well.

For those interested we've put the live scorebot view up on the web over here.

The scorebot view will be up for the duration of the competition (so until Sunday around 12:00PM EST). Our coder has put in a lot of work to make a really slick interface, so if you get this over the weekend browse over there and check it out real quick. Feedback is always welcome (via comments or email).

CCDC 3: RotRC

Seth Fogie (one of the Red Cell members from our second regional qualifier) has posted his writeup about the games from the perspective of the hackers over at InformIT.com in an article titled Collegiate Cyber Defense Competition Year 3: Revenge of the Red Cell. Great writeup from the other side of the exercise, since all my commentary came from what I saw while helping run the games.

By the mid afternoon we were considering our options when one of the members remembered he had some prank software from http://www.rjlsoftware.com/software/entertainment/ that would make things interesting, at least from our perspective. So we uploaded the various executables and set an AT command to launch the prank software to great effect. We can only imagine what the target team thought as their system started to install Windows Vista.

So that's what they did. You miss so much fun when you're not in the room with the hackers.

Can't wait to see what happens in March...

CCDC Regional Qualifier #2

The second regional qualifier for the Mid-Atlantic CCDC took place this past Saturday (the writeup on the first one can be found over here). There were only 4 teams competing this week, and they all did very well, especially considering the amount of pressure they were put under. We'll get the results out of the way first, then move on to the commentary.

The 4 competing teams this week were:

• George Mason University
• George Washington University
• James Madison University
• Northern Virginia Community College

And the two top-ranked teams moving on to the National Qualifier (March 7-9) are:

• George Washington University
• James Madison University

Having the opportunity to watch both regional qualifiers the teams that competed this past weekend had a much rougher time at the hands of the Red Team, and with that in mind did extremely well. The Red Team volunteers their time so about half of the Red Team participated in both regional qualifiers. Immediately after leaving the first qualifier a new email thread started about how to better organize. So with the benefit of experience, and the help of some new blood in the room the Red Team really hit their stride in the second qualifier.

Here's a quick run-down of some of the highlights of Red Team activity from the weekend:

• One of the Red Team decided to try out a new trojan he heard about (that would be Poison Ivy). Poison Ivy is the kind of trojan you would install if you really just wanted to mess with people.
• Enabling the Windows messaging service, then using it to talk to users. This was part of a distraction technique usually, to keep a team busy while another system was being attacked
• Remotely initiating a Windows Vista install on one of the team's systems (Windows Vista was not being used in the games, and I never did find out where that image came from)
• Setting up cron jobs, set to go off at 3:30, to delete the inittab file and reboot the systems (the exercise ended at 4:00)

And more! But I don't want to ruin too many surprises for those moving on to the Regional. So, all in all, the Red Team was much more organized, and at a certain point decided to focus primarily on having fun more than anything else.

We've been helping out with the CCDC games for the past 3 years, as well as helping run or participating in several others, and a similar concern seems to come up at all of them. After having been involved in each aspect of these games (as a participant playing in the games, as a designer building games and as a red team member) I thought I would comment on the mindset behind the games as a whole. Frequently there is a concern about the realism (or real-world application) of the games and how closely the scenarios portrayed in the games might actually occur.

The purpose of most cyber exercises is to test the knowledge and response of a group of people to unknown or unpredictable events. To that end the question is certainly valid. If the purpose of an exercise is to determine a response, then the stimulus should be something at least somewhat realistic in a given target environment. Although, to be clear, while the premise of the stimulus needs to remain grounded in reality, the scope can certainly be modified to be something larger than normal to increase stress. The main issue with a truly realistic cyber exercise is that the competitions are only a few days at most, which makes it inherently unlike a real life situation.

At the International Cyber Defence Workshop last year I was on a red team with a penetration tester from New York state named Jeremy Jethro. He made an excellent comment to the participants about the exercise, which was basically:

"The primary advantage of the attacker in a cyber engagement is time. He has all the time in the world to plan his attack and the defender can only sit and wait. If I were planning a real attack I would do recon then replicate all my target systems in a lab, then I would wait until December 25th or a holiday weekend then launch my attack. But in these exercises we're forced into a very small window to work with, and we have to try and gain access when we know you're watching for us."

I really liked the comment and it made perfect sense to me. Another good quote came out during the debrief after the first regional qualifier from one of the red cell members. One of the teams raised a question about whether a situation would ever happen in real life the way it happened in the competition. His response was:

"Look, this game is stacked against the defenders. When we come in we don't think we're going to get access to your systems, we know it. If every team locked down all their systems so perfectly that we couldn't get in then the administrators would give you injects to open up some vulnerabilities. This isn't a competition for us, as the Red Team, it's a competition to see who the best between all of you is. The whole game is designed to put pressure on you and see who comes out on top."

And the Red Team absolutely does that. For those participating in other games here are the pointers that Tim gave out in the debrief last Saturday:

1. Choose a team leader and assign responsibilities before a competition. (Unit cohesion)
2. Determine the values of servers and services. (Treat the game like a real scenario and not like an assignment overall)
    

CCDC Regional Qualifier #1

It's that time of year again, finally back into the swing of work after New Year's (yes, it does take two weeks) which means that I got to hear my first advertisement for Valentine's Day. Winter is settling in...well, sort of... which can only mean one thing: the annual Collegiate Cyber Defense Competition (or CCDC, to use the parlance of our times). This past Saturday we hosted the first, of two, regional qualifiers for the CCDC.

The CCDC is a DHS funded competition open to accredited colleges within the US (more info available at the CCDC's main site). The competition is a variation on Defend the Flag style games with some modifications. Participating teams inherit a network at the start of the competition. The network is meant to simulate a small to medium size corporate IT infrastructure. The students don't know specifics about the systems prior to the competition, they are only aware of basic information: operating systems, services that must be kept running and their versions, hardware types (Pix/ASA/iptables firewall), etc. There is also a professional red cell sitting nearby ready and willing to dismantle the student networks as soon as they are given the go-ahead.

The student's job is to secure the network as best they can against the red cell while maintaining the availability and integrity of their services, while also completing various business injects.

The competition, sponsored by the excellent folks over at CyberWATCH, is in its 3rd year now and we've helped set up and run the Mid-Atlantic Regional Competition (along with the brilliant people over at the Community College of Baltimore County). The popularity of the competition has also increased incredibly over the past couple years. We've gone from 5 teams in the first year to having 17 interested schools this year (down to 9 divided into the 2 regional competitions).

For those just here to get info about the competition and how it went here are all the details. The competition went from 9am-4pm at which point the winners were decided. The two top-ranked teams at the end will be going on to participate in the regional competition in March (the winner of the regional competition gets a spot at the national competition in San Antonio, TX). We had 5 teams here on Saturday, they were:

• Anne Arundel Community College
• Community College of Baltimore County
• Millersville University
• Towson University
• University of Pittsburgh

All 5 teams did an awesome job. Throughout the day people constantly commented that these were some of the most professional teams we'd seen at the competition yet. At the end of the day the two top-ranked teams that will be competing in the regional competition are:

• Community College of Baltimore County
• Towson University

One of the best parts of any cyber exercise is that at the end you get to have feedback from people that probably wouldn't give you candid feedback after you just had your company's network pummeled around you. There were 5 Secret Service agents (teams can gain points by successfully reporting hacks to law enforcement) and 6 hackers that all participated in the debrief at the end of the game. While there was a lot of good information from our hackers there was one really great war story from the day.

One of the hackers got access to one of the Linux machines on the network. He uploaded some trojan software (to trojan some of the system binaries) and compiled them. But the Linux system had an older version of the libc library, which managed to trash the libsecurity library when he compiled the trojan. Without the libsecurity library lots of necessary system functions started to fail. The hacker didn't want to leave the system this way, but was stuck with an interesting problem. Without libsecurity there is no ssh, sftp or scp access, and the server didn't have any file transfer systems installed (no FTP installed). So he could get a copy of libsecurity from another system, but couldn't actually transfer it to the system he had trashed.

So here was the solution he came up with...

1. Download a good version of libsecurity from another system he had hacked
2. gzip libsecurity
3. uuencode libsecurity
4. Transfer the gzipped, uuencoded libsecurity to one of his system's with a telnet server
5. Stream the good version of libsecurity, via telnet, to the broken system

The Rube Goldberg method of server maintenance. Awesome.

Congratulations to the two teams moving on, and good luck to the other teams participating this weekend.

UCSB Hosts Hacking Competition

The Independent has an article about UCSB hosting the largest hacking competition ever earlier this month. The International Capture the Flag (iCTF) had 35 teams from around the world participating. Teams log in remotely to participate, so no costly travel to bring all the teams together.

The team from Milan, Italy took the win against Americans, Argentines, Australians, Austrians, French, Germans, Indians and Russians (with the percentage of the population wired it seems kind of strange that they didn't have any Koreans or Chinese participating, South Korea usually fields a team at Defcon's CTF).

It looks like the iCTF primarily focuses on web application security. Teams are given a website that they have to secure and defend while at the same time attacking the other team's websites and capturing flags for points. Points were also awarded for answering trivia questions.

The competition looked like fun overall, although for best team name I think I'm gonna have to with UCSB's "Army of Darkness". Good work.