Black Hat Every summer, pandemics permitting, a group of volunteers gather in a Las Vegas hotel to run one of the more unusual examples of IT infrastructure on the planet: the Black Hat network operations center.

With more than 20,000 conference attendees spending the best part of a week attending classes to hone security skills, and talks about the latest exploits, you'd expect the network to be under constant assault. Attacks do happen, but as one of the NOC crew explained to The Register, not as often as you might think.

"It's very different than most environments, because when you look at the environments that you have, there's something to protect," said Dave Glover, who works by day on the RSA Netwitness Platform. "You need to protect your cloud assets, maybe text databases, files, blah, blah, blah.

"Here, there's nothing. There's nothing to really protect outside of the registration network."

That and the conference public Wi-Fi, of course. According to Neil "Grifter" Wyler, normally global lead of active threat assessments at IBM-X Force, the wireless network is as safe as it can be. Yes, people try out things on the Wi-Fi but if an intrusion is detected, the user is booted off and sent a reminder than hacking is not allowed.

This close attention to detail means that some delegates leave the conference more secure than when they arrive.

For example, Grifter said, there have been cases where attendees have turned up with malware already on their laptops, and when that code tries to communicate with its operator's command-and-control servers it's picked up and acted upon by the team.

Similarly, when unencrypted traffic is detected on the network, this is noted. Again, where possible, whoever's sending or receiving that data is identified and asked to pop down to the NOC and have a chat with the crew, who will explain why this is a really bad idea.

A common example, Glover explained, is pet cameras. The vast majority of these send material over the internet unencrypted, he said, and are ripe for snooping on or even phishing. A scammer could snatch a picture of the pet and its name from the feed and craft an email pretending to be from the camera company, included a name and picture for the pet, and ask them to renew a subscription by just adding a credit card to a convincing phony site.

And then there's the issue of people trying to compromise the network itself, which happens pretty much every show. It's understandable that someone being taught new techniques or skills is going to want to try those out, Grifter pointed out, and often the quickest way to sort this out is to locate the source - sometimes in a training session right there and then - and pop your head around the door and tell them to knock it off, please.

All the toys

Planning for Black Hat begins months in advance though the network is built on the week of the event.

The crew don't rely on the conference host's connectivity, and instead they bring in their own dedicated fiber line and build the network from scratch using their equipment. When they identify a particular firewall or router they want to use, they approach a supplier for a loan, and have never been turned down. It gives them access to interesting kit, some of which may cool to use but, due to being too much hassle to maintain, has to be dropped.

"We understand that we're in a unique position where we have essentially an unlimited budget," Grifter told us. "So we have a lot of really shiny toys. For some of them, we ultimately decided we didn't need them. There's more overhead in managing those things. And so we're just like, let's trim that out, and just go with what works."

The NOC is staffed by volunteers on secondment from their employers and it can make for strange bedfellows, Jason Reverri, senior technical product engineer at Palo Alto Networks, explained to The Register. People who would normally compete in business work together on each other's kit and are constantly learning new tips and tricks.

In some cases, he said, he'd brought in his own company's equipment to try out and see how it performed. If it was up to the job, it can be integrated, though there's no way a vendor can buy their way into the NOC: it's strictly done on quality. This also allows a certain amount of overspeccing. This year the NOC was running 288TB of storage, for example.

A lot of staff are either in the military or have served before joining the private sector. This isn't by design, Grifter explained, but it just happens that the type of people they are looking for tend to have that kind of background.

"I just think that it comes down to the level of folks that we have in there, they are people who show up on time," he told The Register.

They just do it, there's no complaint, the mission comes first

"They understand what the mission is, they understand that it's a critical mission, and they understand what's at stake. And so if the shift goes from eight to 12, to 14 to 16 hours, they just do it, there's no complaint, the mission comes first."

The NOC itself has been designed to be a place that's comfortable to work and occasionally play. Music can be heard constantly, sometimes films like Wargames, Sneakers and - of course - Hackers are shown on a big screen with the sound off and subtitles on. The room is dim, for better screen use, and comfy seats and sofas are installed, with the goal of making the environment more conducive to serious computing.

"The head of the Paris 2024 Olympics security operations center [SOC] saw an article about what happened here and then asked if he could come to Black Hat in London and embed with our team so they could learn how to better do SOC operations for Paris," Jessica Bair Oppenheimer, director of Cisco Security Strategic Alliances, told The Register.

"He was impressed, and said that when they set up the SOC for the Olympics in Paris in 2024, they're gonna turn the lights down, put on music, and we're gonna have hacker movies playing. He said it felt like getting together, like with friends essentially, and all doing something versus feeling like going to work."

The minute the conference ends on Thursday night, the teardown begins, and everything gets crated up for shipping back to vendors. Then, for those that indulge, a few drinks are had now that the shift is finally over. ®

https://www.theregister.com//2023/08/12/black_hat_network/