It’s a fall afternoon and we’re driving around Secaucus, N.J. on our way to see the data warehouse where hundreds of ad-tech and digital media companies store the hardware that powers trillions of ad impressions every day. Except the problem is that no one in the car really knows where we’re going, which is also kind of the point.
Located on a nondescript street in a nondescript building is Equinix’s New York data center, where $4 trillion in business every day flows through the more than 825 networks, financial and media companies who house their hardware and technology here. All told, 90 percent of the world’s internet traffic runs through Equinix’s data centers, which are located in 185 facilities across 22 countries.
After walking up to the door, it becomes clear that this isn’t just a warehouse. Getting clearance to enter Equinix’s building includes CIA-like security with biometric scanners that lock the doors.
The reason for today’s visit is get an inside look at where ad-tech company Index Exchange keeps roughly 1,000 ad servers and its hardware. It’s one of the company’s eight locations across the world that store the high-tech stuff that processes 30 billion daily requests to serve ads on clients’ sites like Time Inc. and Condé Nast.
Despite the increase of software moving to cloud-based systems, investing in hardware—and bigger and better data centers to safely keep it—is a priority for Index Exchange. Between 2015 and 2016, the firm doubled its investment and is upping this year’s investment by two-and-a-half times, adding more space each time.
However, it’s no secret that nearly all of Index Exchange’s competitors are in the same building and are beefing up similar capabilities, though you wouldn’t know it from walking by hundreds of so-called cages. Yahoo, BrightRoll, Criteo and Google Cloud Platform are Equinix customers, among others.
“It’s kind of like [New York’s] Flatiron [neighborhood],” says Andrew Casale, president and CEO of Index Exchange. “You walk around Flatiron and you bump into people all over ad-tech.”
Similar to cubicles, companies essentially rent space—or cages—from Equinix, where they keep their hardware and infrastructure. The space is neutral, meaning that companies can build their digital ecosystems and quickly conduct business with each other without causing a conflict of interest.
While the cages are physically lined up next to each other, digital advertising companies keep the location of their cages private since it houses their proprietary technology. Cages don’t include any branding or logos to indicate who they belong to, which is intended to keep competitors away.
“You won’t see their names, you won’t see who is where but just imagine as you look at different cages—a demand-side platform (DSP) could be in one cage, a data-management-platform (DMP) could be in another,” explains Casale. “We’re literally transacting and communicating within the same buildings—nothing is faster than that and that is a borrowed trade of the financial complex.”
Security is another reason for the strict anonymity, says Rick McCalla, vp of information technology at Index Exchange.
“If you don’t know who’s there or where they are, it’s very hard to attack or do anything to damage it,” he said. “You’re there for connectivity, latency and the features it can provide. There’s no benefit to advertising who you are or what you’re connected to.”
In North America, Index Exchange has data centers in New York; Washington, D.C.; Toronto; Los Angeles and Sunnyvale, Calif.. In Europe, there are locations in Amsterdam and Frankfurt and in Asia, there’s a center in Hong Kong with plans to expand into Singapore.
The expansion plan has become big and rapid enough that McCalla’s full-time gig is to manage the data centers and he constantly travels to the locations to repair parts and make changes.
“I think we’re on our sixth or seventh expansion within New York,” McCalla explains. Each time he expands a data center, everything must be moved and built from scratch, taking six to seven months to complete and a hefty amount of manual labor.
The machines are packed with so much information that if one location shuts down, the traffic is automatically transferred to another nearby location without any impact. For example, if the New York center were to get cut off, Index Exchange’s data center in Washington, D.C. serves as a back-up center. The average server lasts for a year and a half before it needs to be replaced.
“They all work as a hive—if you turn off 10 of them, the other 790 will pick up the slack,” Casale says. “Everything can turn off in a second [and] the platform won’t even notice.”
Powering the pipes
Index Exchange’s cage houses 1,000 ad servers and is located in another nondescript nearby building, directly behind a Burlington Coat Factory store that you wouldn’t think twice about while passing by.
Inside, Index Exchange’s space is one of 2,200 cages—which have four mesh walls and a door—that span 181,000 square feet, equivalent to the size of three football fields.
“People sometimes get lost,” someone shouts out as we walk around.
The hum from the colossal amount of computers constantly running is loud. Because of the mass amount of computing power in one building, lights are dimmed while the temperature and humidity is controlled to keep everything running efficiently.
Again, confidentially is key. “At night, the cage lights are turned off and the idea of blue lighting is you can see if you walk around but you can’t see very well—you can’t make out details,” McCalla explains.
Still, companies do boatloads of business with their neighbors. Thousands of high-network fiber cables sitting in yellow trays line the ceiling, which physically connect companies who want to do business with each other.
Installing the fibers—which are called cross-connects—allows Index Exchange to plug into other ad-tech companies instantly. Index Exchange’s New York cage is plugged into about 30 neighboring cages that belong to DMPs and DSPs.
“It’s implicitly secure because it wouldn’t be there unless both parties requested it and then it’s literally instant,” Casale says. “We can close an impression all in lightspeed.”
After a few minutes of walking into Index Exchange’s cage, McCalla already finds a number of things that need to be fixed. There are fans that have stopped working and an amber light blinking on another server indicates that it’s about to die.
“When you have this many servers, you’re going to have failures,” he notes. “One cable—the thickness of your finger—can disrupt the airflow and affect the temperatures of the server.”
Tall cabinets sit neatly in the middle of the space and each cabinet holds roughly 30 servers. McCalla estimates that he has room for 10 more cabinets before Index Exchange will again have to move.
While confidentially is the name of the game, the physical closeness to anonymous competitors is hard to ignore and it’s easy to peer into other cages. Casale motions to a messier neighboring cage and says, “Look at this cage over here—I’m not going to tell you which exchange that is.”
Suffice to say, building and then tearing down data centers only to rebuild them has become a bit of a skill that Index Exchange plans to continue, he added.
“We know how to build these data centers and design for these nuances because we’ve just learned so much over such a long period of time. It’s become an incredible new muscle—you want to look at it as perpetual motion where we’re perpetually upgrading, always expanding.”