Amazon, CloudMQ Create "Social Data Cloud"

CloudMQ LogoI was shocked to hear a Social Data Cloud (SDC) is in full operation, created by Freedom OSS using Amazon Web Services (AWS) technology. CloudMQ is making social data available “in the cloud“, and all the major social networks are participating in the exchange.

Social Data Cloud (SDC)

Facebook, MySpace and LinkedIn all publish data about changes occurring on their social graphs via CloudMQ. FriendFeed and other aggregators then subscribe to this data. Social networks can even subscribe to each others’ data. CloudMQ is the first Social Data Cloud (SDC), and is the beginning of a trend in which all social graph information will be made available real time, on a granular level “in the cloud”.

The emergence of this Social Data Cloud (SDC) raises important questions about user privacy, social network collaboration and future industry business models.

CloudMQ and the Social Web Ecosystem

Perhaps to avoid the bandwidth costs resulting from web page scraping by aggregators (FriendFeed , Twine and Mixx) and spider traffic from search engines (Ixquick, Google), data is now being automatically broadcast by the leading social networks to select partners. Apparently, all major social networks participate in this CloudMQ exchange, accessing and providing massive amounts of real time information about the social graph. The actual data made available to CloudMQ differs by social network, as each have different rules for what profile changes, etc. should be released into the exchange.

Future Impact of the Social Data Cloud

The emergence of a Social Data Cloud will impact both users and social business models.

Users’ privacy expectations are likely out of sync with a massive, AWS powered distribution of their every action into a Social Data Cloud. Users may not expect their social activities to be broadcast this way, but it is certainly how the industry is evolving. Social web business models are moving toward an open model where they feed a Social Data Cloud, and various partners access and syndicate that data.

Facebook Example: Users Resist Openess

Facebook users seem to cry foul every time there is a change that helps Facebook to communicate their information to other Facebook users and/or business partners. Facebook had protests when launching mini-feeds, ran afoul of user expectations with Beacon and got stopped from making a Terms of Service change that gave Facebook the right to exploit users’ content across all platforms, in perpetuity.

Facebook Example: Opening the Business Model

Facebook left other social networks in the dust primarily by opening its user data to third parties, namely Facebook application developers. Users didn’t complain about that move toward openness. Instead, Facebook rose to become the #1 social network globally.

Facebook did not open its business model enough, however. Twitter emerged into prominence with microblogging activity that is completely open, part of a “public timeline”. Facebook is responding in a number of ways, including the “everyone button” Nick reported yesterday.

Nick’s right, the “everyone button” primarily reflects some bizarre Twitter envy on the part of Facebook. But the “everyone button” also shows how immense the competitive pressure is to open up social platforms for more public consumption.

In other words, Twitter’s success is pressuring Facebook to release more information into the Social Data Cloud.

Silver Lining “in the Cloud”

What’s the silver lining? When Facebook steamrolled users’ expectations and chased Twitter’s open business model, it paved the way for new players rise to prominence in the social web landscape. These players are CloudMQ/ FOSS and Amazon Web Services – new entrants who are making the Social Data Cloud a reality.

AWS and CloudMQ take Messaging to the Next Level

Amazon Web Services Logo

I did a double take when I heard there is a messaging system operating on the scale of real time social graph data. Freedom OSS credits Amazon Web Services’s infrastructure on demand with making the CloudMQ service possible. CloudMQ’s Pub Sub model, when used for social data, occurs on an “Internet scale” that can dwarf the number of transactions in similar financial pub/sub exchanges.

Just imagine the amount of social data generated by status updates, user actions and profile changes that occur on social networks. As the social web grows, the amount of information that will be available in the future is mindboggling. Apparently, the cloud infrastructure of AWS is up to the challenge.

FOSS powers CloudMQ with AWS

Freedom OSS Logo

Amazon Web Services (AWS) infrastructure underlies FOSS’ CloudMQ innovation. AWS provides infrastructure on demand (access to as many servers as you need). AWS is a recent service from Internet retailer Amazon.com that makes Amazon datacenters available to software developers as a platform, and on a pay as you go basis. Amazon’s massively scalable processing power is enabling new business models like CloudMQ to emerge in the social web.

Many services you use everyday are actually powered by CloudMQ and AWS. FriendFeed now uses CloudMQ messaging services to drive real time aggregation of social graph data, a quantum leap beyond the 1X a day batch processing that preceded CloudMQ .

CloudMQ Social Web Data Exchange

CloudMQ is creating a large scale exchange for real time data under a “pub sub” model. Social networks are able to publish and subscribe (pub/sub) to data about changes to their respective social graphs. CloudMQ created a system for social networks similar to those in the finance industry, where players like stock exchanges make financial transaction data available to one another as fellow members of an ecosystem. Interested parties can then follow exchange data via a stock trade “ticker”.

CloudMQ makes information available programmatically (computer to computer), using enterprise software like JMS (Java Messaging Services) and web services like HTTP based REST APIs. If you would like to experience this and have the technical skills to do so, you can go to CloudMQ.com, provision yourself an account, and begin accessing the data.

Technical Terms for Understanding CloudMQ

Messaging Systems

Messaging systems are critical to corporations with huge IT departments. Messaging systems send messages that communicate between servers, applications and subcomponents within software programs. Messaging systems include MSMQ and JMS, and are expensive, complex pieces of software.

EDA – Even Driven Architecture

Event driven architectures are, quite simply, event driven. Every time someone updates their Facebook profile (the ‘event’), a message is sent to subscribers of CloudMQ. This is a quantum leap from batch processing, which communicates information in large batches – after long lag times.

Message Queuing as a Service (MaaS)

MaaS describes next generation messaging, like the large scale messaging infrastructure CloudMQ makes available from the cloud. Social networks use this MaaS product to both publish and consume social web data.

Pub Sub Bus

“Pub/sub message bus” is the technical term for systems that create a data bus (message loop) for publishing and subscribing to data from disparate transaction systems.

CloudMQ and the Future of Monetization

Could CloudMQ take social web monetization to a new level?

CloudMQ enables a fuller picture of users to emerge online. CloudMQ is designed to enable “Super Crunchers” to analyze users’ behavior across all the social networks in real time. For instance, marketers watching your behavior via updates to a mini-feed on their Fan Page on Facebook will also be updated about changes to your MySpace profile that indicate your interest in a Prius.

Reality Check?

I had a “Are you kidding me?” moment when talking to Freedom OSS CEO Joel Davne and Chief Architect Max Yankelevich at an AWS event in LA this week. I couldn’t believe that so much social data is already available to those in the know, and that the social network majors must be consuming each others’ data behind the scenes.

I would love reader feedback on the CloudMQ service, as it appears to be a “game changer”.
Please send usage experiences, technical insights or strategic thoughts.