Mining Social Networks

[Cramulus]

Fascinating and terrifying article about how data from social networks is already being used.

http://www.economist.com/node/16910031?story_id=16910031

Summary: lots of companies have been using network analysis software to identify key players within their networks. For example, you can determine who is important at a company by analyzing stuff like e-mail length and reply time.  "People at the top of the office or social pecking order often receive quick callbacks, do not worry about calling other people late at night and tend to get more calls at times when social events are most often organised, such as Friday afternoons. Influential customers also reveal their clout by making long calls, while the calls they receive are generally short."

as social networks become more prevalent, network analysis software is becoming a bigger deal too.

There's an awful lot of data that can be gathered about you. Some can be gathered by mining data you've explicitly posted - where you went to school, where you're employed, your friend network, etc. But there's also a lot that can be inferred about you by examining your data patterns.

Party plans turn out to be a particularly useful part of this picture. Richmond's police have started monitoring Facebook, MySpace and Twitter messages to determine where the rowdiest festivities will be. On big party nights, the department now saves about $15,000 on overtime pay, because officers are deployed to areas that the software deems ripe for criminal activity. Crime has "dramatically" declined as a result, says Mr Hollifield. Colin Shearer, vice-president of predictive analytics at SPSS, a division of IBM that makes the software in question, says it can largely replace police officers' reliance on "gut feel".

The latest version of SAS's software identifies risky borrowers by examining their social networks and Internal Revenue Service records, she says. For example, an applicant may be a bad risk, or even a fraudster, if he plans to launch a type of business which has no links to his social network, education, previous business dealings or travel history, which can be pieced together with credit-card records. Ms Joyner says the software can also determine if an applicant has associated with known criminals—perhaps his fiancée has shared an address with a parolee. Some insurers reduce premiums for banks that protect themselves with such software.

That's fucking scary! So in the near future, if you're trying to start a business and you don't have a facebook account, or your college roommate is in jail, your premium is going to be higher.

Wait -- TERRARISM!

Network analysis also has a useful role to play in counterterrorism. Terror groups are often decentralised, so mapping their social networks is akin to deciphering "a big spaghetti picture", says Roy Lindelauf of the Royal Dutch Defence Academy, who develops software for intelligence agencies in the Netherlands. It turns out that the key terrorists in a group are often not the leaders, but rather seemingly low-level people, such as drivers and guides, who keep addresses and phone numbers memorised. Such people tend to stand out in network models because of their high level of connectedness. To find them, analysts map "structural signatures" such as short phone calls placed to the same number just before and after an attack, which may indicate that the beginning and end of an operation has been reported.

SPADAC, a firm based in McLean, Virginia, performs such analyses on Egypt and other countries in Africa, the Middle East and South-East Asia. Clients include the United States, Mexico and various diplomatic services. Riots, bloody elections and crackdowns, among other things, can be forecast with improving accuracy by crunching data on food production, unemployment, drug busts, home evictions and slum growth detected in satellite images. Mark Dumas, the head of SPADAC, notes that societies with longstanding and strong social and business ties abroad weather change well. In relatively closed countries, like Egypt, rapid shifts in social networks can trigger upheaval, he says. Last year SPADAC's revenue reached $19m; this year it will exceed $27m.

I'm reminded of this episode of Popeye where Popeye and Bluto are both running for president. They're deadlocked, and the only person in the country who hasn't voted is Olive Oyl. Suddenly her opinion is the most important one in the entire country.

but more on target, I'm reminded of Minority Report... one day you're going regret clicking "accept friend request", because it will have linked you, indirectly, to some "high risk" character. Instead of running your licence plate, the cops are going to be analyzing your facebook account for connections to known criminals.

And sadly, this isn't the sort of thing that can be easily circumvented through Chaff. In the past, I've recommended putting all sorts of bunk info out there on facebook to stymie datamining. But if you only communicate with people you actually know, and network analysis focuses on length and timing of communications, it's kind of hard to conceal yourself in a meaningful way.

[Doktor Howl]

Jesus...So I'm fucked even if I DON'T have a "social" network account.

Sonofabitch.   

[Golden Applesauce]

I'll be honest, this stuff really fascinates me.  Not in a "wow, we're all fucked" morbid fascination way but in a "I really wish I had that data so I could run my own experiments on it" kind of way.

All the big movements - riots, crime fads, fashion trends, philosophical shifts - start way before anyone notices them past the individual level.  Being able to see mass movements develop and evolve before they are mass movements is an amazing thing - it's the self-consciousness and metacognition that the so-called "global consciousness" needs.

[Cramulus]

what scares me is that the access to this data is privately controlled. It's not like we have access to these tools...

Let's say the government is able to recognize the indicators of a revolution... they might be able to clip it off invisibly before it even had a chance. That's a really crazy level of social control.

[Triple Zero]

Very interesting stuff.

Also GA, there's a lot of data out there already, free for the taking, so if you wanna play, go ahead.

This guy is doing a lot of things like that with (freely available) Twitter data for instance: http://danzarrella.com/
(even though his site is horrible and he should really get a better pic of himself and the sort of statistics he does are sometimes kinda flawed and he does it in this really greasy "social marketeer" kind of way--it's still pretty interesting)

[Placid Dingo]

Very interesting stuff.

Also GA, there's a lot of data out there already, free for the taking, so if you wanna play, go ahead.

This guy is doing a lot of things like that with (freely available) Twitter data for instance: http://danzarrella.com/
(even though his site is horrible and he should really get a better pic of himself and the sort of statistics he does are sometimes kinda flawed and he does it in this really greasy "social marketeer" kind of way--it's still pretty interesting)

What's fun is that when you have a 'i wonder if' question, you can actually check.

I did the maths on Last.fm to see if insane white supremicist music was listened to by fanatic listeners (fanatic about the band, obviously they're fanatic) as opposed to being as popular as they seem.

If you look at listeners per song count, people who listen to freaky racist music are actually less obsessive about their bands than listeners of, say, coldplay.

BUT those listeners could be said to be fanatic in their genre of music; though arguably this isn't exclusive to rascists.

Anyway the point was WE HAVE DATA! Use it. Obviously the scary people are.

[Cain]

Let's say the government is able to recognize the indicators of a revolution...

They can, and do.  Frequently.