The Data Singularity is Here!

[Triple Zero]

The Data Singularity is Here
by Michael E. Driscoll | March 8, 2010

In the next two blog posts I'll attempt to sketch the forces behind what I'm calling, somewhat sensationally, the Data Singularity, and then (in a following post) discuss what I see as its consequences.

In a nutshell, the Data Singularity is this: humans are being spliced out of the data-driven processes around us, and frequently we aren't even at the terminal node of action. International cargo shipments, high-frequency stock trades, and genetic diagnoses are all made without us.

Absent humans, these data and decision loops have far less friction; they become constrained only by the costs of bandwidth, computation, and storage– all of which are dropping exponentially.

The result is an explosion of data thrown off from these machine-mediated pipelines, along with data about those flows (and data about that data, and so on). The machines all around us — our smart phones, smart cars, and fee-happy bank accounts — are talking, and increasingly we're being left out of the conversation.

So whether or not the Singularity is Near, the Data Singularity is here, and its consequences are being felt.

But before I discuss these consequences, I'd like to expand on the premise. The world wasn't always drowning in this data deluge, so how did we get here?

http://dataspora.com/blog/the-data-singularity-is-here/

[Number_6]

Great find. He really explained the situation quite well.
I'd like to see his criticisms, good and bad, of this situation.

Personally I feel that we wont stop at giving the job of transferring data to computers like the author pointed out, but we'll surrender almost, if not, all of our "knowledge" to the machines.

[Rococo Modem Basilisk]

I'm not sure that the ratio of autonomous processing of data to manual processing of data is new. It's just that since the advent of the digital computer we have become more accustomed to the idea that our external memory devices are used by people rather than external processing devices (or other external memory devices) autonomously.

The two big speedups in the development of the digital computer were (arguably) the 1850 US census (which, because of population growth, took ten years using the mechanical calculating machines already in use) and the world war II codebreaking projects (on both sides). Both are examples of situations wherein there's a lot of data that needs to be summarized and made meaningful. The data singularity came prior to the digital computer, and the digital computer was made to beat it back.

[Triple Zero]

Hm interesting idea. I need to read up on this 1850 census, never heard of it.

[Rococo Modem Basilisk]

The 1850 census took so long that the government decided to pay some guy to figure out how to make it faster. This guy invented an electromechanical tabulating machine, which took a bunch of punched cards and then spit out fewer punched cards (so you'd have humans punching cards with the answers from census surveys, and the output would be a card for each question indicating the number of people who answered any given way). The guy then built more and sold them to people other than the US government. Then he incorporated as the Tabulating Recording Company, which eventually was renamed to International Business Machines, then IBM.

The more you know!

(I did a research paper on this when I was a sophomore in high school)

[Requia ☣]

I thought it was the 1890 census?  I know Tabulating Machines wasn't founded till 1896.

[Rococo Modem Basilisk]

You might be right. I'm shit with dates. That'd make it the 1880 census, though. There was a census that took longer than ten years to finish tabulating, and it was after that one that they hired the guy.

I can dig up some proper references probably.

[Requia ☣]

Ok, digging up the wiki entry on the guy who founded the company, after the 1880 census took 8 years, they estimated the 1890 census would take even longer, and hired him to make the tabulating machines (which helped do it in only one year).  So yeah, the 1880 census was the really long one.

So we were both wrong 

Also, while the early tabulating machines were about processing a lot of data, the code-breaking machines, (Colossus and the electronic La Bombe that preceded it) where about taking fairly small amounts of data and doing very complicated things with it.

[Rococo Modem Basilisk]

*nod* Either way, I would maintain the argument that the data glut preceded the whole electronic digital computing thing and was a driving force behind pushing it forward. I'm not entirely sure if the lots-of-data-that-people-have-a-hard-time-accessing thing would be worse now or better without the computing and communications tech we have, but it's undeniable (probably -- someone could deny it but I'd laugh at him) that when people build a piece of technology to get something under control other people use it as a tool or excuse for making more of whatever it was supposed to eliminate the surfeit of in the first place.

[Requia ☣]

Early computing days a 3 megabyte hardrive would set you back 6 figures.  That won't even hold one filing cabinet of information.  The stuff was used much more for processing a fairly small amount of data than dealing with the glut of information.  it's only really when you get to the 80s that I've heard of businesses starting to do things like move inventories online.

Of course, its about that same time that computers really take off, I suppose the ability to do things like keep a store inventory was a huge part of that... in fact yeah, aside from graphics oriented stuff dealing with huge amounts of data (storing it, processing it, and moving it) has probably been the big reason people upgrade systems (as cool as a multi gigahertz processor sounds, its never mattered as much as a big hard drive and enough RAM to most of the customers I've dealt with).  It's not really the whole computer revolution, but that does look like a driving force of the current cycle.

[Triple Zero]

Every time I pick up one of my 2GB Micro SD cards I am struck with awe, btw.

Smaller than a post stamp, able to carry rougly 2000 books.

No idea what I'm to do with them, they fit in my MP3 player but it already got an 8GB internal memory. With the adapter (which is really no more than a piece of plastic to make it manageable and connect the contact points) they can also click into the SD slot of my netbook.

[LMNO]

I just put in an order for a new external hard drive.


IT'S A MOTHERFUCKING TERRABYTE. 






I'm feeling kind of overwhelmed.

[Rumckle]

Me too, I never thought I'd fill up my 500 Gig one, but now I only have 20 Gigs left.

No idea what I'm to do with them, they fit in my MP3 player but it already got an 8GB internal memory. With the adapter (which is really no more than a piece of plastic to make it manageable and connect the contact points) they can also click into the SD slot of my netbook.

I think micro SD cards would be great for security purposes, as they would be extremely easy to hide, hell if you had a capsule thingy, you could probably even swallow them.

[Eater of Clowns]

My previous laptop had a 30gb hard drive, and after 6 years with it I had 13 gigs left on it.  I replaced with a desktop with a terabyte hard drive.  I have no idea what do with such space.

[LMNO]

Porn.  Of course.