Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

code help - pagination
Hi all, I have this code, basically a user logs into my site and they get this page.

The pro

Printer configuration
Hi All,
Suppose one of the user deleted the printer settings unfortunatly ,then what is the pro

How do I get the row number from from an sql table query
Hi all

I have a table that I query and it returns a number of rows.
mysql_num_rows($query)

socket makes browser hang...
I have a socket server, and I am having a problem at the moment...

A browser sends a http hea

links using header()
Hi All

I'm not sure where to ask for help on this but I hope someone can offer some. I'm at

SWF image using php?
Hello once again.

My latest en devour requires me to produce an image of a static .swf that i

Losing 'page' data
I have this code that allows me to update my database. But after updating, I lose the $_GET['page']

Multi-user card game
I'm writing a batch of games and such. Lottery is finished, Poker,blackjack,etc are next. They will

Buggy registration system
Hey, I just started scripting in PHP, and I ran into a few problems.
Code: <?php
includ

Functions Not Loading Into Div
I had some help doing some of this but what I'm trying to do is get my functions to retrieve its val