Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

MASS PM
Hello all, I'm trying to send mass private messages to users in my database but keep getting an erro

Multiple websites question.
Hi, does php have any functionality that enables a programmer to extract information from a website

php slowing my site?
Hi all,

I think that one of the reasons that my site doesn't work fast is that the code is ve

Spliting paragraph into sentences and attach in
Here is what I am trying to do

example: Para1[123.456.789!] 3 sentences
Para2

Problem displaying "scraped" XML data
I'm basicly having trouble displaying XML data scraped from an URL using cURL.

What the code

mysql UNION | warning mysql_fetch_array !!
Code: <?php
$i = 0;
$query1="SELECT * FROM `products` WHERE `div

Need help/w Page: 1,2,3,4 function!
So I have this code im working on where it's ment to display seperate pages having 20 logs each
I

Combining Two Queries
I have two scripts that each work fine by themselves. One is a search script with a variable that de

Click counter to ignore traffic from search bots
I have a click counter on my site that...well, counts the number of clicks a link gets on the frontp

Warning: session_start() [function.session-start]: Cann.....
hi, can someone help me with this? I keep getting this error....

Warning: session_start() [fu