Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

If a form submits back to same page do you need to use $get
IM trying to pass page number and year that was selected back to the same page.

i have this c

convert PHP array to Javascript array
I have a page that gets a request sent from AJAX, and I am trying to convert a PHP array to a javasc

Display thumbnails as square while retaining aspect ratio
I am trying to figure out a way to make an image display as a square, for example 80x80 pixels, when

phpmailer class & pop.gmail.com?
Code: <?php
$mail->IsSMTP();
$mail->Host = "pop.gmail.com";

convert pps to wsf with php'how?
hi
i want to convert with php
PowerPoint file to swf file(flash)
how please?
thank!!!

Why do I get this error?
Error:

Parse error: syntax error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or

Recognising Revenue daily
Hello.
Has anyone come across daily recognition of deferred income?
We want to implement R

apart from cron
I need to run a php file every one hour. Is there any other solution apart from cron job?

Error in write byte array into document in SM58
Hi Gurus,

Getting the following error status in Transactional RFC (SM58) ..... *Error i

passing data from one page to another
hey guys
i have the follwoing code to get information from one page and place on another: