Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Help With editting and deleting form
Hallo !!

So look at this image :

http://img194.imageshack.us/img194/8272/snapshot5f.pn

Which PHP-Editor do you think is the best?
Hi everyone,
Myself Brian Anderson. For the last 2 years i have been working a

Not connecting to DB using ruby, error:env.c:257:in oci8lib.so: ORA-12154:
Hi All,

I am trying to connect to the database server from the client server using ruby and f

Why does my php page download images over and over
Can anybody please explain to me why my php page keeps downloading the images and other items every

php file erros / need help please
Hello

Someone decoded a zend file for me.. after uploaded it on ftp i get some errors, i trye

i want run some web server under the ip sharer
hello i want run some web server under the ip sharer
my ip is floating ip but it doesnt matter

Mastering Regular Expressions, Third Edition, is available.
The Third Edition of Mastering Regular Expressions is available.

QuoteThis third edition is 5

Echoing If Function?
A script I am using has If statements in the comments form to basically tell the form what to do. Cu

help connecting a form to php and then emailing the form
Ok. So, I made a form, and I need help to where when submitted, I get an email with the submitted de

Edit MySQL Row Using PHP and HTML Form
Hello,

Here's what I'm trying to do. Build a page where a user enters a MySQL row number in.