Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Help on query replacing the date
Hi Pals,

i want to replace the current date in a column to some value say C or D or any n

Transform value 0 in no and 1 in yes help needed
Hello,
I have this in the table: "value" type int 0 to represent false and 1 to represe

help connecting a form to php and then emailing the form
Ok. So, I made a form, and I need help to where when submitted, I get an email with the submitted de

LSB (PHP 5.3) problem with static value!
hello,

i'm having a problem. static::$text variable gets lost at some point. can someone plea

Staff Induction System - No idea where to go from here!!!
Hi there, I'm pretty new to PHP and Mysql so could really do with being pointed in the right directi

Delete all from a table
I can not get the following code to work. If I run the sql statement in the db it will delete all t

PHP Upload issue
Hi guys,

I have stumble across an interesting issue with my script and is doing my head in.

retrieving images from mysql database using php
So I've been trying to figure out how to store images in a mysql database, and as far as i can tell

Switch letters in message (basic cryptology)...
I am trying to switch letters around in a message for an application on my site (see here). The prob

Whats wrong with my query?
I am trying to get this query to show the appropriate ticked vehicle roof height, unless no boxs are