Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

PHP and MySQL on intranet?
Is it possible to use both these on an intranet network?

An odd assignment statement. Can someone explain this assignment to me?
What purpose is served by the bit of code between the two equal signs in the $installurl set? Is th

Pls help with PHP
Pls can anyone direct me on how to create a user account for a customer. For example i want know if

need help modifying script
Hi guys,

I'm fairly new to php and am having a bit of trouble modifying a script.

This

image upload script not working with png
The basic process of this php code is to take the submitted file, convert it to a small JPG thumbnai

Custom CMS
This is a big custom CMS script I'd like to develop and would like some help atleast figuring out wh

Losing 'page' data
I have this code that allows me to update my database. But after updating, I lose the $_GET['page']

my two tables
table1 : col1 = topicid , col2 = topic

table2 : col1 = sentid, col2 = sentence

Cod

drop-down with sub-category appear
Hello,

i know how to build a simple dro-down list, im looking for a code when im gonna choose

Dynamic memory problem
Hey,

At the beginning of my code, I have this line:

Shape* gShape = 0;