Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

using explode() to fill in checkboxes
Hi

I have a field stored in a table that contains regions in the UK separated by commas. Ther

Help with syntax
Hi all!

The $bridge variable is not getting passed while it is contained in the line below:

parsing error
can anyone explain this syntax error:

Parse error: syntax error, unexpected T_IF

The e

PHP and MySQL Question/Help
I have a MySQL db with all my servers and all their details like server name, IP, OS, RAM etc etc 26

SWF image using php?
Hello once again.

My latest en devour requires me to produce an image of a static .swf that i

Using system() and bringing back the results
I am aware that you can use system() within PHP to execute system commands, but I was wondering if t

Shuffle between users ??
I have multiple $users in table. I need to send them $message.

I need to send the next messag

How to refresh a parent page from a modal popup
Hi,I have a modal popup in which I need to upload a file and store in the database should give a mes

Displaying image from database
Hi,

I've got a site where that's got a database behind it. Currently it has loads of items in

Effort Estimation Template for Steady State Support
Hi Everyone,

Is there any SAP provided template/format for system change and enhancements