Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Business Health Check
Hello

I need to create an online business health check for a client it will be multiple choi

split values
I have values that are returned to me in this format:

name=>test,age=>49

HTML Viewer HIDE scrollbar
How can I disable scrollbars in a HTML Viewer control? I can't hide them. Even i have enough space b

button.click += new eventhandler() not works in if(!ispostback)
i want to use button.clcik event in function VIewScrap() which is called in if(!ispostback) but itdo

php global variable
how can we create global variable so we can use its value in any form.. Please give example to

Basic Question about Threading and PHP...
I have a page that I am working on and it is taking several hours to process. The basics of what th

Insert data in Mysql and move to another page
I'm stuck with this simple problem. I need to insert data in Mysql and then hit submit and move to a

DataTable Help Required
Hi all I am creating an app that utilises and MS Access back end and queries it quite a bit at vario

Displaying Multidimensional Arrays...about to put my head in a vice, PLEASE HELP
I am fairly new to PHP and MySQL and I have a problem which I have been trying to resolve for days.

Get ID from Database when Posting with a HTML Form
I have a form I am submitting to a MySql database. Each product has an ID # attached to it that auto