Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

sql error
I have been looking at this code for 20mins and can't work out what I am doing wrong. There is somet

How to set pass login name from htaccess to php
When the user logs into my members page via htaccess, I'd like to retain the username so that I can

read integers that returns a negative or prints the average
Im trying to finish this homework assignment by tommarow afternoon and I keep getting stuck.
Wr

error reporting
hey everyone

I have a production server in which I want error reporting but only on some page

Word filter problem
Hello,

im trying to make a filter for words inputted in to my website but i want to store the

a very simple php header question (sorry!)
Firstly...I do apologise if this annoys anyone....a header error

I'm do not know php &

Query failed issue with php script but works fine in mssql manager!
hi i have the script below which copies data from one table to another but will only insert new data

Help pulling in 'id' with "read more" link
I'm trying to create snippets of my articles in the cms I'm creating, and then have them redirect to

Get Total From While Loop
I'm trying to get the total for each product and add them for a Grand Total to list outside the loop

does anyone know a php script to send mail using gmails/googles free smtp server
cause i heard u can do that but i dont know how