Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

the problem with str_replace
$str="hahahahahahahahahahahahahaha";
$nn=1;
$str=str_replace('ha','MyGod',$str,$nn);

Parse XML
Good day,

I use oempro software and try to parse XMLReturn but it's not working for me, simpl

MySql timezone
Code: $sql = "SELECT *, date_format(date, '%m/%d/%Y at %I:%i %p' )as date FROM comments WHERE t

Need help-Error istream header declaration
Hi Everyone ,

I am having issue with istream declaration. I am new to C programming.
can a

SAP Treasury - Commodities
We are running SAP ECC 6.0 with Treasury Activated.
EA-FIN is also activated (SFW5).

check comment for html
hi, I just wanted to check if a comment a user posts contains HTML, and if it does, to not allow it

Multiple WHILE loops help
Hi there. I'm writing an application that is pulling and displaying results for real estate listing

What's wrong with my code
sobbing for many days, still don't know why please Help. Why my page isn't posted correctly. In a

Images in email problem
I found this script for sending emails, it works but if the email has <img src="image.jp

Display error above the login forum help!
Hi guys. just so you know im not asking for someone to do this for me I just need to be pointed in t