Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

MVC - Code review
I'm in the process of trying to wrap my head around MVC, and as part of that, I'm attempting to impl

problems with contact forms that are only protected against SQL injections
What could be a potential problem with contact forms that are only protected against SQL injections

having probem inserting data into db table
hi

i have a table with following columns in it
Code: candidate_id, degree, cgpa, institute

Strange Oracle Query problem
Ok So I have this PHP class that talks to a Javascript class that basically creates a table of infor

cURL and Sessions
Ohai.

So, I'm trying to cURL a bunch of things off of a page, and put them into a session. I'

. and .. appearing instead of pictures
First of all, thanks very much for providing this forum. It is very much appreciated!

My son

Getting the full city list from maxminds geoip database
I am suprised I can't find any references to this.

I have maxminds geoip lite installed and i

Forgot password won't work
My forgot password thing is not working properly. It is giving me this errorMailbox unavailable. The

The repetition structure: the while statement not working
New programming student, have assignment as follows:
A restaurant has 4 lunch combos for custom

Uploading/Downloading files stored in MySQL database
Hey all,

This problem just came up in my website and I'm having a hard time figuring out what