Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Web Application Recipe
Hi Guys!

I am working with the Web Application recipes. I am currently working on the sen

PHP Code To Change Font Color in Table Cell
I would appreciate help with the following snippet of my php code. I am just trying to change the f

Change log of Cost centre change in Plant maintenance order
Hi,
How to view Change log of Cost centre in Plant maintenance order. Not visible thru 'Docume

New Login Script
Hi all, i attempted to create a whole new login script witch isnt working for some reason i dont kno

How to file_get_contents when login required?
Hello!

I am trying to read data from a page that you have to be logged into to view, I am try

Update not working ... please help
Hi,

I am trying to update a row using an edit form by passing id of the row .. but it is not

Hit counter updating once per IP - IP HIT COUNTER
I have a hit counter, for the amount of views on a tutorial.
It'll do the query and then do..

ME54/ME54N Conditional check for Release of PR
Hi Gurus,

My requirement is to put a check on the release of the PR with respect to the c

Trouble with a select list
Hello All!
I was wondering if any of you guys could help me with a small problem I'm having!

When i am Canseling the Billing Document in VF11 I am getting Shortdump
Hi Experts

When i am Canseling the Billing Document in VF11 I am getting Shortdump.