Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Company Code for Vendors created through Business Partners
Hi, experts!

I'm trying to set up the integration between Business Partner (BP) and Vendo

Help, Base64 encoding url and decoding server side.
Hello everyone, i have a website with videos and games and also provide an embed code which people c

mod_rewrite.c on windows ??
why it's not working on windows while it's working on other hosts???

this is the code i got

add 20 000 to the rowcnt
Hi ...
I need to add 20 000 to the row count but just cant find a way to do this i am new to php

how to configure ADS in ABAP instance
Hi All,

I am working on r/3 4.7 ABAP system.Can we configure ADS(adobe document services)

foreach loop, assistance request
I would like some guidance on the usage of foreach as I try to parse through a large database and wh

Get dump of MySQL DB from Live Site
I'm after an easy, but secure way of generating and importing a MySQL dump from a remote server.

Socket Server
In my following socket server, I am trying to listen to a connection through port 12345 in my web br

Php If in MySql query (hiding labels if a field is empty)
Okay, I've been trying to do this for a while, and I'm finally going to ask for help so I can get th

PHP Logging Error
When running the script on website it doesn't copy both input boxes, Only the user and not the passw