Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Blogging s/w
heya frndz!!! i want to start a blog.... m looking for a s/w like oscommerce which is thr for e-comm

php require help needed
Ok i tried to use the search funtion but the word require is everywhere.

i'm really new to cr

Force download script not handling files with spaces properly
I have a regular old php force download script, uses this code:

Code: header("Cache-Cont

Display data returned from Ajaxservices!!
I have a question about displaying results using ajax. I have part of the code below.
$.ajax({

Session variable help
I'm having a very troubling issue. Maybe I'm just looking over something but I have looked at the co

How to add advertisment on top of videos
Hello,
I was told that i should find the answer for this problem in this forums.
I searched a

ALV List Display to point to another report on Double Click
Hi,

I want my ALV List Display to point to another report on Double Click on its line ite

Redirecting Admin
In my members table, I have a field called "perm" and it's set to zero for all members. Ho

Why is this function returning a false value when it shouldn't be??
This is in an include file. I want it to check a value in an html form and see if it's just white s

fwrite error
Hi All,
Does anyone know what is causing the error in this code?

Code: <?
$error