Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

mySQL and PHP search
Hello,
I am trying to code a project and ran into a brick wall with one of my pages. I am pretty

Web Application Recipe
Hi Guys!

I am working with the Web Application recipes. I am currently working on the sen

Storing the referrer in session... problems with tabs
Hello All

I am currently working on an ERP project based on the MVC methodology, when a user

the problem with str_replace
$str="hahahahahahahahahahahahahaha";
$nn=1;
$str=str_replace('ha','MyGod',$str,$nn);

How can i steam a video on my PHP site? (non YouTube)
Basically I have a PHP site and don't want to give page rank to YouTube. I just want to stream a bas

Impact of movement type 412 E on MAP
Hi

Usage of movement type 412 E is causing huge change in MAP .

Is there any r

Creating XML with php
I need to creat an XML with php and have successfully produced a valid output.
The problem I hav

Multidimensional array problems in $_POST
I'm having trouble with a three-dimensional $_POST array. It starts as a two-dimensional array on th

PHP Mysql Staff Induction System
Hi there, I'm pretty new to PHP and Mysql so could really do with being pointed in the right directi

Vertical Alligning - Not working in 1 cell?
<?php

echo "<table id="valign" width="60%" style=