Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Company Code for Vendors created through Business Partners
Hi, experts!

I'm trying to set up the integration between Business Partner (BP) and Vendo

Basic Question Regarding PHP Includes
Hi

Just starting out with PHP.

Working on a site that uses some basic includes, these

pspell
using pspell, is it possible to get words that would be best in the current phrase?

For examp

How do i use vars in an array
Hiya peeps!

How do I use a var in an array. You will see $id but it isnt working.

Do something every fifth time?
I'm trying to write a loop, but I want it to do something different after every fifth instance. Like

server trace logs
could any one tell me from where do i get error logs in xMII??I want to check the reason for dtabase

Frustrated php Newbie
First off, I am pretty much a PHP nub. I can read and understand the language (most of the time) so

Create PHP table grid help, please
I have a MySQL database setup, now it's time for the table (gridview) design in php. Here's the prob

Extracting Long text from message class with parameters
Hi,

I would like to extract the long text from a message class. However, the long text ha

Is STL important?
I'm just starting programming and I've made it until the Standart Template Library. But the chapter