Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Login Script Issues
I am using a script I got from http://phpsense.com/php/php-login-script.html in order to allow peopl

A href problem under php
Hello...

I tried to explain the issue in an earlier post.. but was not clear enough....

Need help: how to catch acess of undefined class properties
Hello. I am learning OO with PHP and have hit a problem.
Some code runs as perfectly valid code,

DOMDocument parsing
Hello, I am parsing an xml file from an API which I have converted into a DOMDocument in php. This i

Linked Keywords
I am trying to get a script that makes my predefined keyword converted to links and / or converted t

php code to accept and delete incoming data
Using following HTML Code please show me how to write PHP code to accept and delete the incoming dat

The page should be expire when cilck back button
hi,
i'm new to php world.
i create user registration page.
when i submit it,data goes to my

How to store checkbox array in a session, and then be able to add to it?
I'm trying to store an array of checkbox values into a session array, sort of like a shopping cart,

Comparing MySql data and arrays.
I have a mysql table set up like this:
idsubjectbodyuseriddatetimetags1blog subjectblog body111|2

PHP MySQL Return Results Issue
Hi guys,

I'm running joomla and made a module which is using ajax.
Basically i just have a