Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

WORDPRESS - Multiple Loops Only Going to One Post?
I have various pages with multiple loops and i need them to be individual to all other categories.

Undefined offset
The following script checks to see if the user answer matches the correct answer.

Form Fiel

Help with email validation please...
Hi,
Please could you help.
I have a register.php login page where users register, the detail

Populate drop down list from table??
Lets say for arguments sake that i have a table which contains the numbers 1 to 10.
How can i get

[Need Help] php timing issues
I don't know what is going wrong. I need some help with being able to set an image at 9:00am Colorad

Create multiple (n) arrays
Hi there,

I have the following need:

I have 2 arrays (coming from a databases)

why isn't this PHP code working ???
it's suppose to find a name on the database.

<?php

$s = $_POST["lname&

Oracle11g Patch issue
Hi all,

I'm getting the following error while installing Oracle11g Patch 11.1.0.7.0 on IBM-AI

Ajax Issues - Update Panel / Timer. Intellisense doesn't know about them
Type 'System.Web.UI.ScriptManager' does not have a public property named 'UpdatePanel'. That is wha

Multiple Options for a Single Page
For this example I want to use the Handlers option which is under Fed Admin and all the related codi