Best way to cross matching large datasets
Posted on
16th Feb 2014 07:03 pm by
admin
Hi,
Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.
Did you know?Explore Trending and Topic pages for more stories like this.
Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.
While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:
1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….
2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….
3. Array walk where i could save one "hand coded" loop as well.
Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?
I am thankful for any advice that helps me reduce execution time!
No comments posted yet
Your Answer:
Login to answer
343 
48 
Other forums
HTML Email rec'd as Source Code
The code does create and send the email successfully. However, the message shows as html code and n
Default TimeZone
The server I'm working with is hosted in America so all times inserted into the database are coming
Refining of search Criteria
Dear Friends
I need a help, I have a search page which of cars related, which has different s
Sending CC Info by email
I am a little bit less knowledgable in the security area as most developers and I know I have a lot
Progress bar for webpage
Hi,
I want to implement a progress/waiting bar on my webpage - exactly the same thing that the Mi
cURL error
So here is my code... I got it off of here... http://www.youtube.com/watch?v=XcgQUsorF_8
Because
Count on multidimensional array
Hi, i got an array as below:
Array
(
=> Array
(
how to get sn motherboard
Hello.
I want to get serial number motherboard, cpu and else hardware.
Can You show me sam
COde for a Cc
I'm not receiving $ft as a Cc. Why is that??
$to = "$email";
$headers = "Fr
How to get variable value on next page
Hello friends
i am working on payentry page ..there i have this code.........
