From: Peter Nixon (listuser@peternixon.net)
Date: Thu 12 Dec 2002 - 20:02:25 EET
On Thu, 12 Dec 2002 07:32 pm, Enver ALTIN wrote:
> On Thu, 2002-12-12 at 18:45, Peter Nixon wrote:
> > It's good to see. I have a fair bit of Linux experience as I have been
> > using it for just over 8 years. Most of my experience is in networking
> > and Internet type systems and security, but I dabble in some open source
> > programming in C and perl.
>
> Sounds great.
>
> > I would be interested in talking to some databse gurus. In particular are
> > there any Postgresql gurus on this list? (I know mtu is using it thats
> > why I ask..)
>
> Guru? Most of us have been working with several database utilities in
> the past, haven't we? We have yet another list at
> linux-yazilim@linux.org.tr where we discuss about general free-software
> development, so any questions regarding to many SQL servers and database
> packages are welcome. You can easily subscribe by sending an empty mail
> to linux-yazilim-request@linux.org.tr with a subject of "subscribe"
> (sure, without quotes).
>
> Quick note: Hopefully many people on that list should be pretty fine
> with English since many developers are able to read English manuals. If
> you prefer killing another pointless discussion about the state of
> Turkish language for blah blah like me, that's OK, though (just kiddin'
Hmm...
Basically I need to batch sycronise data from (currently) 4 remote Postgres
DBs to a central DB and then remove the records from the remote DBs. This
will happen once per day. There is approx 200-250MB of records per day per DB
with approx 5MB of duplicates between DBs which need to be detected and
removed.
I am trying to come up with a more efficient way of doing a SELECT * from the
remote DB's then querying each record individually to see if it exisits in
the central DB before merging the data.
Any ideas?
I am using perl and DBI...
This would be possible natively in Oracle, but I can't find a way to do it in
Postgres without perl.
This whole process needs to at maximum be able to run between 1am and 9am
(although perferably between 2am and 8am). As I am limited to 4Mbit of
bandwidth to the remote sites, I can see that I am going to be really pushing
it to get the whole process finished if I do it sequentially.
I am open to any suggestions.
--Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc