Best Method For Moving Blocks Of Records From One Db To Another?

Feb 6, 2008

I wanted to set up a mechanism that would transfer blocks of records (a few dozen to in rare cases a few thousand), with slight modification, from one database to another. It's a sort of custom partial archiving process that would be triggered from a web-based admin application. Records in the target db would be identical except:

-- the primary key in the source table, an identity field, would be just an integer in the target table
-- the target table has an extra field, an integer batch ID supplied by the web application that triggers the process

It's a simple, if not efficient matter to do it within the web application: query the source table, suck the records into memory, and insert them one by one into the target db. This will be an infrequent process which can be done at off-hours, so a bit of inefficiency is not the end of the world. But I wondered if there is a more sensible, orthodox approach:

-- Could this process be done, and done efficiently, as a stored procedure with the batch ID passed as a parameter?
-- Is there any way to do a bulk insert from a recordset or array in memory using ADO and SQL? And if so, is that better than inserting records one by one?

Advice on the best general approach would be appreciated, and I will try to figure out the details.

View 2 Replies


ADVERTISEMENT

Best Method To Move Blocks Of Records From One Database To Another

Feb 7, 2008

I wanted to set up a mechanism that would transfer blocks of records (a few dozen to in rare cases a few thousand), with slight modification, from one database to another. It's a sort of custom partial archiving process that would be triggered from a web-based admin application in plain old ASP (not .net alas). Records in the target db would be identical except:

-- the primary key in the source table, an identity field, would be just an integer in the target table
-- the target table has an extra field, an integer batch ID supplied by the web application that triggers the process

It's a simple, if not efficient matter to do it within the web application: query the source table, suck the records into memory, and insert them one by one into the target db. This will be an infrequent process which can be done at off-hours, so a bit of inefficiency is not the end of the world. But I wondered if there is a more sensible, orthodox approach:

-- Could this process be done, and done efficiently, as a stored procedure with the batch ID passed as a parameter?
-- Is there any way to do a bulk insert from a recordset or array in memory using plain ASP, ADO and SQL? And if so, is that better than inserting records one by one?

I realize that the ASP.NET tableadapter and dataset features might provide a good solution, but in the short run I can't rewrite the whole application. Advice on the best general approach from an ASP-ADO platform would be appreciated, and I will try to figure out the details.

View 3 Replies View Related

Which Method Of Creating Records Would Yield The Best Performance?

Sep 13, 2004

This question is regarding a "helper app" I'm building to go along with my ASP.NET appplication. It will be inserting/updating records in the database as a nightly process. It is a Windows application built in VB.net.

I have a table which should always only have one of each type of record in it. This table on average will have between 100k and 500k records.

Which operation would be faster and less strain on the server?

a. Use a "if exists" and see if a record of this type already exists, if it does, update it, if not, insert the new one.

b. Unconditionally issue a delete for the record I'm about to insert, then insert the new one.

c. Create a trigger that will delete the old record if a new one is inserted?

Thanks!

View 1 Replies View Related

Better Method To Count Records In Custom Paging For SQL Server 2005

Jul 24, 2006

heres my problem, since I migrated to SQL-Server 2005, I was able to use the Row_Number() Over Method to make my Custom Paging Stored Procedure better.  But theres onte thing that is still bothering me, and its the fact the Im still using and old and classic Count instruction to find my total of Rows, which slow down a little my Stored Procedure.  What I want to know is:  Is there a way to use something more efficiant to count theBig Total of Rows without using the Count instruction???  heres my stored procedure:
SELECT RowNum, morerecords, Ad_Id FROM (Select ROW_NUMBER() OVER (ORDER BY Ad_Id) AS RowNum, morerecords = (Select Count(Ad_Id) From Ads) FROM Ads)  as testWHERE RowNum Between 11 AND 20
The green part is the problem, the fields morerecords is the one Im using to count all my records, but its a waste of performance to use that in a custom paging method (since it will check every records, normally, theres a ton of condition with a lot of inner join, but I simplified things in my exemple)...I hope I was clear enough in my explication, and that someone will be able to help me.  Thank for your time.
  

View 1 Replies View Related

Moving Records For Archiving

May 18, 2004

I've got two tables, one is an archive of the second (tables are identical). I'd like to migrate records from one to the other (as in, move, insert into one while deleting from the other).

I know I can start a transaction, do an INSERT INTO...SELECT, followed by a DELETE, check rows affected, then closing with a commit transaction (or rollback if the counts don't match), but it seems as though I might be over thinking it. Is this considered the optimal approach?

View 10 Replies View Related

Moving Records During Replication

Feb 28, 2007

I keep reading in various places of the replication documentation that records can be _moved_ as opposed to just being copied. How can I do this? I can't seem to find anywhere during setting up the publication, articles or subscription an option that allows me to do that!



Thanks,

Val

View 1 Replies View Related

SQL Server 2012 :: Set Based Method To Insert Missing Records Into Table - Right Join Not Work

Apr 24, 2014

I have table A (EmployeeNumber, Grouping, Stages)
and
Table B (Grouping, Stages)

Table A could look like the following where the multiple employees could have multiple types and multiple stages.

EmployeeNumber, Type, Stages
100, 1, Stage1
100, 1, Stage2
100, 2, Stage1
100, 2, Stage2
200, 1, Stage1
200, 2, Stage2

Table B is a list of requirements that each employee must have. So every employee must have a type 1 and 2 and the associated stages listed below.

Type, Stage
1, Stage1
1, Stage2
2, Stage1
2, Stage2
2, Stage3
2, Stage4

So I know that each employee should have 2 Type 1's and 4 Type 2's. I hope that makes sense, I'm trying to change my data because ours is very proprietary.

I need to identify employees who do not have all their stages and list the stages they are missing. The final report should only have employees and the associated missing types and stages.

I do a count by employee to see how many types they have to identify the ones that don't have all the types and stages.

My count would look something like this:

EmployeeNumber Type Total
100, 1, 2
100, 2, 2
200, 1, 1
200 1, 2

So I know that employee 100 should have 2 more Type 2's and employee 200 should have 1 more Type 1 and 2 more Type 2's based on the required list.

The problem I'm having is taking that required list and joining to my list of employees with missing data and pulling from it the types and stages that are missing by employee. I thought I could get a list of the employees that are missing information and right join it to the required list where the missing records would be nulls. But, that doesn't work because some employees do have the required information and so I'm not getting any nulls returned.

View 9 Replies View Related

Send Request To Stored Procedure From A Method And Receive The Resposne Back To The Method

May 10, 2007

Hi,I am trying to write a method which needs to call a stored procedure and then needs to get the response of the stored procedure back to the variable i declared in the method. private string GetFromCode(string strWebVersionFromCode, string strWebVersionString)    {      //call stored procedure  } strWebVersionFromCode = GetFromCode(strFromCode, "web_version"); // is the var which will store the response.how should I do this?Please assist.  

View 3 Replies View Related

Update Method Is Not Finding A Nongeneric Method!!! Please Help

Jan 29, 2008

Hi,
 I just have a Dataset with my tables and thats it
 I have a grid view with several datas on it
no problem to get the data or insert but as soon as I try to delete or update some records the local machine through the same error
Unable to find nongeneric method...
I've try to create an Update query into my table adapters but still not working with this one
Also, try to remove the original_{0} and got the same error...
 Please help if anyone has a solution
 
Thanks

View 7 Replies View Related

Blocks And Deadlocks

Jul 25, 2002

I know blocks and Deadlocks are different but how related are they? Seems like when I get reports of deadlocks I always have blocks and the blocks grow as time passes.

View 2 Replies View Related

Lock Blocks

May 24, 2001

Hi, Everyone

Can anyone tell me how to find who blocks the table or record, because the Lock Blocks is extremly high in the performance monitor.

Regards,

View 3 Replies View Related

Blocks On SQL Server

Nov 26, 2001

I need to test a block-alert and would like to test this by causing a block on our test server. Anyone have a script handy that will do this?

View 5 Replies View Related

Logging Blocks

Apr 23, 2008

I would like to log blocks to see what causes it.

One hack of a way to do it is to run code similar to that which populates the blk column in sp_who; on a job every 5 seconds. It would store the results of code similar to the sp_who output whenever any of the rows has a value greater than zero in the blk column.

This way, I can see what was going on at the time there was a problem.
Does anyone have a better strategy for this?

View 17 Replies View Related

History Blocks?

Mar 21, 2007

Is it possiable to go through the history blocks and change field names?...ie

I want to change "PS001_Pump1.RT_Daily" to "PS001_Pump1.Runtime_Daily"

so i can access the historical data thru the new tagname.

If so , is this using an udpate query?

View 1 Replies View Related

Sending Mail In Blocks

Jun 12, 2007

I need to find a way to send mail in blocks of 100 and 500 at intervals of 120 seconds. Does anyone know of any code for this?

Thanks!

Etymon

View 4 Replies View Related

Xp_sendmail Blocks Trigger ?

Jul 20, 2005

Dear All,we are running SQL2000 Sever and make use of the xp_sendmail.For any reason the mail service can run into problems and it lookslike that the statemnt below gets not finished.EXEC @Status = master..xp_sendmail @recipients=@TOList,@copy_recipients=@CCList,@subject='the subject goeshere',@message=@MailText,@no_output=TRUEUnfortunately the statement is in an update trigger and hence itblocks the table for any further updates.My questions are:Can I achieve a kind of timeout check in my trigger in order to bypassthexp_sendmail call ?In general, sending mail in a trigger may not be a good idea.How can this be solved better ?Any hint is highly welcomeRegardsRolf

View 6 Replies View Related

End Conversation Causing Blocks

Apr 9, 2007

Hi



I was trying to clean up some conversation in Service Broker and caused alot of blocking that I seem to unable to kill. there was 1 conversation that I was not able to end, so I wanted to restart sql service, But I can't even restart the SQL service. I get the following in Event Viewer



Timeout occurred while waiting for latch: class 'SERVICE_BROKER_TRANSMISSION_INIT', id 00000001A2B03540, type 2, Task 0x0000000000C2EDA8 : 0, waittime 5400, flags 0xa, owning task 0x00000002DEBCA5C8. Continuing to wait.



Has anyone come across this



thanks



Paul

View 1 Replies View Related

Using Try Catch Blocks In Stored Procedure

Mar 26, 2008

The following two stored procedures works fine. I need to add try catch blocks in the stored procedures. How to do it? Pls modify my stored procedure with try and catch blocks and in the catch block i need to call the Procedure called ErrorMsg which contains error severity.
Procedure 1------------set ANSI_NULLS ONset QUOTED_IDENTIFIER ONgo
-- =============================================-- Author:  C.R.P. RAJAN-- Create date: March 26, 2008-- Description: PROCEDURE FOR IMPLEMENTING SOFT PURGE-- =============================================ALTER PROCEDURE [dbo].[SP_SOFTPURGE] @POID INTASBEGINUPDATE POMASTER SET FLAG=1 WHERE POID=@POIDSELECT * FROM POMASTER WHERE FLAG=0END
Procedure 2-----------set ANSI_NULLS ONset QUOTED_IDENTIFIER ONgo
-- =============================================-- Author: C.R.P. RAJAN-- Create date: March 26, 2008-- Description: Stored Procedure for Hard Purge-- =============================================ALTER PROCEDURE [dbo].[SP_HARDPURGE]  @PoId INTAS
BEGIN TRAN INSERT INTO ARCHIVE SELECT * FROM POMASTER WHERE POID = @POID   IF @@ERROR != 0   BEGIN      ROLLBACK TRAN      RETURN   END
   DELETE FROM POMASTER WHERE POID = @POID   IF @@ERROR != 0   BEGIN      ROLLBACK TRAN      RETURN   ENDCOMMIT TRANSELECT * FROM POMASTER

View 1 Replies View Related

Tempdb Is Locked And Blocks Other Users

Oct 18, 1999

When I am running reports on SQL Server 6.5 database which involves creation of temp tables, whole tempdb is locked. All other users are blocked,

The reports involve queries which fetch data into temporary tables, as,

SELECT c1 , c2, c3 ...
INTO #tmp1
FROM T1
WHERE C1 = xxx
AND .........

When any of these queries are run, it blocks all other operations using TEMPDB, like creation of temporary tables or queries using sorts etc.

It appears that the query locks the system tables (sysobjects, syscolumns) in TEMPDB.

However, the above query works fine if it is constructed as,

CREATE TABLE #tmp1 (a1 ..., a2 ...., a3 ..... )

INSERT INTO #tmp1
SELECT (c1, c2, c3)
FROM T1
WHERE c1 = xxx
AND .........

Could someone please tell me what actually is happening in the first scenario. Is there a way avoid the blocking of TEMPDB usage other than the workaround mentioned above (which means I have to change all my queries) ?

Thanks
Nishant

View 1 Replies View Related

Record Groups In Blocks Via A Query

Jun 11, 2007

I'm trying to place records into groups of 100, so I need a query that will return records but not based upon an autonumber type field. For instance, if I had a set of records that had been sequentially numbered (by autonumber) with IDs to greater than 3,600 but there were actually only 300 IDs remaining ranging from 1 to 3600 (say the others were deleted), then I would want to use a query to make three groups of 100 of the remaining IDs.

I can use SELECT TOP 100 for the first 100 records. How do I get subsequent groups of 100?

It's probably obvious and I am thinking too hard.

View 3 Replies View Related

One Ms Sql Server 2000 Instance Blocks The Other (thru A Dts-job)

Jul 23, 2005

Hello,we use two instances of the ms sql server 2000 (Version: 8.00.760) on a4 processor windows 2000 (sp 4) machine.A dts-packet on one instance completly blocks the work on the otherinstance. Both instances have 2 dedicated processors. The twoprocessors of the blocked instance are in idle states.How is this possible? Has someone had the same behavior of the sqlserver and found a solution for this problem?Thanks in advance for answersJürgen Simonsen

View 1 Replies View Related

SSIS Package BLOCKS At Opening.

Mar 21, 2006

Hallo,

I'am relatively new to SSIS-programming and experiencing a serious problem with a package (6618 KB large) containing 5 dataflows. These dataflows all start from a datareader-origin and flow trough multiple transformations. While editing the fifth dataflow the developmentenvironment suddenly became unworkably slow. After any modification made to the dataflow it takes several minutes before I can continue editing. Meanwhile in the task manager the process devenv takes a very large portion (50 to 100%) of CPU.

Building the package,shutting down SSIS and even rebooting the PC don't make any difference. When you close the package ands reopen it, this takes at least 10 minutes.

We work with Microsoft SQL Server Integration Services Designer
Version 9.00.1399.00.

Does anyone know if there is a limit to the number of transformations in a flow or the size of a package? Could that be the problem?

Before this problem occured I was multiplying transformations by copying ,renaming and altering them within a dataflow. Could SSIS have some problem with that?

Any suggestion for solving this problem, other than chucking it all away an restarting, will be gratefully accepted.



With kind regards,





Paul Baudouin



View 3 Replies View Related

Format Database Disk With 64K Blocks NTFS

Jul 14, 2004

Hi:

Somebody tellme that with a format of my database disk with 64K blocks NTFS, i can have better performance, is that true ? there is any problem in SQL with this block size ?

View 14 Replies View Related

Splitting Script Into Small Blocks And Using UNION ALL

Jun 9, 2014

We are working for a long time on optimizing this script, and latest modification I did was splitting script into small blocks and using UNION ALL. This supported a lot but still it takes some 45 mins on prod environment. I'm not able to find anything more in this script to optimize.

View 9 Replies View Related

SQL Server 2008 :: Notify Only If There Are Certain Blocks By Certain User

Jun 12, 2015

I'm trying to set the query to send email ONLY when it returns records of blocks and i cant seem to get this going.

declare @blocks varchar(max)
set @blocks = (SELECT spid,
sp.[status],
loginame [Login],
hostname,
blocked BlkBy,
sd.name DBName,

[Code] ....

View 1 Replies View Related

Reusable Blocks Of Code In Stored Procedures

Apr 15, 2008

I have several stored procedures with a similar query. The SELECT clauses are identical, but the FROM and WHERE clauses are different.

Since the SELECT clause is long and complicated (and often changes), I want to place the SELECT clause in a separate file and then have the stored procedures include the block of text for the SELECT clause when they compile.

Is this possible? Looking through the docs and searching online, I don't see any way to do this.

View 1 Replies View Related

Blocks Of Time - Log Number Of Visits From Clients

Oct 30, 2014

I have the following SQL that I want to log the number of visits from clients. a session is a block of time, for example, any time between 9-12 is a morning sessions, 12-5 is an afternoon session etc. I need to count the number of sessions in a month not the appointments within the session, make sense?

select sr.resourceid, st.TimeStart, datename(m,sr.scheduledate) as sesmonth, datename(yy,sr.scheduledate) as year,
(CASE
WHEN DATEPART(hour, st.TimeStart) BETWEEN 0 AND 12 THEN 'MORNING SESSION'
WHEN DATEPART(hour, st.TimeStart) BETWEEN 12 AND 17 THEN 'AFTERNOON SESSION'
WHEN DATEPART(hour, st.TimeStart) BETWEEN 17 AND 24 THEN 'EVENING SESSION'

[Code] ....

View 2 Replies View Related

Convert Repeating Blocks Of Columns Into Rows?

Feb 17, 2015

writing SQL code to convert blocks of columns into rows.

Example,

Id A1 A2 B1 B2
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3

into

Id Group Value
1 A 1
1 A 1
1 B 1
1 B 1
2 A 2
2 A 2
2 B 2
2 B 2

View 2 Replies View Related

Delete Statement Times Out And Blocks Reads

Aug 21, 2007

Hi. Periodically I need to run a delete statement that deletes old data. The problem is that this can timeout using ODBC (via the CDatabase and CRecordSet classes in legacy code). Also, while its running the delete, the table its operating on is locked and my application can't continue to run and operate on rows not affected by the delete.

Are there any workarounds for this? Can the timeout be set in the connect string?

Thanks,
Brian

View 1 Replies View Related

Data Application Blocks For Windows Mobile

Feb 5, 2006

I've read that microsoft.applicationblocks.data for .net v2 can't be deployed to a mobile app, and my experience bears that out. So I 'm wondering if there are application blocks for windows mobile 5 that would know how to both talk to sql server mobile and sql server 2005. I see OpenNetCF has a port but as far as I can tell, they only address sql server mobile and not talking to a sql server 2005 remote database. I can use the OpenNetCF version for my sql server mobile requirements, but I'm hoping there is an encapsulation of sqlclient calls for communication with my server db.

thanks

braden

View 3 Replies View Related

Problem With Nested Try Catch Blocks In SQL 2005

May 23, 2006

hi All,

Code:

begin try

begin transaction trans1

//some transaction

begin try

begin transaction trans2

//some transaction

//Raise Error

//commit transaction trans2

end try

begin catch

//Raise error

end catch
//some transaction

//commit transaction trans1

end try

begin catch

//Rollback trans1

end catch]



In the above code when an error is raised in the transaction trans2 the immediate catch is not invoked where as the outer catch is being invoked. I dont know y?. Can anybody help. Thanx











View 5 Replies View Related

SQL Server 2012 :: Display Data In Blocks (Column Name)

Jan 29, 2014

I want to display data in block. Below is the Create,Insert Script and format of desired output.

CREATE TABLE [dbo].[Test_AA](
[CounterpartID] [varchar](2000) NULL,
[CounterpartName] [varchar](2000) NULL,
[SATName] [varchar](2000) NULL,
[NameOfBO] [varchar](2000) NULL,

[Code] ....

Result Should be like This :

ID:17388
Name:Scottie Berwick
SATName:Scottie Berwick
NameOfBO:
SATAccountNumber:

[Code] ....

View 4 Replies View Related

Blocks Hangup Database When Inserting In 1 Specific Table

Jul 20, 2005

Hi,We use a database with about 40 related tables. Some tables contain asmuch as 30.000 records. We use Access97 as an interface to thedatabase. Now recently we have the problem that when we want to inserta row in one specific table (alwasy the same) the database makesblocks.Details:- about 10% of the data was inserted using copying from Excel, beforethis action there was no problem, though there is no evidence thatthis causes the problem.- inserting rows via the Query Analyzer works fine, via Access causestrouble.- the tempdb lofile has grown to 48Mb.Has anyone ideas about what is going on and what I can do to solve theproblem?TAV,Jan Willems

View 1 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved