Use Replication To Identify Updated Data?
May 15, 2006
I am new to SSIS (though have a decent amount of experience with SQL Server 2000), and am trying to design a data warehouse and eventually a comprehensive reporting solution.
Here is our setup:
- We have ~150 studios all operating software which we designed (ie. we have control over the system that will be the Data Warehouse source).
- All studios use merge replication back to Head Office (we maintain a duplicate of each of their databases).
From my limited understanding of Data Warehousing methodology, I believe that I should prepare a 'staging area' which contains all needed data from these databases. My question is how to do this in an efficient manner (without resorting to software-maintained timestamps in all the source tables - this would require significant modification to our software).
Since replication is already identifying which rows are new or updated, I'm wondering if there is some way to use this information to limit the information processed during the Data Warehouse updating process. I realize that I would still need to determine whether the row was updated or new, but it would cut down immensely on the number of rows processed.
I feel like I could be missing out on a simple way to do my data extraction from the source DBs. Does anyone have any advice?
Thanks in advance for your help.
View 8 Replies
ADVERTISEMENT
Mar 6, 2007
Hello,I'm getting the following error message when I try add a row using aStored Procedure."The identity range managed by replication is full and must be updatedby a replication agent".I read up on the subject and have tried the following solutionsaccording to MSDN without any luck.(http://support.Microsoft.com/kb/304706 )sp_adjustpublisheridentityrange (http://msdn2.microsoft.com/en-us/library/aa239401(SQL.80).aspx ) has no effectFor Testing:I've reloaded everything from scratch, created the pulications from byrunning the sql scripts generated,created replication snapshots andstarted the agents.I've checked the current Identity values in the Agent Table:DBCC CHECKIDENT ('Agent', NORESEED)Checking identity information: current identity value '18606', currentcolumn value '18606'.I check the Table to make sure there will be no conflicts with theprimary key:SELECT AgentID FROM Agent ORDER BY AgentID DESC18603 is the largest AgentID in the table.Using the Table Article Properties in the Publications PropertiesDialog, I can see values of:Range Size at Publisher: 100,000Range Size at Subscribers: 100New range @ percentage: 80In my mind this means that the Publisher will assign a new range whenthe Current Indentity value goes over 80,000?The Identity range for this table cannot be exhausted! I'm not surewhat to try next.Please! any insight will be of great help!Regards,Bm
View 1 Replies
View Related
Feb 20, 2015
Is there a script to find which non-clustered indices are replicated? I know i can do this easily through GUI , having a script will make my life much easier ....
View 0 Replies
View Related
Feb 18, 2006
Hi All
I am getting this error during Merge Replication "Unable to synchronize the row because the row was updated by a different process outside of replication."
I could not reproduce this error in my development enviornment, can anyone tell me what is cause of the above error and how do I reproduce this error in my development machine? I am able to get which is outside process?
The Conflict Type is 10 for above error which is logged into Conflict Table.
Thanks
View 11 Replies
View Related
Jul 23, 2005
How can you tell which datatypes in a given database are user defined(with a query, not by looking in Enterprise Manager)? This is for SQLServer 2000.
View 2 Replies
View Related
Dec 27, 2007
I have a query/report that I need to create that needs to look at the size of a company and based on that size apply different rules. I am sure that this is not the only query/report I'll need to do using this and I'm also not so sure that the size ranges won't be changed in the future. Given this, I'd like to store the size ranges in a lookup(global) table. That way, if the ranges ever change I can just alter them in that table and not in all of the queries/reports that use them. What I need to figure out is how to join the live table with the look up table.
Specifically, here is what I have. The look up table would be:
Code Block
CREATE TABLE #gl_sizerange (
glsid int IDENTITY(1,1) NOT NULL,
lowsize int,
highsize int,
sizecat varchar(10),
milestone varchar(25),
days int
) ON [PRIMARY]
INSERT into #gl_sizerange VALUES(1, 24, 'Small', 'Approach', 14)
INSERT into #gl_sizerange VALUES(1, 24, 'Small', 'Interview', 14)
INSERT into #gl_sizerange VALUES(1, 24, 'Small', 'Demonstrate', 21)
INSERT into #gl_sizerange VALUES(1, 24, 'Small', 'Negotiate', 14)
INSERT into #gl_sizerange VALUES(1, 24, 'Small', 'Close', 7)
INSERT into #gl_sizerange VALUES(25, 99, 'Medium', 'Approach', 14)
INSERT into #gl_sizerange VALUES(25, 99, 'Medium', 'Interview', 21)
INSERT into #gl_sizerange VALUES(25, 99, 'Medium', 'Demonstrate', 21)
INSERT into #gl_sizerange VALUES(25, 99, 'Medium', 'Negotiate', 14)
INSERT into #gl_sizerange VALUES(25, 99, 'Medium', 'Close', 7)
INSERT into #gl_sizerange VALUES(100, 499, 'Large', 'Approach', 14)
INSERT into #gl_sizerange VALUES(100, 499, 'Large', 'Interview', 21)
INSERT into #gl_sizerange VALUES(100, 499, 'Large', 'Demonstrate', 21)
INSERT into #gl_sizerange VALUES(100, 499, 'Large', 'Negotiate', 14)
INSERT into #gl_sizerange VALUES(100, 499, 'Large', 'Close', 7)
INSERT into #gl_sizerange VALUES(500, 300000000, 'Super', 'Approach', 28)
INSERT into #gl_sizerange VALUES(500, 300000000, 'Super', 'Interview', 35)
INSERT into #gl_sizerange VALUES(500, 300000000, 'Super', 'Demonstrate', 28)
INSERT into #gl_sizerange VALUES(500, 300000000, 'Super', 'Negotiate', 35)
INSERT into #gl_sizerange VALUES(500, 300000000, 'Super', 'Close', 35)
Then what I have is two more tables, one that indicates the size of the company and another that has the milestone contained within it and I will also calculate how long that milestone has been open and if it is longer than what is in the lookup table for that milestone and size range I need it returned in the report. Here are some quick sample table to represent that data (I've condensed the size and number of tables for the example):
Code Block
CREATE TABLE #en_entity (
enid int NOT NULL,
orgsize int,
) ON [PRIMARY]
INSERT into #en_entity VALUES(1, 5)
INSERT into #en_entity VALUES(2, 18)
INSERT into #en_entity VALUES(3, 24)
INSERT into #en_entity VALUES(4, 25)
INSERT into #en_entity VALUES(5, 47)
INSERT into #en_entity VALUES(6, 101)
INSERT into #en_entity VALUES(7, 499)
INSERT into #en_entity VALUES(8, 500)
INSERT into #en_entity VALUES(9, 10000)
INSERT into #en_entity VALUES(10, 567890)
CREATE TABLE #op_opportunity (
opid int NOT NULL,
enid int NOT NULL,
milestone varchar(25),
daysopen int
) ON [PRIMARY]
INSERT into #op_opportunity VALUES(1, 1, 'Approach', 5)
INSERT into #op_opportunity VALUES(2, 2, 'Interview', 18)
INSERT into #op_opportunity VALUES(3, 4, 'Negotiate', 24)
INSERT into #op_opportunity VALUES(4, 7, 'Demonstrate', 25)
INSERT into #op_opportunity VALUES(5, 7, 'Approach', 7)
INSERT into #op_opportunity VALUES(6, 9, 'Close', 35)
INSERT into #op_opportunity VALUES(7, 8, 'Close', 36)
So, given the sample data, I would expect the results to return me the following opids from the #op_opportunity table because they don't comply with what is in the look up table based on milestone, size and days open: 2,3,4,7
View 4 Replies
View Related
Mar 3, 2015
I am in process to develop TSql code to identify change in data.
I read about Binary_checksum and hashbyte. Some people say hashbyte is better than binay_checksum as chances of collision are less.
But if we may consider following, chances exist in hashbyte too. My question is what is the best way to compare data to identify change (I can't configure CDC) ?
select HASHBYTES('SHA','121'+'34'), HASHBYTES('SHA','12'+'134'),BINARY_CHECKSUM('121','34'),BINARY_CHECKSUM('12','134');
View 2 Replies
View Related
Dec 29, 2014
how to identify the data leakage in a database , as I heard in one of my environment?
what is the meaning for data leakage ?
View 3 Replies
View Related
Jan 21, 2015
I was running an operation to shrink file/emptyfile a data file, and then remove it.
It blocked and caused a huge mess, I suspect on the removal part. But I want to confirm that the emptyfile completed (and that the engine isn't going to try to put more data in there for when I schedule the removal part again a week or more from now).
How does the engine know not to put any more data in there, and how long does that situation last?
View 3 Replies
View Related
Jun 21, 2005
Good morning,
View 7 Replies
View Related
Jun 18, 2014
How to identify different fields with in a group of records?
Example:
create table #test
(ID int, Text varchar(10))
insert into #test
select 1, 'ab'
union all
select 1, 'ab'
[Code] ...
I want to show additional field as Matched as ID 1 has same Text field on both the records, and for the ID 2 I want to show Unmatched as the Text fields are different but with the same ID.
View 1 Replies
View Related
Jan 11, 2007
Hi,
Can anybody pls help me with this,
i have to migrate data from one sql server(1) to another sql server(2) using SSIS,Which is a bit easy,the concern here is that after doing it the data in server(1) is getting updated due to inserts,now i need to bring the updated data alone.Is anybody having experience in this?can anyone pls help me or give me sugessions on how i need to proceed.
Thanks in advance.
Regards,
Sg
View 5 Replies
View Related
Oct 20, 2011
How do i check the size of the datacache allocated from the buffer pool by sql server?
DMV or anything to show me the pool allocation sizes for the various pools in sql server i think i may be able to work from there.
View 9 Replies
View Related
Apr 20, 2015
I have a single data driven subscription that is a batch of 250 subscriptions running on SQL Server 2008 R2. All the parameter values for the report are sourced from a database table. During the execution of the data driven subscription 5 of those subscriptions failed due to timeout error.
Final Status as shown on Subscription screen "Done: 250 processed of 250 total; 5 errors."
The SSRS log file has the subscription GUID which is a single record in the reporting server database tables. How can i identify the individual failed subscriptions? Is there an easy way or a work around that i should implement to identify the failed subscriptions?
View 5 Replies
View Related
Jan 4, 2006
Hello, I need some guidance in the best method to accomplish this task. I have a network with a SBS 2003 server and a SQL 2000 member server in the SBS domain. I have a remotely hosted website at discountasp.net with a SQL 2000 database. The website will host a modified e-commerce kit where corporate clients can order parts. I need to keep the products table on the remotely hosted site as updated as possible. The website orders are placed via email from the commerce kit and the fulfillment department proccesses the order against the SQL server in the SBS domain.
Any ideas on how to keep the data updated on the website?
View 4 Replies
View Related
Jul 23, 2005
I do not know SQL but learning fast and furious.I am programming an agent and working with a group of existing databases.I would like to able to compare the database before and after an update.The testing databases are relatively small.I have no problem programming some compare but how do I go about it.Should I do this in SQL duplicating the database.I would be happy to write some SQL and dump the databases and do the compareexternally.I would appreciate any suggestion.Andre
View 4 Replies
View Related
Feb 13, 2015
I have a SSAS stored procedure with a signature:
public Set DoSomthing(Set toBeProcessed, Set measuresToWorkWith)The set measurseToWorkWith is passed as {[Measures].[Measure1], [Measures].[Measure2] ...}
with the measures being real or query-scoped calculated members.
To get the value of the measure for each tuple in the set toBeProcessed, I create an Expression for each tuple (measure) in the set measuresToWorkWith then for each tuple in toBeProcessed call expression.Calculate(tuple) which returns a MDXValue.
My problem is that in order to make the code generic I need to get the real (.NET) data type of the MDXValue. The class only has explicit conversion methods ToInt16() etc which implies that the data type is known at design time.
However, if one of the measures is a query-scoped calculation then it could return a .NET double, int, bool or string.
If the measure is real then I can look up its metadata. However, it appears that if it is a formula (scoped member) then are all bets are off?
View 0 Replies
View Related
Apr 6, 2008
Hi,I want to save the last modification date when the row is updated. I have a column called "LastModification" in the table, every time the row is update I want to set the value of this column to the current date. So far all I know is that I need to use a trigger and the GetDate() function, but could any body help me with how to set the value of the column to getdate()? thanks for your help.
View 3 Replies
View Related
Mar 22, 2004
Client issues query which sends out individual requests to the 2 OLAP servers that are load balanced. The client evaluates the versions of the returned record sets to ensure the consist data being returned for one single query. Otherwise, this error will be seen:
The cube has been updated by the server the data is now obsolete.
These servers sit behind a cisco 11506 CSS with load balancing based on balance type: least busy server, also persistence based on cookies.
My developer says this worked fine for a long time then just 'started happening'.
any suggestions are appreciated.
View 2 Replies
View Related
Mar 24, 2007
Dear friends, We have a server put at US side and we are trying to load the data into another related database in India site, now the thing is that we want to update India data at an interval of 5 mins?
Is there any way to do it without effecting the performance of the US server?
thanks,
View 9 Replies
View Related
Dec 12, 2006
My package sets a variable in an ExecSQL task. This variable is then used as a parameter in a DataFlow task that follows. Normally everything works just fine. However, sometimes if the package fails inbetween the step that sets the variable and the DataFlow, the default value of the variable is stored in the checkpoint file. (It is not failing in the step that sets the variable) When the package is restarted I can see that the variable is not set to the data value in the database, but rather it has its default (design time) value.
Anyone else see this bug?
View 2 Replies
View Related
Nov 27, 2007
I am sending a GUID to a form via the query string. If it exists I use helper functions to load most of the form text boxes. However, if it does not then a blank form is presented and the GUID is stored in a hidden field.
Regardless, I use this hidden field to populate a grid that is attached to a sqldatasource.
If I then add new datarows to the backend database programmatically, I cannot 'requery' the datasource to include those row upon a postback. I cannot seem to find a simple way to force the sqldatasource to rerun the query.
Can anyone help.
View 2 Replies
View Related
Dec 11, 2006
Trigger not running when trying to select data from a column that is being updated?
View 1 Replies
View Related
Dec 14, 2007
hi,
I created a SSIS package. It imports data from a flat file and then transfer to different data types and load it into destination table. I use look up transformation. Actually before I created final table, I created another intermediate data table for references.
Now I get a new source file once in a month. Then I'm supposed to connect new file and run the package. only difference in new source file is the data not data type. But when I connect the new flat file, package does not work. first one and fifth one are red when I run the package.
Can anyone help me to fix this?
Thanks
p/s;
I get the following error messages in execution result page.
[Lookup 5 [541]] Error: Row yielded no match during lookup.
[Lookup 5 [541]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Lookup 5" (541)" failed because error code 0xC020901E occurred, and the error row disposition on "output "Lookup Output" (543)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
[DTS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Lookup 5" (541) failed with error code 0xC0209029. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
[DTS.Pipeline] Error: SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread0" has exited with error code 0xC0209029. There may be error messages posted before this with more information on why the thread has exited.
[Flat File Source [1]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.
[DTS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Flat File Source" (1) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
[DTS.Pipeline] Error: SSIS Error Code DTS_E_THREADFAILED. Thread "SourceThread0" has exited with error code 0xC0047038. There may be error messages posted before this with more information on why the thread has exited.
View 1 Replies
View Related
Dec 17, 2007
Hi
We are building a near real time application in Windows Forms, that stores data in SQL Compact Edition.
The client receives messages via UDP socket and performs insert / update operations to the db accordingly, and the requirement is to update a data grid with the changes.
We tried querying the db and receive a ResultsSet which is sensitive and scrollable, and bind a ResultSetView to the Grid. But, since the updates are made from another thread, the grid is not being refreshed.
What is the best approach for achieving this?
Thanks in advance,
Guy Burstein
http://blogs.microsoft.co.il/blogs/bursteg
View 3 Replies
View Related
Dec 11, 2012
I have a table thats updated daily with monthly data totals:
Month, Total orders1, Total orders2, etc
12/01/2012, 5, 8, etc
11/01/2012, 6, 5, etc
How do I pull data from this table in SQL Server for ONLY the current month? I was thinking using the getdate() function to get the current month, but it doesn't match exactly so I get no results
View 7 Replies
View Related
Apr 21, 2015
I need the rows updated today in catalogue table irrelevant of timing. I tried all the below queries.
select * from CATALOGUE where CAT_DATE=CONVERT(datetime, CONVERT(varchar, GETDATE(), 101))
select * from CATALOGUE where CAT_DATE= CONVERT(date, getdate())
select * from CATALOGUE where CAT_DATE= cast(GETDATE() as date)
I am getting output only if the information updated on 04/21/2015 00.00.00.000 but not for other timings, For example
04/21/2015 03.30.00.000, 04/21/2015 07.17.00.000 and all.
How can I retrieve all the records updated today.
View 4 Replies
View Related
Jun 7, 2007
The only way the job success is if I select the option add rows wich is copying all rows even the ones that are already there, I tried to drop amd recreate the table option but i just dosn't run that way, help please..
View 1 Replies
View Related
Feb 1, 2012
I am task with identifying the source database name, id, and server name for each staging table that I create. I need to add this to a derived column on all staging tables created from merging same tables on different servers together.
When doing a Merge Join, there is no way to identify the source of data so I would like to see if data came from one database more than the other servers or if their are duplicates across servers.
The thing that bugs me about SSIS Data Flow task is there is no way to do an easy Execute SQL Task after I select my ADO.NET Source to get this information because my connection string is dynamic and there is no way of know which data source is being picked up at runtime.
For Example I have Products table on Server 1 and 2:
Server 2 has more Products and would like to join the two together to create a staging table.
I want see the following:
Product ID, Product Name, Qty, Src_DB_ID, Src_DB_Name, Src_Server_Name
1 IPAD 1000 2, MyDB1, Server1
100 ASUS Pad 40 1, YourDB, Server2
get database name and server name in DATA FLOW only (without using a for each in Control Flow)
View 5 Replies
View Related
Jul 28, 2006
Hi,
I have a VB.net app that access a SQL Express database. I have transactional repliaction set up on a SQL 2000 database (the publisher) and a pull subscription from the VB.net app. I use RMO in the VB app to connect to the publisher. My problem is I am getting some strange behaviour as follows
- if I run the app and invoke the pull subscription it works fine. If I then close my app and go back in, I can access my data without any problem
- If I run the app and try to access data in my SQL Express database it works fine. I can then close the app, reopen it and run the pull subscription it works fine
however.......
- if I run the app, invoke the pull subscription (which runs fine), and then try to access data in my local SQL Express database without firstly closing and reopening the app, I get a login error
- if I run the app, try to access data in my local SQL Express database (which works fine), and then try to run the pull subscription I get a "the process cannot acces the file as it is being used by another process" error. In this case I need to restart the SQL Express service to be able to run replication again.
I get exactly the same behaviour when I use the Windows Sync tool (with my app open at the same time) instead of my RMO code to replicate the data.
I am using standard ADO.Net 2 code to access my SQL Express data in the app and closing all connections etc
Any advice appreciated !
Thanks
Ronan
View 2 Replies
View Related
Sep 3, 2007
Hi There
This one has bothered me ever since sql server 2000.
When you do an insert into a table with literally hundreds of char or varchar columns and you get the error that the insert failed due to data loss/truncation on a column.
Is there anyway in 2005 to actually find out what column ? Since there are hundreds is is literally a long process of going though each column 1 by 1 manually.
The database engine surely MUST know what column this occurred on so why can it not tell you which column the truncation occurred on ?
Can this be done in 2005 if not will this information be available in 2008 ?
Thanx
View 7 Replies
View Related
Nov 17, 2006
Hi,
I am trying to setup Trans Replication with updating subscriber on sql2000. One column on few tables got data with single quote (').
How do I handle in this case? Did any one come across such case?
Can I Change default QUOTED IDENTIFIER from ' (single quote) to something else (@@@) on SQL2000?
If yes, how to do?
Thanks
mka
View 1 Replies
View Related
Apr 17, 2008
Hi,
How would I know that a dll component I built is in .NET framework 1.0 or 1.1??
annej
View 4 Replies
View Related
