I have a question regarding a project I am working on and I'm not sure what the best way to do it may be, I am new to SSIS and would like to gain some exposure using it, but don't know if this project is overkill or even possible using SSIS.
*** PROBLEM ****
Basically I have a large number of flat files each file contains different columns and data. What I want to do is import all of the data from the flat files into corresponding tables in SQL Server. The tables in SQL server are already defined with data structure to match each file. The first 10 characters of the flat file are also the same name as the SQL table. So it seems easy enough to just import flat file to table one by one, but I cannot figure out how to match the flat file structure to each table without setting up a task for each table (there are roughly 50-60 tables). Another reason I want to do it with SSIS is these flat files have to be imported on a weekly basis.
**********************
Is it in my best interest to just create a stored procedure that basically loops through the files and imports the data? Or do you think using SSIS is the best solution? I have looked at SSIS a bit and thought maybe I could use the bulkinsert task, but it appears it cannot dynamically change due to column specs. I figured out how to loop through the flat files using the foreach task, but I could only get it to import into one SQL table. I also started looking at the dataflow task and thought about creating a script task, but I'm not sure if that would work before I spend a lot of time trying to figure that out.
A client of mine, a group of 4 doctors, are interested in implementing a small office network in which they will switch to a digital filing system. I have recommended a Microsoft Server 2003 system, therefore the patient information will be stored in the SQL server.
I have never administered, administrated?? (see, I really need help here!) a database, but have built small databases from scratch in C and with Access, so I feel confident that I could set up a system for them. I need to have some kind of estimate, both a fee and implementation schedule. Can you help me get a handle on the scope of this?
I have a little problem that I think I solved with a sledge-hammer andthought I'd post here to see if there was not perhaps a more elegantand probably quicker solution.Essentially I have a unique display condition on a table that mustreturn data in a certain way. The problem column in question is thesecond col of a compound PK which thus makes the data in this columnunique for any given col1 value. The problem is that regardles of whatdata is in col2, if one of the values is 'NO' ,then this MUST appearat the top of any resultset, query, or report that can be generated,otherwise standard alpha sort is fine (it is a varchar(30) field)My inelegant solution was to create a third column in the table, achar(1) that can only ever have 2 values, a '1' or a '0'. So allentries for col2 which are 'NO' get assigned the value 1, all othersare 0 so when you sort using this extra column based on any col1value, 'NO' always gets to go on top.Can I lose the extra column which is there purely for sorting/logicand thus is irrelevant to the table data or its meaning(<hangs head indue deference to normalisation>) It obviously works OK the way it is,but it makes all queries/sprocs cumbersome because they always havethe overhead of sorting an additioal column, or an inefficient indexwhere nearly all the values in a vital column are always '0' except teodd few, also all the updates and inserts need to be wary of changesor additions that are make a 'NO' value for col2 and thus reset or setthe sort column to a '1'.Some sample DDL and values below.CREATE TABLE [dbo].[tmpTest]([colA] [varchar](30) COLLATE Latin1_General_CI_AS NOT NULL,[colB] [varchar](30) COLLATE Latin1_General_CI_AS NOT NULL,[colN] [varchar](50) COLLATE Latin1_General_CI_AS NULL CONSTRAINT[DF__1] DEFAULT ('UNK'),[Reference] [char](1) COLLATE Latin1_General_CI_AS NULL DEFAULT('0'),CONSTRAINT [PK_tmpText]PRIMARY KEY CLUSTERED ([colA] asc , [colB] asc))ON [PRIMARY]insert into tmpTest (colA,colB,colN) values ('1','a','z')insert into tmpTest (colA,colB,colN) values ('1','b','z')insert into tmpTest (colA,colB,colN) values ('1','c','z')insert into tmpTest (colA,colB,colN,reference) values('2','NO','z','1')insert into tmpTest (colA,colB,colN) values ('2','a','z')insert into tmpTest (colA,colB,colN) values ('2','b','z')insert into tmpTest (colA,colB,colN) values ('2','c','z')insert into tmpTest (colA,colB,colN) values ('2','d','z')select * from tmpTestwhere colA='2'order by colA,reference desc,colB
I need to estimate the effort required in writing some SSIS packages. Could anyone provide some pointers to the various factors (E.g. number of tables, source, etc.) which influence the effort estimate for SSIS packages?
Is MCDBA worth of the time and money one has to put into to acheive it? I am talking in terms of getting a job. Do you think after getting MCDBA you have gained sustantial knowledge about the subject?
I havent appeared for any of the MS exams as yet but would like to start with 70 228 and 70 229.
I like reading BOL, but i guess for certfn exams you have to have a diff approach...
I currently have several databases running under SQL 7. My Shop is migrating to Windows 2000. I was wondering, does SQL 7 run as good under Win 2000 like it does under Win NT? Since we are migrating should I upgrade to SQL 2000 or should I stay with SQL 7?
Hi,I have a sql 2000 server with 8 processors, server settings are asdefault. I read on Technet that it is good practise to remove thehighest no. processors from being used for parallelism, correspondingto the no. of NICs in the server. One of our 3rd party developers hasrecommended only allowing one processor to be used as there is aperformance hit by the server working out which processor to use. Doesanyone have a definitive answer to this? I suspect he's wrong but I'dlike some hard evidence if possible, thanks.Kev.
Very often during dev. I add processes that don't change the metadata of a data item (for example a new sort) yet the linieage id is changed. This causes lots of headaches down the pipe. If metadata of a data item is not changed in the pipe, why is the lineage id changed?
If lineage is not smart enough to figure out when some metadata was actually changed, should it be made smart enough? Or optional?
Hi,I am trying to write a method which needs to call a stored procedure and then needs to get the response of the stored procedure back to the variable i declared in the method. private string GetFromCode(string strWebVersionFromCode, string strWebVersionString) { //call stored procedure } strWebVersionFromCode = GetFromCode(strFromCode, "web_version"); // is the var which will store the response.how should I do this?Please assist.
I'm trying to set up a backup schedule in SQL 2005 that will keep 5 .bak files within the folder I specify. I can get it to keep one backup only even though I specify that it is to keep 5 days worth of backups. Each backup happens at 12:00 am. I think the problem I may be having is I don't know how to get the maintenance plan to name the file mmddyyyy.bak or whatever it may need to determine that the file is 5 days old. Currently I am specifying a name of gmsqlbackup.bak. Any idea's that may help me out?
Hi, I just have a Dataset with my tables and thats it I have a grid view with several datas on it no problem to get the data or insert but as soon as I try to delete or update some records the local machine through the same error Unable to find nongeneric method... I've try to create an Update query into my table adapters but still not working with this one Also, try to remove the original_{0} and got the same error... Please help if anyone has a solution
I'm not sure if this is the correct forum for this this question but I'll give it a shot.
The only db development that I have ever done is in MS Access. I have a project at work that is being accomplished in VB and I need a db engine to use as the back end. Visual studio came with a copy of MSDE. Is this tool worth using or should I invest in mySQL? Are there any advantages to using MSDE over mySQL?
is this the best and fastest method to connect to a database for DataGrid databind?
--------- Dim MyConnection As SqlConnection = New SqlConnection(ConfigurationSettings.AppSettings("ConnectionStringSQL")) Dim MyCommand As SqlCommand = New SqlCommand("sp_BuddiesPendingSelect1", MyConnection) MyCommand.CommandType = CommandType.StoredProcedure MyCommand.Parameters.Add(New SqlParameter("@UserID", intUserID))
MyConnection.Open() Dim dr As SqlDataReader = MyCommand.ExecuteReader()
DataList1.DataSource = dr DataList1.DataBind()
dr.Close() MyConnection.Close()
----------------------- or should i use a DataAdapter and fill a DataSet and then attach the DataSet to the DataGrid?
I have been using VB6 for a long time now and had no problems using ADODB.Recordset.I had a module to which I would send my recordset (byref) and the SQL command (byval) and use the resultant recordset for adding, modifying or deleting records.How can I do the same with VS2005? This is a major problem as there are dataadapters, grids, datareaders etc.Does someone have a simple method to get the recordset so that I can modify the record using the code (eg: .Addnew/.Delete/.Update) and Close the connection?Please note that I do not need to display anything and have to run 40/50 AddModDel for every click of the program.Any help is greately appreciated. Thanks in Advance.
If you not familiar with it, the <a href="http://www.stephenforte.net/owdasblog/permalink.aspx?guid=2b0532fc-4318-4ac0-a405-15d6d813eeb8">Rozenshtein Method</a> uses SQL to create a crosstab. The concept is a stroke of genius but I'm having trouble getting it to work on one of production databases.
I successfully used the Northwind example explained at Stephen Forte's site (see the link above)...but no luck on my real world problems.
I can get the date statements to resolve to 0 correctly, but when I try to aggregate the data - the statements are turning into 1 multiplying the aggregate data for each cell...which fills in the same data across the entire row.
For example (the columns represent the time period), GROUP T1 T2 T3 T4 group1 9 9 9 9 group2 3 3 3 3 group3 5 5 5 5
My sql code is: SELECT dbo.tblHassBatch.ProdLine, COUNT((dbo.tblHassUUT.UnitID)*(1-ABS(SIGN(datediff(dd,dbo.tblHassBatch.StartTime,ge tdate())-0)))) AS Today, COUNT((dbo.tblHassUUT.UnitID)*(1-ABS(SIGN(datediff(ww,dbo.tblHassBatch.StartTime,ge tdate())-0)))) AS [This Week], COUNT((dbo.tblHassUUT.UnitID)*(1-ABS(SIGN(datediff(mm,dbo.tblHassBatch.StartTime,ge tdate())-0)))) AS [This Month], COUNT((dbo.tblHassUUT.UnitID)*(1-ABS(SIGN(datediff(mm,dbo.tblHassBatch.StartTime,ge tdate())-1)))) AS [Last Month] FROM dbo.tblHassBatch INNER JOIN dbo.tblHassUUT ON dbo.tblHassBatch.BatchID = dbo.tblHassUUT.BatchID GROUP BY dbo.tblHassBatch.ProdLine
Am I overlooking something here? I'm pulling my hair out b/c if this works it's really going to provide a great solution for a project I'm working on...but I can't seem to figure it out.
I'm not really strong in SQL. My goal is to compare the beginning mileage of a vehicle record with it's previous ending mileage reading. I have something that works, but it feels clunky. I wonder if there is a better method, ie a join. Here's what I have:
SELECT A.Trolley_num, A.Date, A.Speedo_start, A.Speedo_end, (SELECT B.Speedo_end FROM Daily_Trolley AS B WHERE B.Trolley_num = A.Trolley_num AND B.Date = (SELECT Max(Date) FROM Daily_Trolley AS C WHERE C.Trolley_num = A.Trolley_num And C.Date < '1/23/2005')) AS PrevSpeedoEnd FROM Daily_Trolley AS A WHERE A.Date='1/23/2005'
ps: I inherited this db; I'm aware that "Date" should not have been used as a field name.
'TOP 1' or 'DISTINCT' or 'MAX' Any sugestions on which is better to use if I need to select a record that has the highest value - could be a INT or sometimes a DATETIME.
Can someone please tell me how to fix this following error in SQL 2005 When trying to create a maintenance plan, Method Not Found:'Void Microsoft.SqlServer.Management.DatabaseMaintenance.TaskUIYtils..ctro()' (Mircosoft.SqlServer.MaintenancePlanTasksUI)
Is there an efficient scripting method to update the connection string for ALL reports that reside on a reporting/web server? "(automating the process, rather than having to change the data source for each individual report that resides on that server)".
I have never used replication before and I have been asked to considerit in a project I am currently working on.I have created an application for a sales team which is loaded on theirmachines, it uses ms sql as its data source and connects via theinternet back to the central server in the office.Problem is this has shown to be too slow causing time out errormessages and so on. I have been told to research the possibility ofreplication, but am unclear what type of replication to use or where tostart.Any assistance would be appreciated.Regards,Ben
Hi,Given 2 tables:Table1:id Auto,int,PrimKeytable2id inttxt nvarchar50Table2:id Auto,int,PrimKeyorder intThe scenario: Table2.order defines how the table1.txt is should be ordered.Table1.table2id contains the id of the order. This cannot be changed :(How do I select all Table1.txt values but ordered by their correspondingvalues of the table2.order field?--Thx,PipHans---Outgoing mail is certified Virus Free.Checked by AVG anti-virus system (http://www.grisoft.com).Version: 6.0.516 / Virus Database: 313 - Release Date: 01-09-2003
HiSQL Server 7.0 using stored procedures.I need to do a SELECT Top but I always need to find the record countfor the entire SELECT statement.ID Name-- ----1 Abraham20 Barrington32 Callaway54 Dennis58 EmmettIf I do a SELECT TOP 3, I'll get the required records but get arecordcount of 3.I cant use the ROWCOUNT method either becuase the recordcount willalso be 3.I can see two different options for getting back the information Ineed:1 - Run the select statement twice, first to retrieve the recordcountand then with top to get the recordset2 - Use a temp table to get the entire recordset and recordcount, andthen extract the Top N recordsI'd like to know if anyone has any other suggestions. Currently, I'mtempted to go for the second option....ThanksSam
I want to implement the following query using SSIS Data flow Source component :
SELECT * FROM someTable WHERE someColumn = 'H'
How do I restrict the data coming from my data source? By that I mean how do I apply the WHERE clause in SSIS?! Should I use a conditional Split component?! but that would mean retrieving all records first then adding the split component (not the most efficient method surely).
I need to add a value to a column that I know to be DT_CY. However, there is no PipelineBuffer.SetCurrency() method or anything similar.
I checked out the docs for PipelineBuffer.SetDecimal() and it states "If the DataType of the buffer column is not DT_NUMERIC, an UnsupportedBufferDataTypeException occurs"
Similarly for PipelineBuffer.SetDouble() it says "If the DataType of the buffer column is not DT_R8, an UnsupportedBufferDataTypeException occurs"
So it seems I can't use either of those two methods. How do I push a valueto an output column of DT_CY when building an async component?
By the way, I could ask the same about
DT_DBDATE
DT_DBTIME
DT_FILETIME
DT_IMAGE
DT_TEXT
DT_NTEXT
DT_R4
DT_UI1 as well because according to the documentation there doesn't seem to be a method that supports those either. So, please could you answer the same question for those as well!
I have 1 project in VC++ and 1 in Vb.net I have added the VC++ project in VB.Net project using 'Add Existing item option' Now i want to use a method from VC++ project in VB.net How can i do so?????
Hi, I want to learn what is going on Example: Production Scheduling at http://msdn.microsoft.com/sql/learning/prog/clr/default.aspx?pull=/library/en-us/dnsql90/html/sqlclrguidance.asp
First, please interview Production Scheduling example. I'm curioing about SendResultsEnd method.
foreach (Item item in items) { // Compute the schedule and report it item.ComputeSchedule(); item.OutputSchedule(pipe); }
pipe.SendResultsStart(record); for (int i = 0; i < size; i++) { // Pipe each day of production. Omit zero production // days. if (schedule != 0) { record.SetDateTime(1, (DateTime)(dates)); record.SetInt32(2, schedule); pipe.SendResultsRow(record); } } pipe.SendResultsEnd();
Second code block is a part of OutputSchedule method. When pipe.SendResultsEnd method executed. First code block moves to next item. If SendResultsEnd method sends a resultset to client. This code will send many of resultset to client how many items array has in. If it is sending many resultsets. How can client can catch them? I was confused about this situation. And I don't have SQL Server 2005 in my computer. If you test it and explain what is going on I'll be appreciate to you. Please review example in the link.
I am developing an application in which i have to send data from local Sql Server compact edition database[Which is in a Windows Mobile Device,] to central server[SQL Server 2005]. I am using RDA method for communication
Can i use push method to send data from local DB to Central DB?
Is it must to use PULL method before using PUSH method?
