Capture Data From Excel File And Load Into Staging Table - Annual Count

Jul 31, 2013

I am working on an HR project and I have one final component that I am stuck on.

I have an Excel File that is loaded into a folder every month.

I have built a package that captures the data from the excel file and loads it into a staging table (transforming a few bits of data).

I then combine it with another table in a view.

I have another package that loads that view into a Master table and I have added a Slowly Changing Dimension so that it only updates what has been changed. (it’s a table of all employees, positions, hire dates, term dates etc).

Our HR wants to have this data in a report (with charts and tables) and they wanted it to be in a familiar format. So I made a data connection with Excel loading the data into a series of pivot tables.

I have one final component that i cant seem to figure out. At the end of every year I need to capture a count of all Active Employees and all Termed employees for that year. Just a count.

So the data will look like this.

|Year|HistoricalHC|NumbTermedEmp|
|2010|447 |57 |
|2011|419 |67 |
|2012|420 |51 |

The data is in one table labeled [EEMaster]. To test the count I have the following.

SELECT COUNT([PersNo]) AS HistoricalHC
FROM [dbo].[EEMaster]
WHERE [ChangeStatus] = 'Current' AND [EmpStatusName] = 'Active'

this returns the HistoricalHC for 2013 as 418.

SELECT COUNT([PersNo]) AS NumbOfTermEE
FROM [dbo].[EEMaster]
WHERE [ChangeStatus] = 'Current' AND [EmpStatusName] = 'Withdrawn' AND [TermYear] = '2013'

This returns the Number of Termed employees for 2013 as 42.

I have created a table to report from called [dbo.TORateFY] that I have manually entered previous years data into.

|Year|HistoricalHC|NumbTermedEmp|
|2010|447 |57 |
|2011|419 |67 |
|2012|420 |51 |

I need a script (or possibly a couple of scripts) that will add the numbers every year with the year that the data came from.

(so on Dec 31st this package will run and add |2013|418|42| to the next row, and so on.

View 20 Replies


ADVERTISEMENT

Store Procedure To Load Data From Flat File To Staging Table Dynamically - Column Metadata

Apr 9, 2015

I am having one store procedure which use to load data from flat file to staging table dynamically.

Everything is working fine.staging_temp table have single column. All the data stored in that single column. below is the sample row.

AB¯ALBERTA ¯93¯AI
AI¯ALBERTA INDIRECT ¯94¯AI
AL¯ALABAMA ¯30¯

After the staging_temp data gets inserted into main table.my probelm is to handle such a file where number of columns are more than the actual table.

If you see the sample rows there are 4 column separated by "¯".but actual I am having only 3 columns in my main table.so how can I get only first 3 column from the satging_temp table.

Output should be like below.

AB¯ALBERTA ¯93
AI¯ALBERTA INDIRECT ¯94
AL¯ALABAMA ¯30

How to achieve above scenario...

View 1 Replies View Related

SQL 2012 :: Using Excel In SSIS To Import Data From Spreadsheet To Staging Table?

Feb 5, 2015

I'm trying to use Excel in SSIS to import the data from spreadsheet to a staging table. The package runs well from the web server using SSMS. But when I deploy and try to execute the package, I'm getting the below error. I've a question, whether I've to install the AccessDatabaseEngine driver in SQL database server or the web server where I'm executing the SSIS?

Error: The requested OLE DB provider Microsoft.Jet.OLEDB.4.0 is not registered. If the 64-bit driver is not installed, run the package in 32-bit mode.

View 3 Replies View Related

SQL 2012 :: Load Consolidated Members Into MDS Staging Table - Hierarchy Name

May 28, 2015

I'm attempting to load some data into an explicit hierarchy in MDS 2012 via the staging table and struggling with the HierarchyName field. Specifically I'm loading data into stg.[Entity Name]_Consolidated and using the exact name of the explicit hierarchy I've set up in the front end web application.

Originally my hierarchy was labelled "Reporting Hierarchy" and when loading the data into staging using this name then running the batch from the Import Data screen I can see the error message "Error - The HierarchyName is missing or is not valid.". I've checked the table mdm.tblHierarchy and can see that the name there is exactly as it was in the staging table and have since renamed the hierarchy as "Reporting_Hierarchy" with the same results.

View 0 Replies View Related

SQL Server 2012 :: How To Create Staging Table To Handle Incremental Load

Jan 2, 2014

We are designing a Staging layer to handle incremental load. I want to start with a simple scenario to design the staging.

In the source database There are two tables ex, tbl_Department, tbl_Employee. Both this table is loading a single table at destination database ex, tbl_EmployeRecord.

The query which is loading tbl_EmployeRecord is, SELECT EMPID,EMPNAME,DEPTNAME FROM tbl_Department D INNER JOIN tbl_Employee E ON D.DEPARTMENTID=E.DEPARTMENTID.

Now, we need to identify incremental load in tbl_Department, tbl_Employee and store it in staging and load only the incremental load to the destination.

The columns of the tables are,

tbl_Department : DEPARTMENTID,DEPTNAME

tbl_Employee : EMPID,EMPNAME,DEPARTMENTID

tbl_EmployeRecord : EMPID,EMPNAME,DEPTNAME

How to design the staging for this to handle Insert, Update and Delete.

View 9 Replies View Related

Integration Services :: How To Load Data Without Changing Excel File (CSV Format)

Jul 14, 2015

Facing problem while loading date in MS SQL Server 2005 from excel file (csv format).

How to load the excel file data without changing the excel file (csv format) .

see the [Start Date] and [ Exp End Date] having values like this : " 2015/07/31"

View 5 Replies View Related

Integration Services :: How To Load Multiple Tables Data Into Single Excel File

Aug 26, 2015

My Requirement ,In Source Database 5 tables are there ( Emp,Loc,dept,Time,Product ), Destination is Single Excel file.But Dynamically how to load each table information to load into each sheet wise through SSIS Package?

View 3 Replies View Related

SSIS Load Data - Capture Error

Jul 24, 2015

I want to create a SSIS package as follows

Conditions
If there are about 100 records in text file, if there is an error at 43 and at 67 record respectively , it should capture 43 and 67 record in failure folder and remaining 98 records , should be processed

1) Successful record into table and move the success record from the folder
to new path say( Success folder) (98 records to table)
2) Unsuccessful records to new path (Failure folder) (2 lines )
3) Error message to capture the failed records and store them in another folder(Error log) (2 line failure information)

While writing the 3rd condition to error log table , it has to point out the record which is failed for what reason, say it may be due to invalid data type for column 10 for 43 record, and incorrect syntax error at 67 record.

View 9 Replies View Related

Data Access :: How To Load Data From CSV File In Temp Table At Run Time

May 28, 2015

how I can load the CSV file data into the sql server table. I know there are ways like bulk insert and other to load the csv file data into the table. But in my case the table doesn't exist and has to be created at the run time. With simple insert in temp table we do like select * into #temp from tablename and that creates the temp table. So. I need something like that which create the temp table and load the data into it. because the CSV file would have different number of columns and names so I can not create the table structure in advance. I have to create the table at run time. 

View 3 Replies View Related

How To Bring File From FTP To Staging Table In SQL Using SSIS

Nov 6, 2007

I have been working on this without any success. Anybody out there to help?

1. There are new files uploaded in the FTP site everyday by other regions. 3 regions and 1 file each. That means 3 files at 3 different times. The file name will be that same day's date.csv. Example; today's file will be A_20071106.csv , B_20071106.csv and C_20071106.csv. Tomorrow's will be A_20071107.csv,B_20071107.csv and C_20071107.csv. How do I do this to run everyday and take the file only for that day from FTP straight into SQL Staging table?


2. Everyday, immediately after each file is uploaded, to indicate the FTP file is loaded successfully, there will be a 20071106.dummy and so on for all three files will show up on the same folder in FTP. I must make this package runable only if the .done file arrives for that days date and then execute the .csv file. Otherwise check after 30 minutes if the .dummy is there yet or not. Do this until the .dummy file comes. Then execute the package that is on that time. Then do the same for the other two for that particular day. The times are 4pm for A, 6pm for B and 8.30pm for C

3. Then, if there is a new row, it should INSERT, any changes (based on 5 columns), it should update and if there was a row yesterday and it is not there today, DELETE.

4. All this must be automated.

View 4 Replies View Related

Data Load: Want To Redirect Extant Rows In Table To A File

Sep 29, 2006

We are working on a DataWarehouse app. The DW has been loaded wiith transactional data from the start of September. and we want refresh the DW with a full load from the original source. This full load wil consist largely of the same records that we loaded initially in the DW but some records will be new and others will have changed.

During the load I want to direct input records NOT already in the DW to a "mods" table and ignore those input records that alreayd exist in the DW. Can SSIS help out with this task?

TIA,



Barkingdog

View 1 Replies View Related

Integration Services :: How To Load Data From CSV File To Dynamic Table

Jun 11, 2015

I have a requirement to load bulk of csv files to sql table. some times, some columns could not come in csv file(some times 100 columns and some times 80 cloumns). That time the package is getting failed. How to create a table dynamically based on csv file structure.

View 8 Replies View Related

Load Data From Text File Into Some Table Implemented In Stored Procedure.

Nov 8, 2006

Hello, I want to load data from text file to MS SQL DB table.

In MySQL, it is the "LOAD DATA INFILE..." query statement.

What is sutable query if I want to migration from Mysql to MS SQL Server 2005 Express?

View 2 Replies View Related

SQL Server 2012 :: Data Transfer From Staging Table Into Main Table

Jan 14, 2014

What is the best way to transfer data from the staging table into the main table.

Example:
Staging Table Name: TableA_satge (# of rows - millions)
Main Table Name: TableA_main (# of rows - billions)

Note: Staging table may have some data same as the main table.

Currently I am doing:
- Load data into staging table (TableA_stage)
- Remove any duplication of rows from the staging table (TableA_stage)
- Disable all indexes on main table (TableA_main)
- Insert into main table (TableA_main) from staging table (TableA_stage)
- Remove any duplication of rows from the main table using CTE (TableA_main)
- Rebuild indexes on main_table (TableA_main)

The problem with the above method is that, it takes a lot of time and log file size grows very big.

View 9 Replies View Related

SQL Server 2008 :: Inserting Data From Staging Table To Main Table

Feb 23, 2015

I am trying to insert bulk data into main table from staging table in sql server 2012. If any error comes, this total activity is rollbacked. I don't want that to happen. I want to know the records where ever the problem persists, and the rest has to be inserted.

View 2 Replies View Related

SQL Server 2012 :: Number Of Column Mismatch In Source File And Staging Table

Apr 14, 2015

I am having one store procedure which use to load data from flat file to staging table dynamically. everything is working fine.Staging_temp table have single column.All the data stored in that single column below is the sample Data.

AB¯ALBERTA ¯93¯AI
AI¯ALBERTA INDIRECT ¯94¯AI
AL¯ALABAMA ¯30¯

[Code]....

View 2 Replies View Related

Creating Different Staging Table With Data Extensi

Jun 11, 2008

Hi All,

I am extracting source data which is in txt fille to OLE DB destination. But data of each day I want to save in different staging table. For Eg; tblProduct20081206, tblProduct20081207. How can it be done. I have seen lots of posting and script when destination is Txt. I want to use same table for staging but want to create different table for each day with adding date extension.

Please Help

View 2 Replies View Related

Copy Data From Staging Table To Production?

Apr 12, 2015

I am trying to insert data from staging table to production table. In the staging table I only have period or date but no primary key.

This is my staging table

Create stagingtable(
[Period] [char](7) NOT NULL,
[CompanyCode] [varchar](100) NOT NULL,
[total] [int] NULL,
[status] [varchar](50) NULL
)

Create Production(
[Period] [char](7) NOT NULL,
[CompanyCode] [varchar](100) NOT NULL,
[total] [int] NULL,
[status] [varchar](50) NULL
)

I get this every month. What can I do to make sure only unique record are loaded into production table with no duplicate from previous month.

View 5 Replies View Related

Staging Data In ##Table In A OLEDB Command Task

Apr 24, 2008



Hi All,

In one of my interfaces,i am trying to Stage data in a ##table from a Flat file,so that i can iterate through the data in for each loop afterwards.

I have written a stored Proc to do this.But what i am observing is in the ##Table only the last Row will be persisted.

The SP looks like this




Code Snippet
CREATE PROCEDURE USP_SVInsS093Data_V001
@DetailRECTYPE nvarchar(5),
@StoreNumber int,
@Product nvarchar(20),
@Planogramref nvarchar(20),
@DisplayGroup nvarchar(5),
@ModSequence int,
@Shelfnumber int,
@Productposition int,
@StartModnumber nvarchar(20)

AS

SET NOCOUNT ON
BEGIN
if object_id('tempdb..##S093_TempTable') is not null
drop table ##S093_TempTable


CREATE TABLE ##S093_TempTable(
[DetailRECTYPE] [nvarchar](5) NULL,
[StoreNumber] [int] NULL,
[Product] [nvarchar](20) NULL,
[Planogramref] [nvarchar](20) NULL,
[DisplayGroup] [nvarchar](5) NULL,
[ModSequence] [int] NULL,
[Shelfnumber] [int] NULL,
[Productposition] [int] NULL,
[StartModnumber] [nvarchar](20) NULL
)

INSERT INTO ##S093_TempTable
([DetailRECTYPE]
,[StoreNumber]
,[Product]
,[Planogramref]
,[DisplayGroup]
,[ModSequence]
,[Shelfnumber]
,[Productposition]
,[StartModnumber])
VALUES
( @DetailRECTYPE ,
@StoreNumber ,
@Product ,
@Planogramref ,
@DisplayGroup ,
@ModSequence,
@Shelfnumber ,
@Productposition ,
@StartModnumber
)






If i have a select query after the Data Flow task only the last row is returned.

Now how to make the ##Table keep all the rows.

Any help will appreciated.

Thanks in Advance...

Cheers

Srikanth Katte

View 8 Replies View Related

SQL Server 2012 :: Unable To Load Data From Flat File To Table Using Bulk Insert Statement

Apr 3, 2015

I am unable to load data from flat file to sql table using bulk insert sql statement

My code:-

DECLARE @filePath VARCHAR(200)
DECLARE @sql VARCHAR(8000)
Declare @filename varchar(100)
set @filename='CCNVZ_150401054418'
SET @filePath = 'I:IncomingFiles'+@FileName+'.txt'

[Code] .....

View 1 Replies View Related

Integration Services :: Handling Empty Text File Load Into Table Through SSIS Data Flow?

Jun 16, 2015

We have created SSIS package to load a text file into a table. Source system shares 10 text files and recently they stopped generating data for one of the text file (comping empty), after few months they will start generating the data for the empty file batch processing. 

The Issue here is Data Flow task is getting failed while loading empty text file into table. How to handle this empty file load issue in SSIS package.

View 3 Replies View Related

Move Data From Excel File To A Table (MS SQL)

Aug 22, 2005

I have an application , user will read information in Excel file and insert that data into my application, I think it spend a lot of time. I want to make a tool which move data from Excel file to a table in My application (MS SQL) automaticly. How to do it, anybody has tool or know how to do, pls help me.thanks.

View 2 Replies View Related

Power Pivot :: Incremental Load Excel File

Jun 4, 2015

We have build a powerpivotmodel that uses a couple of datasources. The user extracts on monthly base an excel file with 20 samples (randomly) from the model to check some aspects of the input. He can put Yes or No into a column if he agrees or not and fill in a remark field. Next this excel file is loaded back into the powerpivot model (in a nightly batch) and the Yes/No field and the remark field are added to the original data. If the user refreshes his excel sample file from the pivot, the remark and YN field will be filled with the values he added. So far so good. One moth later when he extracts the file for a new month a problem arises. Since the excel file only has data for the new month, when the PP-model is reloaded the data for the previous month will be lost. So we need some sort of incremental load for the excel file. Is that possible and is this the best way to handle this situation?

View 4 Replies View Related

Excel File - Migrate All Data Once Into Database Table

Apr 2, 2012

I have an excel sheet with 10 tabs (work sheets) and each work sheet contains data. I want to migrate all data once into my database table.

View 2 Replies View Related

Write Data From A SQL Server Table Into An Excel File

Apr 30, 2007

Hi,



I need to export data from a table of SQL server 2000 database, into an Excel 2000 sheet. I tried following query from the sql query analyzer




Code SnippetSELECT * INTO [Excel 4.0;Database=F:ew.xls].[sheet1] FROM tab1





It gave me following error




Code SnippetServer: Msg 2760, Level 16, State 1, Line 1
Specified owner name 'Excel 4.0;Database=F:ew.xls' either does not exist or you do not have permission to use it.




The F:ew.xls file is present and it has no additional security applied, hence I think it should be writable.



Please tell me where am I doing wrong.

I don't know if this is the correct news group or not. Let me know if I have to post to some other news group.


- Abhijit

View 3 Replies View Related

HELP,,Imports A Data From Excel File Into A Table In SQLserver Database

May 6, 2008

Dear all,,
I need your help,,I'm work in website project using ASP.NET,,I have to register the users of this site,, the users are over 200,,so,,I'm thinking in away to save my time,,All the information of these users are stored in Excel file,,What I want to do is to imports these data from the excel file into a table in my database(SQLserver database),,Could you help in coding by VB.NETThanks in advance,,

View 9 Replies View Related

DB Engine :: How To Insert Excel File Data Into Temp Table

Jul 9, 2015

I have an Excel file with .csv extension . it has on sheet with name Sheet1.

Now, I'm trying to insert this Excel data into one #temp table. I tried with syntax:

----------------
Exec sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
Exec sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0' , N'AllowInProcess' , 1; 

[Code] ...

But, I'm getting error:

The OLE DB provider "Microsoft.ACE.OLEDB.12.0" for linked server "(null)" reported an error. The provider did not give any information about the error.

Cannot initialize the data source object of OLE DB provider "Microsoft.ACE.OLEDB.12.0" for linked server "(null)".

If I'm executing for .xls file this statement is working finr and rows are inserted into #temp table. How to take excel file of .csv extension??

View 3 Replies View Related

Loading The Different Language Data From Excel File To The Single Table

Feb 29, 2008

can anyone help me to solve this problem
i have created a ssis package to load the data from excel file to the table, but we are getting the data in different language ie in french,english and in china after loading the data when we view the data it is showing as junk characters for chinese data but we are able to see other language data ie french and english.
so please tell me how to solve that
reply to my mail id(sandeep_shetty@mindtree.com)

View 11 Replies View Related

Need To Capture Actual Data In Row On Error To Flat File

Sep 18, 2007

Let me preface by saying I am not very familiar with SSIS.

Ideally, since the Transfer SQL Server Objects task can do all tables, I would like to use it to copy only data from one server to a new server that has the tables pre-created. When I encounter any kind of error, in addition to the error information provided by SSIS, I also need the actual row data.

If using the Transfer Object task can't do that, how would I loop through all the tables on an OLEDB source and capture the same error information on the destination? I figured out how to do the Data Flow a table with a redirect error output but that does not give me the actual row data.

View 7 Replies View Related

Exporting Data From SQL Table To Excel File - How To Delete Rows Before Inserting New

Feb 5, 2007

Hi,

Question pls. I have an MS SQL local package where it exports data from SQL table to Excel file. My question is, how can erase all the records in my excel file before i export the new data from SQL table?

What i want is to delete the rows in the destination file before inserting new records.

Thanks a lot.

View 7 Replies View Related

How To Update Selected Columns Of A Table In SQL Server Db Using Data From A Excel File?

Apr 4, 2007

Hi,I have an Excel file with 400 rows of old values and the correspondingnew values. My table currently has 10 columns out of which 3 columnsuse the old value specified in the excel file. I need to update thoseold values in the columns with the new values from the Excel file.Please guide me as to how to proceed with this.Thanks in advance!

View 4 Replies View Related

Integration Services :: Load Excel File Dynamically With Different Columns And Worksheet Names

Apr 2, 2014

 I have a situation where I want to load the Excel file dynamically, and the excel file have different columns or even worksheet name. How I could approach this? I believe there's no way to modify the meta data (specifically the mapping) in the data flow.

View 6 Replies View Related

Capture System.IO Rows From Incoming File And Insert Into Table

Mar 24, 2006

Ok, I'm not quite sure how to approach this one.  This is a VB.NET console app in which I want to capture each row and throw it into a table.  The reason being, they want a report on what was processed...which I'll be able to do easily in Reporting Services 2005 once this crap is in a table where it should be.
1) What should I use to do this, dataset?  I want to use stored procedures also, not inline SQL
Function here takes an incoming file, and splits it up into separate files.  I want to insert each row that is succesfully split
    Public Sub ProcessFiles(ByVal sIncomingfile As String, ByVal sOutputDirectory As String)
        If sIncomingfile <> "" And sOutputDirectory <> "" Then
            Dim f As New Security.Permissions.FileIOPermission(Security.Permissions.PermissionState.None)            f.AllLocalFiles = Security.Permissions.FileIOPermissionAccess.Read
            Dim file As New IO.FileInfo(sIncomingfile)            Dim filefs As IO.FileStream = Nothing            If file.Exists Then                Try                    filefs = New IO.FileStream(file.FullName, IO.FileMode.Open)  'Place: 1                Catch ex As Exception                    SendEmail("Incoming .mnt or .naf Filename Invalid or not found", "Place: 1")                    Application.Exit()                End Try            End If
            Dim reader As New IO.StreamReader(filefs)            Dim counter As Integer = 0
            Dim CurrentFS As IO.FileStream            Dim CurrentWriter As IO.StreamWriter            Dim extension As String = IO.Path.GetExtension(file.FullName)
            If extension = ".mnt" Then                While Not reader.Peek < 0                    Dim Line As String = reader.ReadLine                    If IsNumeric(Line.Substring(0, 1)) Then                        Dim Parts() As String = Line.Split(" "c) ' split row into parts                        If Parts(0).Length = 8 Then ' if first part is 8 then know we hit another header so cut and then write to file                            counter += 1                            If Not CurrentWriter Is Nothing Then CurrentWriter.Flush() : CurrentWriter.Close()                            CurrentFS = New IO.FileStream(IO.Path.Combine(IO.Path.GetDirectoryName(sOutputDirectory), Line.Substring(59, 4) & "[" & counter.ToString & "]" & Now.ToString("MM-dd-yyyy") & IO.Path.GetExtension(file.FullName)), IO.FileMode.Create)                            CurrentWriter = New IO.StreamWriter(CurrentFS)                        End If
                        If Not CurrentWriter Is Nothing Then                            CurrentWriter.WriteLine(Line)                        End If
                    End If                End While
                If Not CurrentWriter Is Nothing Then CurrentWriter.Flush() : CurrentWriter.Close()
                MoveFilesFTP(sOutputDirectory, "mnt")
            ElseIf extension = ".naf" Then                While Not reader.Peek < 0                    Dim Line As String = reader.ReadLine                    If Not IsNumeric(Line.Substring(0, 1)) Then ' if first part is not a number, then we know it's a header so split the file                        counter += 1                        If Not CurrentWriter Is Nothing Then CurrentWriter.Flush() : CurrentWriter.Close()                        CurrentFS = New IO.FileStream(IO.Path.Combine(IO.Path.GetDirectoryName(sOutputDirectory), Line.Substring(6, 4) & "[" & counter.ToString & "]" & Now.ToString("MM-dd-yyyy") & IO.Path.GetExtension(file.FullName)), IO.FileMode.Create)                        CurrentWriter = New IO.StreamWriter(CurrentFS)                    End If
                    If Not CurrentWriter Is Nothing Then                        CurrentWriter.WriteLine(Line)                    End If
                End While
                If Not CurrentWriter Is Nothing Then CurrentWriter.Flush() : CurrentWriter.Close()
                MoveFilesFTP(sOutputDirectory, "naf")            End If        Else            'input file not valid            SendEmail("Incoming .mnt or .naf Filename Invalid", "Place: 1")        End If    End Sub

View 2 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved