Variable Number Of Input Columns
May 2, 2007
We are trying to use the Import/export wizard to load a text file to a SQL Server 2005 database. The input file has a variable number of columns per row. For example, the first row has 3 columns, the second has 7, the third has 3, etc. The number of columns varies from 2 to 9 in the input file. The columns are separated by an uptick (`) and the rows are terminated by {CR}{LF}. We are using code page 1252. On processing, the wizard reads the first row (with 3 columns) ok, but then assumes all the other rows have 3 columns and parses the rows accordingly, ignoring the field and row terminators.
The process worked fine with SQL Server 2000. Is there some setting that we are missing, or some configuration on the database that we should be checking?
Thank You
View 3 Replies
ADVERTISEMENT
Dec 12, 2007
I am building an SSIS package that loops through a table in SQL Server and dynamically builds a select statement that i would like to use as an ole db source. I have been having a difficult time with this as the select statement that i am generating is over 200,000 characters long so using an sql variable is out of the question.
I ended up placing the select statement into a table where each row of the table represents a piece of the select. I then use an execute_sql task that selects the entire rowset from this table into a variable object. I then use a for each loop to shred the variable and concatenate it into on big string variable called user:: sql_statement that is my select.
After setting up the loop and testing to see if the user:: sql_statement variable populates correctly i then added a data flow transfer with an ole db source and destination. I then go into the advanced editor for the source and set it to accept an sql statement from a variable and use my user:: sql_statement variable. I was forced to set validate external metadata option to false to avoid an error since there is no way to validate the columns until the for each loop runs during run time.
Now thats all fine and good but what is causing my problem is that during run time, when the package gets to the data flow task, the select statement doesn't seem to be populating the input columns of the data source. I have been searching to no avail on a way to tell the data source to update the input columns but every time it gets there, the package bombs out telling me the ole db source has no available output columns.
Specifically the error i get is :
[DTS.Pipeline] Error: "output "OLE DB Source Output" (6616)" contains no output columns. An asynchronous output must contain output columns.
Any help with this would be much appreciated.
View 18 Replies
View Related
Jul 13, 2007
Im trying to determine the best way to store data gathered from a form that a user will fill out online. The form is dynamic and is customized at run time based on group-specific criteria. The end result is a form that might have 3 extra text boxes, 2 extra sets of radio buttons and a freeform textbox, whereas for another group, there might be a slightly different set of input fields. Now comes the issue of storing this data. Since the fields can be somewhat dynamic, it could get tricky to define table columns for each possible input field. So Im considering storing the data as xml. Has anyone else had to build custom forms and ended up storing the data as xml ?
View 2 Replies
View Related
Feb 6, 2008
Hi, I need to update a number of columns in a number of tables - I just don't know how many. In this case, I am updating all varchar fields and nvarchar fields to be converted to lower case. The problem is that the table structure is amended over time as columns are added programmatically, so I do not know which tables have which columns and if so which of them are varchars.
I can get a table of which fields I need to update using:
SELECT TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, DATA_TYPE
INTO tblTempLCase
FROM information_schema.columns
WHERE DATA_TYPE LIKE '%varchar'
and I can do the update with UPDATE tblxxx SET column = LOWER(column)
But what I don't know is how to step through my temporary table and do the updates. I can do it in ASP.NET, but that involves pushing commands and data between ASP and SQL, and will be too slow. How do I do it in SQL?
Thanks,
Dave Stephens
View 3 Replies
View Related
Feb 6, 2008
Hi,
I need to update a number of columns within a number of tables - I just don't know how many. In this case, I want to convert all varchar and nvarchar columns to lower-case versions of themselves. The problem is that the table structure is changed programatically, and so at any point in time I cannot be certain what fields are in which table, and what data type they are.
I know that I can get a lit of columns using:
SELECT TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, DATA_TYPE
INTO tblTempLCase
FROM information_schema.columns
WHERE DATA_TYPE LIKE '%varchar'
and do the update using:
UPDATE tblABCDE SET column = LOWER(column).
In ASP.NET I can pull in this temporary table using a SQL Data Adapter, and then step through the records to formulate the UPDATE statements and execute them all. However, I hope that this is possible in SQL too, so that I do not have to keep firing data/commands between ASP and SQL, as it should be quicker, and is also neater.
If so, how do you do it?
Thanks,
Dave Stephens
View 1 Replies
View Related
Feb 6, 2008
Hi,
I need to update a number of columns within a number of tables - I just don't know how many. In this case, I want to convert all varchar and nvarchar columns to lower-case versions of themselves. The problem is that the table structure is changed programatically, and so at any point in time I cannot be certain what fields are in which table, and what data type they are.
I know that I can get a lit of columns using:
SELECT TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, DATA_TYPE
INTO tblTempLCase
FROM information_schema.columns
WHERE DATA_TYPE LIKE '%varchar'
and do the update using:
UPDATE tblABCDE SET column = LOWER(column).
In ASP.NET I can pull in this temporary table using a SQL Data Adapter, and then step through the records to formulate the UPDATE statements and execute them all. However, I hope that this is possible in SQL too, so that I do not have to keep firing data/commands between ASP and SQL, as it should be quicker, and is also neater.
If so, how do you do it?
Thanks,
Dave Stephens
View 1 Replies
View Related
Nov 8, 1999
How do I create a temporary table in a stored procedure with differeent number of columns?
That is: sometimes ten columns, sometimes 24 etc.
View 1 Replies
View Related
Jan 26, 2006
Hi,
We are building an application with
a database that contains Jobs. These Jobs have properties like Name, Code etc.
and some custom properties, definable by the application admin. For bulk import
of Jobs, we want to allow the import of an Excel sheet with the columns Name,
Code and a variable amount of columns. If the header names of these columns in
the Excel sheet match the name of a custom property in the system we want to add
the value of that cell into the database as property
value.
In our Data Flow of our Import
Package in SSIS we added an Excel Source that points to a test excel sheet with
the Name and Code columns and €“ for this example - 3 custom property columns
(Area, Department, Job Family). When we configure the Excel Source in the Excel
Source Editor, we have the option to select the Columns from the Available
External Columns table. But here lays the problem, we do not know at design
time, what custom property columns to expect. We DO expect the Name and Code
columns, but the rest is uncertain at design-time.
That raises the question: Is there
some way to select all of any incoming columns (something like a SELECT * in
T-SQL)? This looks like a big problem since it would mean that the .DTSX XML that is
being generated at design-time would need to be updated at run-time to reflect
the variability of the columns that might be encountered while reading the excel
sheet.
Then, we thought, we could add a Script
Component to our data flow that passes some kind of DataSet (or DataReader) in
which we can walk through the columns ourselves? But then still, we miss the
option to include ANY of the columns while reading an Excel sheet (or any other
datasource by the looks of it)
We are aware of the option of
optional columns in combination with the RaggedRight option, but it seems that
we would have to put all of the columns of a row in just one column and then
extract all the columns later with Derived Columns. But then, since the source
import file is being prepared by an application admin, we want don€™t want to
burden him with this horrendous task of putting everything in one
column.
We would like to have some way of
iterating through all the columns, either in a Script Component or maybe with a
Pivot/Unpivot mechanism.
Does anyone have any suggestions? Are there other options we should have considered?
View 2 Replies
View Related
Jul 13, 2007
Hi
I have a dataset with 2 columns, a rownumber and a servername - eg
rownumber servername
1 server1
2 server2
....
15 server15
I want to display the servernames in a report so that you get 3 columns - eg
server1 | server2 | server3
server4 | server5 | server6
...
server13 | server14 | server15
I have tried using multiple tables and lists and filtering the data on each one but this then makes formating very hard - i either end up with a huge gap between columns or the columns overlap
I have also tried using a matrix control but cant find a way to do this.
Does anybody know an easy way to do this? The data comes from sql 2005 so i can use a pivot clause on the dataset if somebody knows a way to do it this way. The reporting service is also RS2005
Thanks
Anthony
View 1 Replies
View Related
Feb 26, 2007
I am writing a package that will process delimited flat files that will come in one of a few different versions. Within each flat file, the number of delimited columns will be the same, but each version of the file has a different number of columns. I have tried configuring the flat file data source to expect the version with the largest number of columns, but it will then throw away rows that have less than this number of columns (warning: There is a partial row at the end of the file).
Is it possible to use a single flat file data source that will work with all of the different width files?
View 1 Replies
View Related
Jun 23, 2015
I want to load flat files into a single table. But the flat files can have variable number of columns upto a maximum of 10 columns. The table in my database has 10 columns in it. So in case if I load a flat file having 6 columns then rest of the columns in the table will have nulls. I don't want to use script task for this as I am not good in writing C#code.
View 5 Replies
View Related
Sep 24, 2007
Hi
I want to enter rows into a table having more number of columns
For example : I have one employee table having columns (name ,address,salary etc )
then, how can i enter 100 employees data at a time ?
Suppose i am having my data in .txt file (or ) in .xls
( SQL Server 2005)
View 1 Replies
View Related
Nov 13, 2007
Hello All,
I have a requirement, where the number of parameters being to a stored procedure, is not fixed. It is to have a list of computers, belonging to a particular Domain, or, more DOMAINS or maybe just a list irrespective of the Domain. For this, the @Domain parameter, could have one value, or more values, or no values as well. Can you please let me know how do I go about this?
Thanks a lot.
Mannu.
View 3 Replies
View Related
Jun 5, 2007
Hi,
I am currently designing a SSIS package to integrate data into a data warehouse fact table. This fact table has about 70 columns among which 17 are foreign keys for dimension tables.
To insert data in that table, I have to make several transformations and lookups. Given the fact that the lookups I have to make are a little complicated, I have about 70 tasks in my Data Flow.
I know it's a lot, but I can't find a way to make it simpler. It seems I really need all these tasks.
Now, the problem is that every new action I try to make on the package takes a lot of time. At design time, everything is very slow. My processor is eavily loaded each time I change a single setting in one of the tasks, and executing the package in debug mode takes for ages. If I take a look at the size of my package file on disk, it's more than 3MB.
Hence my question : Are there any limitations in terms of number of columns or number of tasks that can be processed within a Data Flow ?
If not, then do you have any idea why it's so slow ?
Thanks in advance for any answer.
View 1 Replies
View Related
Apr 15, 1999
Does anyone know how to parse an input variable to a stored proc? For example, if I have an input variable that is: 'ford,chevy,pontiac' how do I parse through this variable and handle each value independently?
Since I do not know how many values may exists I can't do it based on a set amount of input variables.
View 3 Replies
View Related
Mar 18, 2008
Hey Guys,
Heres my issue. please help
I have xml files and xslt files are in seperate folders. and each xml file has its own xslt. I've got the for loop riunning which loops thro the metadata table. This table has the file and folder localtions of xml and its corresponding xslt.
I'm using the transform xml task in control fllow. i dont see an option to set input xml name and folder from Variable in transform xml component properties? It seems to expect fully qualified path. is there an alternative? or what am i doing wrong?
View 5 Replies
View Related
May 4, 2004
I am trying to Execute a Stored Procedure using Call OSQL from a .bat file, and passing a DOS variable to the Stored Procedure.
The DOS variable is established in the DOS command...
set SERVERJOB=JOBNAME
I have tried...
EXEC sp_procedure %SERVERJOB%
With this, I get an error...
sg 170, Level 15, State 1, Server ABCDEFGH, Line 20
Line 20: Incorrect syntax near '%'.
If I put the variable name in quotes on the EXEC statement above, the value used is the variable name, %SERVERJOB% itself rather than the value the variable was set to, which would be JOBNAME in the above example.
Any ideas??? Thanks!
View 2 Replies
View Related
Oct 29, 2007
In my data flow task, I use a data reader source to get data from our 400. Currently, I have a simple sql command that is pulling data from a file qualifying by date, which is hard-coded. I would like to pass in a parameter which has the date in it. How can I do this? Thanks!
View 8 Replies
View Related
Sep 21, 2006
Hi ,
Im trying to build a package that will copy data from excel to SQL
in a program
unfortunately , when I open the package xml file and I drill into the
oledb destination I see that I have no available input columns
what could be the problem ?
thanks ahead
Eran
p.s
the script:
Dim p As Package = New Package()
Dim e As Executable = p.Executables.Add("DTS.Pipeline.1")
Dim thMainPipe As TaskHost = CType(e, TaskHost)
thMainPipe.Properties("Name").SetValue(thMainPipe, "Data Flow")
Dim dataFlowTask As MainPipe = CType(thMainPipe.InnerObject, MainPipe)
' Create excel connection MANAGER
Dim excelCon As ConnectionManager = p.Connections.Add("EXCEL")
excelCon.Name = "ExcelSourceConn"
excelCon.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=e: ry ry ry.XLS;Extended Properties=""Excel 8.0;HDR=YES"""
' Create sqldev connection Manager
Dim sqlCon As ConnectionManager = p.Connections.Add("OLEDB")
sqlCon.Name = "sqldevConn"
sqlCon.ConnectionString = "Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;User ID=sa;Initial Catalog=InsFocus_Admin_Eran;Data Source=SQLDEVSQLDEV"
''create source component
Dim excelSource As IDTSComponentMetaData90 = dataFlowTask.ComponentMetaDataCollection.New()
excelSource.Name = "ExcelSource"
excelSource.ComponentClassID = "DTSAdapter.ExcelSource.1"
Dim excelInstance As CManagedComponentWrapper = excelSource.Instantiate()
excelInstance.ProvideComponentProperties()
excelSource.RuntimeConnectionCollection(0).ConnectionManager = DtsConvert.ToConnectionManager90(p.Connections(0))
excelInstance.SetComponentProperty("AccessMode", 0)
excelInstance.SetComponentProperty("OpenRowset", "business_codes$")
excelCon.AcquireConnection(Nothing)
'excelInstance.ReinitializeMetaData()
excelInstance.ReleaseConnections()
Dim sqldev As IDTSComponentMetaData90 = dataFlowTask.ComponentMetaDataCollection.New()
sqldev.Name = "sqldev"
sqldev.ComponentClassID = "DTSAdapter.OLEDBDestination.1"
Dim sqldevInstance As CManagedComponentWrapper = sqldev.Instantiate()
sqldevInstance.ProvideComponentProperties()
sqldev.RuntimeConnectionCollection(0).ConnectionManager = DtsConvert.ToConnectionManager90(p.Connections(1))
sqldevInstance.SetComponentProperty("AccessMode", 0)
sqldevInstance.SetComponentProperty("OpenRowset", "business_codes")
sqldevInstance.AcquireConnections(Nothing)
sqldevInstance.ReinitializeMetaData()
sqldevInstance.ReleaseConnections()
Dim path As IDTSPath90 = dataFlowTask.PathCollection.New()
path.AttachPathAndPropagateNotifications(excelSource.OutputCollection(0), sqldev.InputCollection(0))
MsgBox(excelSource.OutputCollection.Count)
'For Each input As IDTSInput90 In sqldev.InputCollection
' Dim vInput As IDTSVirtualInput90 = input.GetVirtualInput
' For Each vColumn As IDTSVirtualInputColumn90 In vInput.VirtualInputColumnCollection
' ' Call the SetUsageType method of the design time instance of the component.
' sqldevInstance.SetUsageType(input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY)
' Next
'Next
Dim app As Application = New Application()
app.SaveToXml("c:myXMLPackage.dtsx", p, Nothing)
View 2 Replies
View Related
Sep 10, 2007
I am working on a Statistical Reporting system where:
Data Repository: SQL Server 2005
Business Logic Tier: Views, User Defined Functions, Stored Procedures
Data Access Tier: Stored Procedures
Presentation Tier: Reporting ServicesThe end user will be able to slice & dice the data for the report by
different organizational hierarchies
different number of layers within a hierarchy
select a organization or select All of the organizations with the organizational hierarchy
combinations of selection criteria, where this selection criteria is independent of each other, and also differeBelow is an example of 2 Organizational Hierarchies:
Hierarchy 1
Country -> Work Group -> Project Team (Project Team within Work Group within Country)
Hierarchy 2
Client -> Contract -> Project (Project within Contract within Client)Based on 2 different Hierarchies from above - here are a couple of use cases:
Country = "USA", Work Group = "Network Infrastructure", Project Team = all teams
Country = "USA", Work Group = all work groups
Client = "Client A", Contract = "2007-2008 Maint", Project = "Accounts Payable Maintenance"
Client = "Client A", Contract = "2007-2008 Maint", Project = all
Client = "Client A", Contract = allI am totally stuck on:
How to implement the data interface (Stored Procs) to the Reports
Implement the business logic to handle the different hierarchies & different number of levelsI did get help earlier in this forum for how to handle a parameter having a specific value or NULL value (to select "all")
(WorkGroup = @argWorkGroup OR @argWorkGrop is NULL)
Any Ideas? Should I be doing this in SQL Statements or should I be looking to use Analysis Services.
Thanks for all your help!
View 1 Replies
View Related
Jul 9, 2001
Is there any way to use table variable as input to a function or stored procedure?
View 3 Replies
View Related
Aug 6, 2007
Hello,
I need to create a view which will include an additional table every week of our financial year.
Example: Every week we create a table called Sales_YYYY_WW where YYYY is the Year and WW is the Week, so after the first week we have the table Sales_2007_01 and after the second week we have Sales_2007_01 and Sales_2007_02 etc.. These tables are created evey week and don't exist at the start of the financial year.
I'd like to create a view at the beginning of each year to encompass each table evey week as it is created. If all of the tables were there at the start of the year it would be simple:
CREATE VIEW VIEW_SALES_2007
AS
SELECT * from Sales_2007_01
UNION ALL
SELECT * from Sales_2007_02
etc.
But I don't know how to achieve this when the tables don't already exist. I don't want to have to edit the view each week to add in the new table.
Can anyone out there help me?
View 3 Replies
View Related
Oct 20, 2007
Hi,
I have an Execute Process Task that calls a batch file.
I need to pass in a date value to the batch file.
If I enter the value 20071018 in the "Arguments" section, it works fine.
However, I use the "Standard Input Variable", User::MyDate (which gets assigned in a previous Execute SQL Task), it bombs. The query I am using returns the date back in the format yyyymmdd. It gets assigned to a string variable, called MyDate.
e.g.)
SELECT TOP 1 CONVERT(VARCHAR, MyDate, 112) AS MyDate
FROM myTable
which result is mapped to User::MyDate
Is there any obvious reason why this doesn't work?
Thanks
View 3 Replies
View Related
Nov 6, 2006
Hi Everyone:
I have a quick but imp SSIS question. I have a For Each Loop Container, and inside that I wish to add a Execute SQL Task item, so I can call a sp to do some inserts/updates. The ForEachLoop container is looping thru a ADO Object source variable(which is the user variable defined by me, as a FullResultSet). In my Execute SQL Task, I would like to utilize one of the columns from the result set as an input parameter to my Stored procedure. Can someone please advise on how to do this? Please let me know if you have any more questions. I am waiting for a response... Thanks in advance.
MA
View 1 Replies
View Related
Nov 24, 2006
Microsoft says it is possible but I just do not see how. Here is the
link to the help file where it said that variables could be pass as
input to web methods...I do not see the check box they mention on my
IDE.
Any help would be greatly appreciated.
Thanks,
Catherine
View 4 Replies
View Related
May 28, 2007
I did make the SQLServer 2005 tutorial, but the LokUp point I see an error "This component has no available input columns". How I can to solve this?
View 6 Replies
View Related
Dec 22, 2006
Is there a way to preselect the input columns a custom component wanna use. Lets say by DataType (select all of DT_CY for example).
I tried to use the
OnInputPathAttached()
to iterate over the inputColumnCollection and set the UsageType. Didn't worked.
Thanks in advance
View 4 Replies
View Related
Jul 5, 2006
Hi,
In my data flow taks, The Source data is coming from AS400 has 4 columns,
I need to achieve the followings and require your help.
1. Generate a new column which will be combination of concating these 4 columns.
2. Need to add an extra row for Header & Footer.
Please Help.
View 7 Replies
View Related
May 29, 2007
I have three input columns from my flat file, defined as follows:
Col_A
Col_B
Col_C
Now, what I want to do is pick the one that is populated, and put it into my table, in the column called
ABC_INPUT
I assume that I want to use a derived column, but how do I filter out columns that contain only NULL values?
Thanks!!
jim work
View 5 Replies
View Related
May 25, 2007
I'm getting a type mismatch error (currently string, when I think I need Int32 ?) with the code below. I'm having difficultly setting my session variable to type Int32 and having it take up the value of RawCoDeptRowID. When I "Watch" it, it's value remains as "snCoDeptRowID". Also, not sure if my Update command and it's snCoDeptRowID input parameter is well formed? Any advice would be greatly appreciated. Thank you.
Web Page 1:
Session["snCoDeptRowID"] = Convert.ToInt32 (RawCoDeptRowID);
Web Page2:<asp:SqlDataSource ID="SqlDataSource3" runat="server"
ConnectionString="<%$ ConnectionStrings:ConnectionString %>"
UpdateCommand="UPDATE [CompanyDepartment] SET [User_Name] = @User_Name, [FirstName] = @FirstName, [LastName] = @LastName, [Company_Name] = @Company_Name, [Department_Name] = @Department_Name WHERE [User_ID] = @snCoDeptRowID ">
<updateparameters>
<asp:parameter Name="User_Name" Type="String" />
<asp:parameter Name="FirstName" Type="String" />
<asp:parameter Name="LastName" Type="String" />
<asp:Controlparameter Name="Company_Name" Type="String" ControlID ="ListBox1" PropertyName ="SelectedValue" />
<asp:Controlparameter Name="Department_Name" Type="String" ControlID ="ListBox2" PropertyName ="SelectedValue" />
<asp:QueryStringParameter Name="User_ID" Type="Int32" QueryStringField ="@snCoDeptRowID" DefaultValue ="@snCoDeptRowID" />
</updateparameters>
</asp:SqlDataSource>
View 1 Replies
View Related
May 6, 2015
Is it possible to have an entire sql select statement as the input variable to a stored procedure? I want the stored procedure to execute the select statement.
ie.
exec sp_SomeFunc 'select * from table1 where id=1'
It may sound weird, but I have my reason for wanting to do it this way. Is this possible? if so, how do I implement this inside the stored procedure?
View 4 Replies
View Related
May 15, 2008
I'm using a Row Count Task to count the number of records passing thru a particular path in my data flow. I created a package variable and referenced this variable in the Components Properties tab in the Row Count Task. I believe this is the minimum I need to do to get the row count.
However, as I explore the other tabs in the editor, I see there is something called Input Columns tab. What is this for? I didn't select anything in there and things are working fine. At first I thought that I had to choose the columns that I want to have available to me for further processing after the Row Count Task, but this isn't the case. I am able to see all my columns coming out of the Row Count Task even tho I didn't do anything in the Input Columns tab.
So, what is this setting for?
View 4 Replies
View Related
Oct 17, 2007
Can anyone please help
I am trying to get a table from production to development created and populated with data in Prod. When I create a package to set the data flow on the available input columns in mappings I do not see any columns there. The source has been defined as the production table.
Thanks a bunch
View 3 Replies
View Related
