Problem Ith DTS And Delimited File With Empty Last Column
Jan 22, 2006
Hello,
I'm not getting any response to this on the SQLDTS newsgroup, so I
thought that I would try here:
I just ran into this problem and I can't find any other mention of it
through Google. I have a text file that is comma-delimited. It also
uses double quotes as text identifiers. A new column has been added to
the file, but currently has no values. I would like to finish my
development so that when it does finally get some values, they will be
imported as well. The problem is, the last column does not show up in
DTS.
I can reproduce this problem easily enough... create a text file with
the following two lines in it:
1,"test",
2,"test2",
Now, create a new DTS package and add a text file connection. Point it
to the new file and go through the properties for the file. You will
notice that on the second screen where it displays the preview of the
data there are only two columns shown.
This does not happen if there is no text qualifier or if at least one
row has the final column value filled. Is there any way around this
problem?
I have a particular issue that has been causing me some problems for a while. I have an SSIS package that imports an excel file into my database, and then performs various data manipulation that I won't go into. The problem I am having is at the import end. The excel source file I am working on is provided to me by my client. It is a fixed format and doesn't change, it contains a header row and there are 32 headings. The trouble I am having is that quite often, the last column is empty, i.e. it contains no data. The header is still there, but theres no data underneath. When I try to import this file using my SSIS package it fails, and complains about needing to remove the metadata for this final column from the External Columns list (VS_NEEDSNEWMETADATA). When I try to preview this file in the properties of the Excel Data Source, the last column does not exist. It's as if it's determining that as there is no data in that final column, that it's unnecessary and not part of the data set, even though it has a header.
Now I've done a bit of research, and found cases that a sort of like mine, I know that the excel file has the first 8 records sampled to determine the data format. This problem suggested to use the IMEX=1 extension in the connection string, which didn't help. I also discovered that when using flat files, if you have odd numbers of columns in your comma seperated list there can be problems. But neither of these issues seem to match the issue I'm facing.
Has ANYONE had a similar problem to me, and can anyone offer any kind of assistance regarding what I need to do to import an excel file that may or may not have data in the final column?
Using Flat File Connection Manager, I am specifying Text Qualifier = Double quotes{"}, and i have TXT file with one column for lastname and first name as "LN,FN", and settings are set to comma delimted, now the connectin manager is creating two different columns for LN and FN,
I have a table with millions of rows, 1 particular attribute "FromPerson" contains a string of email addresses, names, etc... for example (formatting done by this window... not exists in database...completely cleaned to semi-colon delimited)
tomh@gmail.com; Snyder, John N.; jsnyder@yahoo.com; Miller, Jim; millerj@gmail.com; Tenbrow, Jack; Katie Winslow (can be x number of names, emails...)
as you can see... some of the delimited values match up with an email address, others have just names, or emails... I have successfully split this into 1 row per value per say, but how do i get them to match up the values that do. I think the splitting is really not helping the cause... what i want is this...
tomg@hotmail.com tomg@hotmail.com Snyder, John N. jsnyder@yahoo.com Miller, Jim millerj@gmail.com Tenbrow, Jack Tenbrow, Jack Katie Winslow Katie Winslow
there could/can be other attributes, like datetime stamp, domain(if any)... etc... but I think I can add that later...
I've production sql server 7 sp3 on windows NT. I had a 8GB data file ofwhich 5GB were used and 3GB were unused. I wanted to take back the unused3GB.So I did the following with EM GUI:1. I tried to "truncate fre space from end of the file". Didn't truncatethe file. I believe there was no empty space at the end of the file.2. Next I chose the option to "shrink file to 5GB". And to my horror thedata file instead of taking just 5GB took the empty spaces also and the sizeof the used data file went to 8GB.Any idea what's going on?TIA,SP
Here's a really annoying problem. Let's say you have a text file with 2 million rows.Delimiters all look good and rows are previewed well but the file has a missing row at say lin 1234567 - way deep in the file. When SSIS encounters the blank row, an error is raised and processing on the file STOPS! I verified this in by checking the SSIS log and have even developed an error routine to notify me via email when the error occurs (really cool if I do say so myself ). The main problem still remains - how to resume processing from the point of failure in the file? Any help is appreciated. Thanks.
I am trying to return as a dataset of this that shows the ID as column 1 then each value in the 2nd column as individual columns if I use SELECT LEFT('2ndColumn,PATINDEX(',',2ndColum)-1) I can return the first value as a column but then can't return any further values individually after the first column, I am just learning the new functionality in SSIS so not sure whether this would be my answer as apposed to T-SQL, if anyone has any advice on this it would be greatly appreciated?
While importing a tab delimited file.. it seems ssis interprets the incoming col as 50 chars in length even though it is far smaller. any ideas how this could be??
used bcp utility to send data to output file in tab-delimited format (-t ), but headerfile is separate entity in this query.
when I set FILEheader = firstname,lastname...what must I use to change the comma to tab in the header string. I have tried various ways , {t}, [-t], and others. what am I missing?
Is there a faster way to create my pipe delimited BCP file, besides from creating a format file? Actually, my problem is that I am having issue with the file. It looks perfect, like:
But when I load it to DataStage it puts the entire row as one column. I already specified the | as the delimiter in DataStage. I think the issue is from the column collation. If my data is as simple as my example above, what column collation should i use for the format file? Currentyl, i have something like:
I have a tab delimited file with 122 columns. Can any one let me know if there is a better way of parsing/extracting few columns (say about 15) from the file and loading it into a table using SSIS.
Why shrinkfile empty file does not redistribute data evenly in the primary file group with multiple files:
Please run the script attached to see what the end result is.
This is what I set up last night on my test machine.
1) Create database [FGTest] size 200MB 2) Create table called TEST on primary 3) Insert 40MB of data into test 4) Create another file group called temp in primary size 200MB 5) Shrinkfile('FGTest',emptyfile) so that all data is transfered from FGTest into temp file group. 6) Add another 2 files called DATA2 and DATA3. Both are 200MB. 7) We now have 3 empty files that I want data distributed evenly on. FGTest, DATA2 & DATA3 8) Shrinkfile('temp',emptyfile) to move all the data from temp over the 3 file groups evenly
I would expect at this stage to have the following:
FGTest = 13MB, DATA2 = 13MB, DATA3 = 13MB
(40MB of data over 3 files should be about 13 MBish in each file)
What I actually end up with is this:
FGTest = 20MB DATA1 = 10MB DATA2 = 10MB
It looks as though SQL Server is allocating 50% of all data to the original file and then 50% evenly over the remaining files in PRIMARY.
I have a scenario wherein one of the column values in a row contains a string value which is non-delimited (as shown below). I need to split them by 2 characters and generate as many rows as count of set of 2 digits in that string.
I already have a solution in place to run it thru cursor and then do a while loop on the CountyList column by taking 2 digit value using Substring function (keeping start position dyanamic and jumping 2 positions).
I don't know if this is possible, but I haven't been able to find anyinformation.I have two tables, for example:Table 1 (two columns, id and foo)id foo--- -----1 foo_a2 foo_b3 foo_cTable 2 (two columns, t1_id, and bar)t1_id bar------ ----1 bar_a1 bar_b1 bar_c2 bar_d3 bar_e3 bar_fWhat I'm shooting for is returning the result of a subquery as atext-delimited column. In this example, using a comma as thedelimiter:Recordset Returned:foo bars----- -----foo_a bar_a,bar_b,bar_cfoo_b bar_dfoo_c bar_e,bar_fI know that it's usually pretty trivial within the code that isquerying the database, but I'm wondering if the database itself can dothis.Is this possible, and if so, can someone please point me to how it canbe done?
Hi,I would like to parse out each value that is seperatedby a comma in a field and use that value to join to another table.What would be the easiest way to do so without having towrite a function or routine ?EX.Table AAACOL1 COL21 11, 124, 1562 11, 505, 600, 700, ...Table BBBCOL1 COL211 Desc11124 Desc124156 Desc 156
Hi I am new to VB, and am looking to write some code to import a delimited (by a ~) .txt file into a SQL server table. It doesn't need to append, just to totally overwrtie the table. Is this possible? I have been looking at the Bulk Insert, but this doesn't seem to be quite right. Can anyone help? Cheers, S
Hi I'm pretty new to using Microsoft Visual C# .NET and I want to upload a comma delimited text file from my local machine into a table in an sql server database through a web app. How would I go about programming this and what controls do I need? Any help would be much appreciated. Thanks in advance.
I'm trying to upload a small Web application with a one table database. The hosting company, GoDaddy requires that I upload the database as a comma delimited file. I created the database in Visual Web Developer Express but also have Visual Studio and SQL Server Management Studio Express. I can't figure out how to export the database into a comma delimited file using any of these tools. This should be simple like it is in Access but that doesn't seem to be the case. This is holding up deploying my Web Application.
Is there anyway Sql Server reads a "Tab Delimited Text File" and Compare each record with the Column in a table..
my question is..
I've a Country_Code table which has 3 letter Country Code and the Actual Country names are listed in a Tab Delimited Text File "Country Data" with Country Code and Country Name, how do i read each record and compare to get the Actual Country Name for Display.
Hi,On SQLServer 2000, I have a table with a following structure:MYTABLEcol1 char,col2 date,col3 numberMy Objective:------------Externally (from a command line), to select all columns and write theoutput into a file delimited by a comma.My method:---------1. Probably will use OSQL or BCP to do this.2. Use the following syntax:select RTRIM(col1) +','+ RTRIM(col2) +','+ RTRIM(col3)from MYTABLE;My 3 Problems:-------------1) If there is a NULL column, the result of concatenating any value withNULL, is NULL. How can I work around this? I still want to record thiscolumn as null. Something like say from the example above, if col2 isnull, would result to: APPLE,,52) The time format when querying the database is: 2003-06-24 15:10:20.However, on the file, the data becomes: 24 JUN 2003 3:10PM. How can Ipreserve the YYYY-MM-DD HH:MM:SS format? Notice that I also lost theSS.3) Which utility is better? BCP or OSQL?For OSQL, it has a "-s" flag which gives me the option of putting acolumn separator. But the result is:"APPLE ,14 JUN 2003 , 5"I don't need the extra space.While for BCP, there is no column separator flag.You will notice from my inquiry above that my background in SQLServer isnot very good.Thanks in Advance!!RegardsRicky*** Sent via Developersdex http://www.developersdex.com ***Don't just participate in USENET...get rewarded for it!
I'm trying, through the SQL 2005 Mgt Studio, to export a simple table in a delimited format. I'm selecting a double quote as the text qualifier. My expectations were that only text type fields would be exported with the double quotes an numerical fields would not have any quotes around them. SQL 2000 does this just fine, but 2005 is exporting all my text type and numeric fields with double quotes. Is this a change to SQL 2005 or am I doing something wrong.
I have a problem... When I insert data from a comma delimited file using this mehod a flat file connection sorting, merge join and inserting into the database I get "" around all the data!! The quotes end up around the column names and everything! I had to go in and manually remove the quotes in the text file to get some of my data conversions to work. I know there is a better way. How do I get SSIS to load the data without the quotes? This is an example of the data in the file:
"1007","1","A","","Congratulations - No Health Violations Found","11/02/2005","1007"
When I remove the quotes I do not have any problems. How do I do this without modifying the underlying data? Any ideas would be greatly appreciated!!
I got this code from another one of the MSDN forms. When I run the report and try to export using this format, it still gives me a csv file instead of tab delimited file.
Can someone please help me fix this code so I can get tab delimited text files. Thanks a lot, -Rohit
Hi, I need to search a column in my database (varchar:50) that contains a comma delimited string of numbers (i.e. 1, 2, 3, 4, 5, 10). Currently, I am doing the following SQL query:
SELECT * FROM people WHERE clubs_belongs_to LIKE '%1%'
Where 1 is the number I'm searching for. The problem with the query above is that it returns records that contain 1, 10, 11, 12, 13, etc. in the clubs_belongs_to field. I want the query to only return those people who belong to club number 1, not 10, 11, 12, etc. Please help. Thanks in advance. I have tried using IN instead of LIKE, but that didn't seem to get the results I wanted either.
I have a stored procedure that returns a single row based on a parameter of employee ID. This particular procedure uses a CTE to traverse our org structure. One of the columns is returning a delimited string of Windows login values and due to the business rules, it can contain duplicate values. I need to have that column contain only unique values - no dupes.
For example, this one column could contain something like this:
I'm working on a sales commission report that will show commissions for up to 5 sales reps for each invoice. The invoice detail table contains separate columns for the commission rates payable to each rep, but for some reason the sale srep IDs are combined into one column. The salesrep column may contain null, a single sales rep id, or up to five slaes rep IDs separated by the '~' character.
So I'd like to parse the rep IDs from a single column (salesreplist) in my invoice detail table (below) to multiple columns (RepID1, RepID2, RepID3, RepID4,RepID5) in a temp table so I can more easily calculate the commission amounts for each invoice and sales rep.
Here is my table:
CREATE TABLE invcdtl( invoicenum int, salesreplist [text] NULL, reprate1 int NULL, reprate2 int NULL, reprate3 int NULL, reprate4 int NULL, reprate5 int NULL, )
As you can see, some records have trailing delimiters but some don't. This may be a result of the application's behavior when multiple reps are entered then removed from an invoice. One thing for sure is that when there are multiple reps, the IDs are always separated by '~'
I'm sure there is probably a very easy solution that I am just not seeing or can't Google...
I have a DataFlow that includes a column of Delimited values (i.e. Value1,Value2,etc..). As this DataFlow is populating a parent table, I need split the values into their own dataflow and populate a child table. I've tried a script transformation and couldn't figure out how to accept 1 delimited input row and output multiple rows after a split. Any ideas?
As part of xml parsing, I use multicast to direct output of nodes to their corresponding relational tables and I do have a comma-delimited list for some nodes which basically needs to be converted into rows as illustrated below
Hi, I'm trying to deploy my Web site to GoDaddy. They told me I have to export the SQL Server Express database to a comma delimited file and then upload that file. The export procedure is simple in Access but I don't see any way to do it in SQL Server or from Visual Web Developer or Visual Studio. Also, I can ask them, but I assume I have to export each table separately and also export the ASPNETDB as well. Thanks for the help
Hi, I want to export data/records coming from the database and save it as a .txt file but tab-delimited. The flow of my project is something this.
Web Form->SQL Database->Web Report->Tab-Delimited file.
I will explain more..What we want to do is an online application form. We have a form and will save all the data to sql server database. We also want to save all those information in a tab-delimited file. I would like to save this first in the database(no problem in this part). Then later on export this in tab-delimited file.
If you can give me a little bit tutorial of this i really appreciated..Even 3 records can do as long i can see how to do this..Ooops btw, i also want to name the .txt file as (userid+transactionid).
