Parsing Text File And Inserting Into DB

Mar 19, 2008

Hello all,

I have a question regarding importing text file data into SQL Server.  I'm hoping someone can point me in the right direction, as my searches haven't turned up anything specific enough.

I'm trying to parse a large (24MB) text file.  It's a fixed-width file, with multiple columns.  I need to parse this file, check if a record already exists, and then import the data into the database.  But I don't need to insert every column.  There's only a few columns from the file I need to insert.  This parsing also needs to occur at regular intervals (daily).

I looked at BULK INSERT, but I can't find an example that uses only some of the columns.  Every example uses all columns, and the file is delimited, not fixed-width.

Is there anything within SQL Server that can accomplish this?  I haven't turned up anything that will solve my problem.  The only other solution I can think of is an application that parses the file for me and inserts the data into the database.  But can I schedule that application to run every night at midnight (for example) through SQL Server?

I'm not too familiar with SQL Server, so I appreciate any help offered.

Thanks,
Jay

View 7 Replies


ADVERTISEMENT

Need Suggestions On Text File Parsing Into Database

Feb 28, 2007

I have a website, where people upload tab delimited text files of their product inventories, which the site parses and inserts into a database table.  Here's the catch: Instead of insisting that each user use a standardized format, each user can upload the file in whatever column order they want, they just have to let the site know through a GUI which column is in which order.   And, they may upload columns that if not mapped, will be ignored.  Right now, I am doing all of this in code and it runs slow, I was thinking of offloading this to either a stored procedure, ssis, or bulk upload.   But, with the varying format of the uploaded text file, I am not sure how I could do that.  Any suggestions? Thanks! 

View 1 Replies View Related

Inserting Data To Text File From Database And Inserting Data Back To Database From Text File

Aug 7, 2007

Hello friends....
I am looking for 2 things(using c#.net or vb.net and sql svr 2000)
1.convert data from sql server 2000 database (say customers table from northwinds database) to a text file(separated by commas or just plain space)
2.Insert the data from text file back to database.
Can someone pls give me the detailed code to achieve this....really need this on urgent basis.......Thank You.

View 10 Replies View Related

Inserting Data From Text File

Jul 22, 1998

If this is the wrong place for this question, would someone please tell me so. I am new to SQL Server and still feeling out resources. I have a few books on SQL Server but none cover this question.

I have a text file of dates and numbers that I want to insert into a table. There are way too many rows of data in the file to do this by hand.

Question == How can I essentially insert the text file into my table?

Thank you,
Doug

View 2 Replies View Related

Inserting Data From Text File To SQL ME Table

Nov 24, 2006

Hello,

I have an application taht requires the use of a table. The device that this application works on, has a local memory that does not allow me to insert the 800,000 records that I need. Therefore I have two approaches:

1. To insert less records into my local memory database e.g 40,000 but not row by row, bulk insert is better. How do I do the bulk insert?

2. This is the most prefferable way: To find a way to insert all 800,000 records into a table on the storage card which is 1GB. What do you suggest? Will using threads be helpfull? Any ideas?

I use C# from VS 2005, SQL ME, compact framework 2.0 and windows 4.2.

Thanks in advance,

John.

View 6 Replies View Related

DTS Issue -- Dtsrun Can't Find My Text File That I'm Inserting Into My Sql Table

Apr 21, 2005

Here is the error message that I'm getting
   Error string:  Error opening datafile: The system cannot find the path specified.
The file it's bombing out on is the text file that Im importing into one of my tables through a dts package (which is getting called by the dtsrun statement -- that is giving off this message.)  If anyone would know what may be sending this message off to me let me know.
Thanks in advance.
RB
 

View 2 Replies View Related

Inserting A Control Record Into A Flat Text File Through SSIS

May 2, 2006

I am working on an SSIS project where I create two flat files for submission to a data contractor. This contractor requires a control record be the first line in the file. I create the control record based on the table information being exported.

What I would like to know is, is it possible to utilize the Header Section of the Flat File Destination Editor to insert the control record? And, as it is dynamic, what kind of coding must I do in order to utlise this functionality?

Thanks.

View 4 Replies View Related

Integration Services :: Inserting Data Into Tables From Text File By Having Certain Condition

Nov 8, 2015

I have a text file which has rows 7 rows.I want to insert the data into SQL table using ssis In text file we have a  column which has values as Y or N...I wanted to take only those rows which are Y...But we have only 6 rows in SQL table.It does not have the column with Y or N.

View 2 Replies View Related

Parsing A Text Field

Oct 11, 2001

Does anyone have a script already written to parse a text field in SQL?

View 1 Replies View Related

Parsing Text Using A Cursor

Nov 1, 2001

I am trying to parse a text column using a cursor. Basically here is the statement I am trying to convert to the cursor:
SELECT DATA_ROW,
SUBSTRING(FAILURE_MESSAGE,35,5) AS INVALID_1
SUBSTRING(FAILURE_MESSAGE,70,5 AS INVALID_2
fROM TBL_ERRORS
WHERE LEFT(FAILURE_MESSAGE,200) LIKE '%ORA%'

There can be multiple errors in one string

Thanks

View 1 Replies View Related

Parsing A Text Field Value Into More Than 1 Value

May 9, 2006

Hi,

My table has 2 fields as 'count' and 'codes'. The 'codes' field has 'count' # of code values in each record. Size of each code is 4. For example, if my record is 2,'abcdefgh' then there are 2 codes and the values are 'abcd' and 'efgh'.

Currently I am using 'script component' to parse the field into multiple values. Since I have to read 1 million records and on an average, each record has 10 codes, it is taking hrs to load it.

Can it be done without 'script component' using some other transformations?

Thanks.

View 3 Replies View Related

Parsing Long Text Blobs

Apr 15, 2004

Hey Guys I knwo this may sound impossible but lets say I have a number of fields one of which is a Long blob or long text

is there a way to have MYSQL search the blobs for keywords and then to extract them to other fields?
basically what I am asking is it possible to parse a long text blob for keywords and then grab data before or after those keywords?/


anyone know a way???

View 3 Replies View Related

Reporting Services :: Parsing SSRS Config File And Dynamically Changing File Path Of Config File In Code

Sep 2, 2015

Currently have a single hard coded file path to the SSRS config file which parses the file and provides the reporting services web service url.  My question is how would i run this same query against 100s of servers that may or may not share the same file path as the one hard coded ?

Is there a way to query the registry to find the location of the config file of any server ? which could be on D, E, F, H, etc. 

I know I can string together the address followed by "reports" and named instance if needed, but some instances may not have used the default virtual directory name (Reports).

Am I going about this the hard way ? Is there a location where the web service url exists in a table ? I could not locate anything in the Reporting service database. Basically need to inventory all of my reporting services url's.

View 2 Replies View Related

Flat File Connection Manager Not Parsing File Correctly

Apr 3, 2007

Hi,



I have a flat file, comma-delimited, with strings in double-quotes.



In the connection manager for the file, I have specified that the Text Qualifier = ""



However, in the preview tab, it still shows the strings as surrounded by the quotes, e.g. "mycol1" whereas it should show mycol1 without the quotes.



Next, when I examine the data in the database after the load, it's messed up there also.



"mycol1" ends up in the database as "mycol1



"mycol2" ends up as "mycol2



This is not right.



I have format set to delimited, header row delimiter crlf, etc.



Any ideas?



Thanks

View 3 Replies View Related

T-SQL (SS2K8) :: Parsing Text Field Into 4 Fields

Sep 5, 2014

I have a text field that I need to parse out into 4 fields.

30.125x23.625-18.875x25.375
6.1875X19.375-2.4375x15.625
0X0-0x0
26.875X11.375-11.125x22.875
0X0-0x0
0X0-0x0
6.1875X26.875-2.4375x23.125
6.1875X26.875-2.4375x23.125
6.1875X26.875-2.4375x23.125
26.625X14.875-11.125x22.875
26.625X14.875-11.125x22.875

I need to 26.625X14.875-11.125x22.875 should be 26.625 then 14.875 then 11.125 then 22.875

View 5 Replies View Related

Transact SQL :: Parsing Formula Text To Float

Apr 27, 2015

There is a column in a table that has values like '23 + 45 + 63', '2 + 54 - 22'... and so on. I want to get the result of this formula as float like 131, 34... and so on. Is that possible with a SELECT statement. I tried:

SELECT CAST('23 + 45 + 63' AS FLOAT)

Didn't work.

View 14 Replies View Related

SQL 2012 :: Parsing Out Data String With Text Delimiters

Nov 5, 2014

I'm trying to parse out a line of data that is separated by the text "atc1.", "atc2." etc.

For example,

[atc1.123/atc2.456/atc3.789/atc4.xyz/]

If I only want the data after atc2., then I could search the string for "atc2." and collect all the characters afterwards. But how can I make sure to trim off all the data after "atc3." to make sure I'm only collecting "456" from the example above?

View 2 Replies View Related

String Parsing - Grab Some Key Value Pairs From Text In Column

Jul 21, 2014

I want to grab some key value pairs from the text in sql column

e.g.

some text[Key1=Val1]some text[Key2=Val2][Key3=Val3]some text

I want a function which takes Keyname as input & returns the value related with it if found.

GetValueFmKey('Key1') should return Val1 and like on.

View 1 Replies View Related

SQL Server 2008 :: Text String Parsing To Apply Operators To Datasets?

Aug 7, 2015

I have a problem at the moment, where the client wants to be able to type in a custom algebraic formula with add/minus operators, and then to have this interpreted, so that the related datasets are then added and returned as a single dataset.

An example would be having a formula stored of [a] + [b] - [c]

and if I were to write the SQL to apply that formula, I might write something like (let's assume 1:1 relationships with the ID's)

select a.a + b.b - c.c as [result]
from z
inner join tblA a on z.id = a.id
inner join tblB b on z.id = b.id
inner join tblC c on z.id = c.id

The formula can change though, maybe things like:

[a] + [b] + [c] + [d]
[a] + [b]

The developer before me wrote something SQL-based where they parsed the string and assigned each value of the formula as either positive or negative (e.g A is positive, B is positive, C is negative, now sum the datasets to get the result), and then created one large table of values then summed them. This does (kind of) work, I'm just contemplating potential alternatives, as it is quite a slow process, and feels like it is quite convoluted, when I get into the details. If I were to do something like this in SQL, I'd normally want each part of the expression to be a column, and then to just apply the operators, but because the formula can change, then the SQL would need to be somehow dynamic for this approach.

View 5 Replies View Related

Parsing RTF File

Dec 3, 2007

Hi:

I need to parse an regularly outputted rtf file and was wondering if it is possible in SSIS. I am trying to use the flat file connection manager to do this.

Now, I can't treat tab stops in an rtf like tab stops in a csv, since when you treat an rtf as a text file, you see the format code of the rtf. If I open the rtf in a text editor, the entire file is one line, with lines breaking with:

par}

Columns are tab delimited in the rtf, and they look like this when you treat the rtf as a text file.

plain abfs16f4cf0cb1

(or something like that, the word "tab" is the important part.)

So I use the "plain ab" part to delimit in SSIS, since that is consistent (planning to parse out all the garbage later on). The problem is, sometimes lines don't have a "city" and "state", so it "tabs" right over to the next field. So like this (looking in MS Word):

Phone <tab> City <tab> State <tab> Date <tab> Other fields.....
847-111-2222 <tab> Omaha <tab> NB <tab> 9/14/2007 <tab>
222-222-3333 <tab> 9/14/2007 <tab>
555-121-1212 <tab> Houston <tab> TX <tab> 9/14/2007 <tab>

Now, if you treat an RTF as a text file, it has only one "plain abfs16f4cf0cb1" after the phone number, so even for the missing line there is only one tab, not 3. This is because in the beginning of the row tabs for each row are defined like this:

tql x90 ql x840 ql....etc...

with "tql" and "tx" tags basically saying where all the tab stops are for that row. So for the row above with missing info, it lists fewer tab stops. So the "date" (and associated garbage) ends up under "City" for this row. All of the "Houston" row's data starts appearing in the sql server output table's 2nd last field, as you might expect.

Any suggestions how to pull this in in SSIS during the transformation? I could deal with it after I pull it in, I still have all the data. I'm thinking the logic to do this could be complicated though. I take the data out of the last two fields of the missing row into some other table, use UPDATES to shift the values 2 fields to the right, and then figure out a way to take the data I just put in a temp table back in, but it all sounds a bit complicated.

Let me know if this makes sense--I've almost got it going, I just need to sort this last bit out.

Thanks,
Kayda

View 4 Replies View Related

Parsing A QFX File?

Jan 19, 2008

i am trying to read a qfx file from quicken.
it looks like xml, but its not, but i cannot figure out how to grab what ive got to parse the line.
i put this into a derived column, but its not getting it

SUBSTRING([Column 0],FINDSTRING("<STMTTRN>",[Column 0],1),FINDSTRING("</STMTTRN>",[Column 0],1))

because inside the data, it lools like that's what brackets a tranasction; the data looks like this and varies by trntype, but the columns are tagged like so


<STMTTRN>
<TRNTYPE>POS
<DTPOSTED>20070129160000
<TRNAMT>-0000000000026.50
<FITID>20070129011
<NAME>SUNOCO
<MEMO>01/24 ENGLWD CLIFF NJ 8015V200006
</STMTTRN>
<STMTTRN>
<TRNTYPE>POS
<DTPOSTED>20070129160000
<TRNAMT>-0000000000023.47
<FITID>20070129012
<NAME>KFC
<MEMO>01/26 NANUET NY 8015V215116
</STMTTRN>


i tried the xml transform and unpivot, but have not cracked it.
thanks for any light you can shed
drew


View 1 Replies View Related

Parsing A Tab Delimited File

Dec 5, 2007

I have a tab delimited file with 122 columns. Can any one let me know if there is a better way of parsing/extracting few columns (say about 15) from the file and loading it into a table using SSIS.

View 1 Replies View Related

Complex File Parsing Issue

Nov 28, 2007

Hello,

I have a file that looks like this:

Summary
A ABCD
A Category MarketValue Margin
A category1 1.0000000 1.000000
A category2 2.0000000 2.000000

H Totals Total Cash Net
H 2.00000 200000 2000000

Another Summary
B BCDE
B Activity MarketValue Margin
B activity1 3.00000 3.000000
B activity2 4.00000 4.000000

The items in blue are headers. I don't want to capture those. However, I want to capture all the data in black, and put it into 3 separate tables (or maybe the same table, under the appropriate column names)

This situation differs from anything I've done before in that you can't identify what row contains what data by what's in the row itself. That is, what's in the data rows is random and subject to change. So you can't search the row itself to determine which table it goes to.

However, if there's a way to capture all the rows after a certain header before the header changes again, that might work.

That is, get all rows between A Category MarketValue Margin and H Totals Total Cash Net
and
get all rows between H Totals Total Cash Net and Another Summary
and
get all rows after B Activity MarketValue Margin

Any examples of how I might script this?

Thanks


View 2 Replies View Related

Reading File As One String, Then Parsing - How To Do This?

May 7, 2007

Hi,



The suggestion to do this is buried deep in one of my posts, however I still do not have a clear idea of how to do this.



I have a flat file which has several "bad rows" in it. Because file error redirection is buggy, I need a manual approach to get rid of these incomplete rows in my data file.



Phil, you suggested I read the file as one long string, then parse out the bad rows (using a script?).... however I have no idea as to how to actually do this.



I was wondering if it's possible to clarify the steps involved in doing this, or perhaps point me to an example I can look at, as I cannot seem to get around this problem on my own.



Thanks much!!

View 24 Replies View Related

Another Flat File Parsing Problem

Dec 5, 2006

Hello All!

I know this has come up before and I have tried several of the solutions found within the forum but I just can't seem to import my file correctly and could use some input, please.

Sample file (less fields than actual file):

Name (str), Phone# (str), Description(str), Resolved(bool), Met(bool)

"Kay, Mary","123-4567","Used a "."not a"," in text", "1", "1"

The text is qualified with " and columns delimited with commas but the description field has embedded quotes and commas. Normally it works except if there embedded quotes and commas.

I have tried unqualified data and undouble, but that does not work either because of the embedded commas in quotes.

Do I need to do something before the data flow? Do I need to do custom code similar to undouble (I tried modifying undouble but using unqualified fields caused the source file to not like the data and go red)? Should the row be read as one field and parsed?

Thanks in advance for any help you can give!

View 12 Replies View Related

SQL Server 2008 :: Parsing Unstructured CSV File?

Oct 1, 2015

I have a CSV file with roughly 6 million rows. The file is unstructured; that is, some rows have 5 fields, others have 15, and there are as many 50 fields in one row.

I am using bulk insert to read the entire file into a table in database, with each row being a database record. With that, I have one column that contains a row of comma delimited fields. All fields are character string and I want to find a quick way of parsing each row and placing each comma-delimited value in a column. For example:

CREATE TABLE MyTable
(
CSVString varchar(1000),
C1 varchar(20),
C2 varchar(20),
...
C50 varchar(20),
)

Column CSVString contains the a CSV row (I don't know how many filelds (no. of commas + 1) in the row, but if the row contains 10 fields, I need to populate columns C1-C10. If the row has 15 fields, I populate columns C1-C15.

How can I do this in a very efficient way? I tried CTE but performance was not very good.

View 8 Replies View Related

Help, Fairly Complicated File Parsing Issue

Jun 7, 2007

Hi,



I have a situation where I'm having to extract key data from a financial file. Problem is, the columns are not nice and tidy.



Basically the file looks like this:



row 1: "788","Company","OPENING BALANCE:", 2084587.76
row 2: "313947","04/01/07","3","CS","FF", 170.00,"AZT","XYC INC", 20.8, 351.00
row 3:"788","06/06/07 CLOSING BALANCE:", 206203893.03



So, I'm going to need to get the OPENING BALANCE and CLOSING BALANCE figures, as well as all the data in between, ie) row 2 through n.



Does anyone have an example of a script that can be used for extracting very specific values from a file?



I have a script that checks for incomplete rows, but it is not sophisticated enough for this situation.



Thanks much




View 9 Replies View Related

Flat File Source Column Parsing Error

May 12, 2006

Hello All,



I have come across this issue with the Flat File Source when the delimiter is set to a comma.

"""KAILUA KONA,HI""","CA",

In the data snippet above and with the setting of using a comma as a column delimiter

and a " as the text qualifer.

the data will be parsed in this fashion:

"""KAILUA as a column:

HI""" as a column

CA as column

when it should be

"KAILUA,HI" as a column

CA as column.



Is there a way to let the Flat File Source to let it know not to parse the data in multiple quotes ?



Thank you

Eric Flores

View 5 Replies View Related

Text Inserting

Oct 21, 2007

I fixed all the free text stuff but now I have another problem. I created a web app one page to enter data and anoter to display it. I have a multiple row text box and if you enter something with returns it it when you display the data it ignores the returns and puts everything on the same line  BTW the column type is text
 
example
 
this
"want to hest this out.
========================
 
now I am going to put in some test data and see how it comes out
 
finished"
 
will come out like this
"want to hest this out.========================now I am going to put in some test data and see how it comes out finished"
 
how do you fix this?

View 16 Replies View Related

Flat File Source - If An Error Occurs, Continue Parsing The Remaining Columns In The Row Before Failing

Jan 14, 2008

Hello everyone,


I have a package that extracts data from a Flat File. If any errors or truncation occur during the extraction of the input data, the package should fail. All fields that have erroneous values should be reported in the log file.


My Solution:
- I have created a Data Flow Task that contains a Flat File Source Adapter and a dummy destination.

- I have left the default "Error Output" configuration of the Flat File Source adapter, namely if a truncation or an error occur for a certain column, then the reaction is "Fail Component".


Problem:
This configuration gives me only the first erroneous column in the row being processed.


Question:
Is it possible to make the Flat File Source adapter continue parsing the current row before it fails? This way, I would be able to get all the erroneous columns in the row in one shot.


Thanks in advance...
Samar

View 6 Replies View Related

Inserting XML String (Text)

Feb 19, 2004

Hi SQL Gurus,

I have an XML question to which I am kind of new.

I have an XML string that I get by using a query similar to the following....

SELECT 1 AS Tag,
NULL AS Parent,
NULL AS [Zip_ROOT!1!],
NULL AS [Zip!2!Value!element],
NULL AS [Zip!2!State!element]
FROM ZipCodes
UNION
SELECT 2,
1,
NULL,
Zip,
State
FROM ZipCodes where id < 3
FOR XML EXPLICIT

This returns an XML string, which is very long. I need to take this string and INSERT/UPDATE to a Text column in a table.

Any kind of help in this regard will be grately appreciated.

Regards,
Varma

View 4 Replies View Related

No @@IDENTITY When Inserting Text

Apr 26, 2002

I have an odd issue with a stored procedure using output parameters and a database text field.

I am using custom VB6 COM+ objects in the middle tier and SQL Server 2000 on the back end. The stored procedure has an int as it's first parameter; it is an output parameter which gets set to @@IDENTITY after the data is written to the tables. The last parameter is a text.

The COM+ object executes the proc; ADO 2.7 incorrectly identifies the text parameter as an adVarChar, so I explicitely convert it to an adLongVarChar when I detect that the incoming data is > 8000 characters.

The proc writes the data correctly to the database in all cases. However, when the data is > 8000 characters, SQL doesn't appear to correctly set the output variable. No errors are generated, I simply don't get any value written to the variable. I've searched through MS's documentation, but can't seem to find anything on this issue.

Any help or thoughts are appreciated.

View 1 Replies View Related

Text Inserting Problem

Oct 21, 2007

I fixed all the free text stuff but now I have another problem. I created a web app one page to enter data and anoter to display it. I have a multiple row text box and if you enter something with returns it it when you display the data it ignores the returns and puts everything on the same line BTW the column type is text



example



this

"want to hest this out.

========================



now I am going to put in some test data and see how it comes out



finished"



will come out like this

"want to hest this out.========================now I am going to put in some test data and see how it comes out finished"



how do you fix this?

View 7 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved