Transpose Source Data From A System Via Metadata Lookup Table Into Destination Table
Apr 1, 2014
I am stuck on finding a solution to transpose source data from a system via a metadata look-up table into a destination table. I need a method to transpose/pivot the source data into columns (which are by various data-types). The datatypes for each column are listed in a metadata table.
Source Data Table:
Table Name: Source
SrcID AGE City Date
01 32 London 01-01-2013
02 35 Lagos 02-01-2013
03 36 NY 03-01-2013
Metadata Table:
Table Name:Metadata
MetaID Column_Name Column_type
11 AGE col_integer
22 City col_character
33 Date col_date
Destination table:
The source data to be loaded into the destination table(as shown below):
Table Name: Destination
SrcID MetaID col_int col_char col_date
01 11 32 - -
01 22 - London -
01 33 - - 01-01-2013
02 11 35 - -
02 22 - Lagos -
02 33 - - 02-01-2013
03 11 36 - -
03 22 - NY -
03 33 - - 03-01-2013
View 7 Replies
Jul 23, 2005
I'm playing around with the following tables and need some advice.TablePeople (5000 records)peopleIDsexageraceTableExam (5000 records)examIDdatepeopleIDTableQuestions (5000 records)questionIDquestionFieldname (QFN0001,QFN0002,QFN0003, ... ,QFN5000)TableAnswers (25,000,000 records)examIDquestionIDanswerSQL Query Statement Needed!Query the database to return fields from TablePeople and TableAnswers(from 1 to 255 questionFieldnames)Example Query and return data from:ExamID,sex,age,race,QFN0001,QFN0002,QFN0003,QFN000 5,QFN0007,QFN0011I know how to join the tables and return the following recordsetExamID, sex, age, race, QuestionFieldName, AnswerID100001,M,40,White,QFN0001,0100001,M,40,White,QFN0002,5100001,M,40,White,QFN0003,6100001,M,40,White,QFN0005,3100001,M,40,White,QFN0007,5100001,M,40,White,QFN0011,4100002,M,55,White,QFN0001,1100002,M,55,White,QFN0002,4100002,M,55,White,QFN0003,3100002,M,55,White,QFN0004,4100002,M,55,White,QFN0007,5100002,M,55,White,QFN0011,6etc105000,M,48,White,QFN0001,5105000,M,48,White,QFN0002,4105000,M,48,White,QFN0003,6105000,M,48,White,QFN0005,3105000,M,48,White,QFN0007,4105000,M,48,White,QFN0011,5We would like the resulting table to look like the following:ExamID,sex,age,race,QFN0001,QFN0002,QFN0003,QFN000 5,QFN0007,QFN0011100001,M,40,White,0,5,6,3,5,4100002,M,55,White,1,4,3,4,5,6etc105000,M,48,White,5,4,6,3,4,5Any suggestions appreciated.Gregory S. MoyInformation Processing ConsultantEpiSense Research ProgramDepartment of Ophthalmology & Visual SciencesUniversity of Wisconsin - Madison
View 1 Replies
View Related
Mar 30, 2007
Hi
I have a "transaction" table of diagnostic tests, each row consisting of one test performed on one person as follows:
Patient Test Time
ID ID of Test
--------------------------
1 1 May 1
1 2 May 1
1 4 May 2
2 5 etc.
3 1
3 3
4 4
4 2
I'd like to transpose this into a table with one row per patient, so I can then describe, for each patient, the sequence of tests that he or she had, e.g.
Patient T1 T2 T3 T4 T5 TEST_SEQUENCE
1 1 2 4 - - "1 2 4"
2 5 - - - - "5"
3 1 3 - - - "1 3"
4 4 2 - - - "4 2"
How do I do this in SQL, preferably without having to specify the maximum number of tests? (In a statistical package we use, this is called a table "reshape" operation)
Thanks in advance.
View 7 Replies
View Related
Oct 31, 2007
We did some "at scale" fuzzy lookup tests today and were rather disappointed with the performance. I'm wanting to know your experience so I can set my performance expectations appropriately.
We were doing a fuzzy lookup against a lookup table with 25 million rows. Each row has 11 columns used in the fuzzy lookup, each between 10-100 chars. We set CopyReferenceTable=0 and MatchIndexOptions=GenerateAndPersistNewIndex and WarmCaches=true. It took about 60 minutes to build that index table, during which, dtexec got up to 4.5GB memory usage. (Is there a way to tell what % of the index table got cached in memory? Memory kept rising as each "Finished building X% of fuzzy index" progress event scrolled by all the way up to 100% progress when it peaked at 4.5GB.) The MaxMemoryUsage setting we left blank so it would use as much as possible on this 64-bit box with 16GB of memory (but only about 4GB was available for SSIS).
After it got done building the index table, it started flowing data through the pipeline. We saw the first buffer of ~9,000 rows get passed from the source to the fuzzy lookup transform. Six hours later it had not finished doing the fuzzy lookup on that first buffer!!! Running profiler showed us it was firing off lots of singelton SQL queries doing lookups as expected. So it was making progress, just very, very slowly.
We had set MinSimilarity=0.45 and Exhaustive=False. Those seemed to be reasonable settings for smaller datasets.
Does that performance seem inline with expectations? Any thoughts to improve performance?
View 4 Replies
View Related
Feb 18, 2015
I am trying to loop through a table which has table metadata and create a:
- 'STORED PROCEDURE'
- This will SELECT all the data in the table and add a hashed column at the end
- Each table has a unique ID
-
Instead of a cursor would there be a set-based approach to achieving this?
METADATA TABLE :
IDTableNameColumnNameHashByteCalculation
111dbo.TableAColA CAST(ISNULL(LEFT(CONVERT(VARCHARColA, 120), 10),'NA') AS varchar) + '|' +
111dbo.TableAColBCAST(ISNULL(LEFT(CONVERT(VARCHAR,ColB, 120), 10),'NA') AS varchar) + '|' +
111dbo.TableAColCISNULL(ColC,'NA') + '|' +
111dbo.TableAColDISNULL(ColD,'NA') + '|' +
222dbo.TableBColAACAST(ISNULL(LEFT(CONVERT(VARCHAR,ColAA, 120), 10),'NA') AS varchar) + '|' +
222dbo.TableBColBBISNULL(ColBB,'NA') + '|' +
222dbo.TableBColCCISNULL(ColCC,'NA') + '|' +
From the above data I want to generate:
SELECT
ColA
,ColB
,ColC
,ColD
, (CAST(ISNULL(LEFT(CONVERT(VARCHARColA, 120), 10),'NA') AS varchar) + '|' +
[Code] ....
View 2 Replies
View Related