Tracking Forums, Newsgroups, Maling Lists
Home Scripts Tutorials Tracker Forums
  HOME    TRACKER    Visual Basic

Set Precision In Doubles


how can I set the precision in vb? My program does some calculations and I
don't want all the decimal fact, I want any result to be 12 numbers
no matter if it's 1,23456789012 or 123456,789012...however, because I use
variable with type Long, the numbers are bigger...

any ideas?

View Complete Forum Thread with Replies

See Related Forum Messages: Follow the Links Below to View Complete Thread
Doubles: Accuracy/Precision
It seems that VB stores more digits than it shows when displaying doubles. Here is an illustration, from the Immediate Window:

VB Code:
?2/3 0.666666666666667 ?0.666666666666667=(2/3)False?0.6666666666666666=(2/3)True

Do you agree that VB is hiding digits? If so, is there a way to get around this?

Val() And Doubles...
Hi! This is the problem:

Valtemp = "1.00000000000236"
MsgBox (Val(Valtemp))

Why does the msgbox give 1 and not 1.00000000000236? It looks like Val() operator rounds the number!! Is there an alternative?


Can you read a value stored to a file in QuickBasic using the MKD command.

Is it possible to decode in VB ? Need help !

Big Endian Doubles

I am having problems converting a double (8 bytes) from intel to unix format for writing to a file. Can anyone give me any ideas what's going wrong with this code?

Private Type udtTwoLongs
Long1 As Long
Long2 As Long
End Type

Private Declare Function HostToNetLong Lib "ws2_32.dll" Alias "ntohl" (ByVal v_lng32BitLittleEndian As Long) As Long
'Converts an Intel (Host) 32 bit byte order to Unix (Net)

Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (Destination As Any, Source As Any, ByVal Length As Long)
'Copy bytes of memory from one location to another

Public Function WriteDouble(ByVal dblValueToWrite As Double, Optional ByVal blnIEEE As Boolean = False)
Dim typTwoLongs As udtTwoLongs
If blnIEEE Then
'Convert to bigendian
CopyMemory typTwoLongs, dblValueToWrite, 8
typTwoLongs.Long1 = HostToNetLong(typTwoLongs.Long1)
typTwoLongs.Long2 = HostToNetLong(typTwoLongs.Long2)
CopyMemory dblValueToWrite, typTwoLongs, 8
End If

'Write out the value
Put #m_intFileNumber, , dblValueToWrite
End Function



VB Bug - Rounding Doubles Or Something?
The weirdest thing happens in my Application. It is extremely sensitive to calculations, and well, things go absolutely crazy! I have particles with exact same mass. I am calculating gravity F = mg. THen in another loop, i loop them all again and do a = F/m to get acceleration. I run the application, and the particles fall to the ground.

BUT!!!! If i start 2 particles at y = 500 and let em fall to y = 0, The Difference between their y values is like 1.7E-15, then it goes to -1.7E-15, then 0.2E-16, and just like that... This part i absolutely dont understand. They differ by that amount, and yet i increment them in the EXACT same way...

Anyone has any ideas what the hell is happening?

Rounding Doubles
i have a problem. i need to make my double variables rounded up to the nearest single digit. no, i cannot put these into integers, or even singles, the numbers have to do with up to like 13 million.

is there a way to round the double variable up, getting rid of the decimals, so i have a whole number? (this would be alot easier if integers didnt have limitations...)

Mod Function For Doubles?
I'm trying to do the following mod

2.3 mod .056

I get a division by 0 error. I assume what's happening is that .056 is being rounded to zero.

I need to somehow obtain the answer .004

Any ideas??

Watches And Doubles
I have the following code and the following problems.

Dim i as double
Dim j as double

for i = 0 to 1 step .01
for j = 0 to 1 step .5

' have some other function calls
DebugFile.writeline(i & " " & j)
next ' end for j
next ' end for i

Main Problem:

When "i" gets up to .8, the step is screwed up. I should note that all the numbers in the for loops are variables.....I'm simply setting them in this example to what breaks. What I see in my "DebugFile" is

.80 ' goes through the j steps

I'm thinking maybe one of my function calls in which I pass i is corrupting i. However I can't prove it because:

When I try and do a watch where I break when true with following it never breaks:

i = .8
i= cdbl(.8)

Anyone have any ideas?

Searching Doubles
I have an Access2K db that is connecting to an AutoCAD drawing. The X/Y coordinate values of objects in the drawing have been extracted to a DB.
I would like to assign a unique ID in the db based on other information found in the drawing. So I have created a SELECT statement to match the X/Y coords with an object in the drawing then place the new ID I created with that recordset.

There are 3 records that continually elude the SQL statement and a new duplicate record is created. If I trap the SELECT statement, copy it and go to the db directly, and create a query in the SQL statement builder it wont find it either. However when I run a "find duplicates" query on the table after the new duplicate record has been created they both come up.

The Coordinates are stored in the db as a DOUBLE with the decimal places set to AUTO. I have looked at other recordsets that were successful and can find no difference in the format.

I have pasted the code that is performing the search below:

Public Sub GetAtt(NewBlock As AcadBlockReference)
'On Error GoTo errhandler

Set rsAtt = New ADODB.Recordset

'loops through all the attributes of the block to do an initial match
Attributes = NewBlock.GetAttributes
dXcoord = NewBlock.InsertionPoint(0)
dYcoord = NewBlock.InsertionPoint(1)

For i = LBound(Attributes) To UBound(Attributes)
sAtt = Attributes(i).TagString

Select Case sAtt
Case "RTE"
vRoute = Attributes(i).TextString
Case "QUAN"
vCable = Attributes(i).TextString
Case "PED#"
vNumber = Attributes(i).TextString
End Select
sAtt = ""
Next i

sSQL = "select * from Blocks where [XCoord]=" & dXcoord & " and [YCoord]=" & dYcoord & ""

'Opens the DB for editing, and querries based on criteria above
rsAtt.Open sSQL, sConn, adOpenKeyset, adLockOptimistic, -1

If Not rsAtt.EOF Then
'If a record is found then DB is updated ID, Handle and Blockname are added to record
If IsNull(rsAtt!id) Then rsAtt!id = sDwgName & vObjId
If IsNull(rsAtt!Handle) Then rsAtt!Handle = vObjId
If IsNull(rsAtt!BLOCKNAME) Then rsAtt!BLOCKNAME = bObj.Name
Debug.Print rsAtt!id

'loops through the attributes and writes the value to the DB
For i = LBound(Attributes) To UBound(Attributes)
sAtt = Attributes(i).TagString
Select Case sAtt
Case "PED#"
If IsNull(rsAtt![PED#]) Then rsAtt![PED#] = Attributes(i).TextString
Case "UNIT"
If IsNull(rsAtt!UNIT) Then rsAtt!UNIT = Attributes(i).TextString
Case "YR"
If IsNull(rsAtt!YR) And Attributes(i).TextString = "" Then
rsAtt!YR = Null
rsAtt!YR = Attributes(i).TextString
End If

Case "TD"
If IsNull(rsAtt!TD) Then rsAtt!TD = Attributes(i).TextString
Case "QUAN"
If IsNull(rsAtt!QUAN) And Attributes(i).TextString = "" Then
rsAtt!QUAN = Null
rsAtt!QUAN = Attributes(i).TextString
End If
Case "RTE"
If IsNull(rsAtt!RTE) And Attributes(i).TextString = "" Then
rsAtt!RTE = Null
rsAtt!RTE = Attributes(i).TextString
End If
Case "MCST"
If IsNull(rsAtt!MCST) And Attributes(i).TextString = "" Then
rsAtt!MCST = Null
rsAtt!MCST = Attributes(i).TextString
End If
Case "TCST"
If IsNull(rsAtt!TCST) And Attributes(i).TextString = "" Then
rsAtt!TCST = Null
rsAtt!TCST = Attributes(i).TextString
End If
End Select

Next i

Set rsAtt = Nothing

'If a current record does not exist a new record is created and the data inserted

If IsNull(rsAtt!id) Then rsAtt!id = sDwgName & vObjId
If IsNull(rsAtt!Handle) Then rsAtt!Handle = vObjId
If IsNull(rsAtt!BLOCKNAME) Then rsAtt!BLOCKNAME = bObj.Name
If IsNull(rsAtt!XCoord) Then rsAtt!XCoord = dXcoord
If IsNull(rsAtt!YCoord) Then rsAtt!YCoord = dYcoord

For i = LBound(Attributes) To UBound(Attributes)
sAtt = Attributes(i).TagString
Select Case sAtt
Case "PED#"
rsAtt![PED#] = Attributes(i).TextString
Case "UNIT"
rsAtt!UNIT = Attributes(i).TextString
Case "YR"
If IsNull(rsAtt!YR) And Attributes(i).TextString = "" Then
rsAtt!YR = Null
rsAtt!YR = Attributes(i).TextString
End If
Case "TD"
rsAtt!TD = Attributes(i).TextString
Case "QUAN"
If IsNull(rsAtt!QUAN) And Attributes(i).TextString = "" Then
rsAtt!QUAN = Null
rsAtt!QUAN = Attributes(i).TextString
End If
Case "RTE"
If IsNull(rsAtt!RTE) And Attributes(i).TextString = "" Then
rsAtt!RTE = Null
rsAtt!RTE = Attributes(i).TextString
End If
Case "MCST"
If IsNull(rsAtt!MCST) And Attributes(i).TextString = "" Then
rsAtt!MCST = Null
rsAtt!MCST = Attributes(i).TextString
End If
Case "TCST"
If IsNull(rsAtt!TCST) And Attributes(i).TextString = "" Then
rsAtt!TCST = Null
rsAtt!TCST = Attributes(i).TextString
End If
End Select
Next i
Set rsAtt = Nothing
End If

Exit Sub

End Sub

Is there anything special I should know about searching Double Data Types?
Thanks in advance for your assistance


Detect Doubles?
Just wondering, is it possible to split a text box for exact doubles? the doubles are going to be this plausible? itd make my life alot easier

btw, the 'find' button in wordpad wont cut it

Doubles - Overflow
Does anybody know how not to get "Overflow" in the last line of this code??

VB Code:
Dim dLevel As Double    Dim dBitWiseAnd As Double     'Just in case, the value of that field is rs!DAEGR_Level=2147483648     dBitWiseAnd = CDbl(rs!DAEGR_Level) And dLevel

I'm Scared Of Doubles!
Fire up VB, choose standard .exe, and paste this code into the code window for it:

Option Explicit
Private Sub Form_Load()
Dim i As Double
For i = 5 To 6 Step 0.1
Debug.Print i '& " " & Int(i * 100) / 100 & " " & j
Unload Me
End Sub

run it, and observe. Cool.

ok, now change the loop as follows:

For i = 0 To 6 Step 0.1

Whats up with the last line of output? Change the start and end range in the For line and watch the `random` results.

I`m aware of rounding errors, but what is THIS all about?

Comparing Doubles
I have code that goes something like this:

Dim Double1 As Double
Dim Double2 As Double

'...Code which allocates values to Double1 and Double2 based
'on user input

If Double1 <> Double2 Then
MsgBox "Double1 does not equal Double2"
MsgBox "Double1 equals Double2"
End If

I have noticed that the double comparison does not work reliably. ie.Sometimes Double1 equals Double2 yet the first msg box appears.

Changing the comparison to:
If CStr(Double1) <> CStr(Double2) Then

seems to solve the problem but can anyone explain this behaviour of double variables?

Dissecting Doubles
Cheers mate, I was on the same kinda track.

Are you really from Peterborough too?

Characters To Doubles
I'm receiving an array of bytes (characters) from a winsock connection, groups of 8 of these bytes represent doubles.
How can I take these 8 bytes and form the appropriate double?
And conversly, how can I breakdown a double into its 8 bytes to send?


Do Doubles/Singles Really Look Like This?
Do to another Query, I started trying to determine the algorithm for conversion to Double, and the appearance of doubles in memory.

Not wanting to reinvent the wheel, I checked a few Web Sites, but was unable to find a precise description of double variables (The IEEE site did not have a readily available description, even though that organization is responsible for the specifications).

I developed a simple VB application which accepted some input data, converted to Doubles, singles, and Longs (if possible), and wrote to a file. I then investigated the file using a Hex Editor (The one supplied with Ztree).

Here are some results. Do you believe them? If not, can you provide the Doubles and singles corresponding to my Decimal values? Better yet, do you know the algorithm which describes conversion of decimal data to double/single? I think 9 or 10 bits of a double make an exponent and the rest comprise a binary value.

Decimal 450,123 (Hex 6D,E4B) results in Long: 4BDE 0600
Corresponding Double: 0000 0000 2C79 1B41
Corresponding Single: 60C9 DB48

Integers and Longs arrange the digits differently than a human being would, but the algorithm is fairly obvious. It boggled my mind when I compared doubles with correspondiing Singles, and found no obvious correlation. This makes me wonder about possible bugs in my application and/or makes me suspect my Hex Editor.

Decimal .0001 (Hex .0006 8DB8 BAC7 1CA, I think).
Corresponding double: 2D43 1CEB E236 1A3F
Corresponding Single: 17B7 D138

If interested, I have other data (, including some powers of two, integers 1-9, fractional powers of two (EG: 1/2, 1/4, 1/8), and other values. These examples provide some clues, but I have not yet figured out what is happening.

I expected to see something obvious, and am wondering if my VB application or the Hex Editor have errors.

Thanx for help --- Gouverneur Cadwallader (Dinosaur from prehistoric Mainfame era)

One Question About Doubles
Dear all
I have a problem which I cannot understand clearly.
Say you have a number (984,4) stored in a Table Field (declared as Double) of a Database.
In my VB program I do as follows:

Dim A as double
A = rs!Field

If I see the contents of A after the assignment, it is 984,40024411....
Is it an error it is correct like this.
In practice can this cause error on the calculations thereafter?

Thank you for yr kind help and attention
Best Regards


Get The Bitstream Of Doubles
Hi all!

I need to convert some Double Data into appropriate 8-Bit parts as I must transmit them over a 8Bit interface.

How can I access the binary Data of a Double in VB?

Thanks in advance


Fortran Doubles
Hi there, hope someone can help me. I am trying to use VB to read in some doubles from a file. The doubles were written to the file by fortran, and on a different platform. I can read them into a C program and they look like the right numbers, so I'm pretty sure they're not byte-swaped etc. So my problem is how to get VB to take these 8 bytes and call them a double. Does anyone know how to do this?


Phil Drew

Problem With Doubles Format
hey again,

i got a problem trimming down the length of a number... a double.

i want it to be like 82.27 but what i actually get is 82.2748489

heres the code...

'General Declarations:

Dim DiskUsageInfo(0 To 3) As String
Dim TotalSpace As Double
Dim FreeSpace As Double
Dim SpaceUsed As Double
Dim UsedInSession As Double

Private Sub CalcInfoAndDisplay()

lbl_TotalSpace = " Total Number Of Bytes: " & Format$(TotalSpace / 10240, "###.##") / 102400 & " Gb"
End Sub

This function should convert TotalSpace in bytes to TotalSpace in Gb's

is there something wrong with my formatting?

in waiting


EDIT: just to note.. the variables are 11 or 12 digits long. might be part of the problem

EDIT 2: I changed the general declaration variables to Currency and removed the division and the formatting worked. how could i get around that?

Strangeness When Comparing Doubles
Can anyone tell me why VB6 thinks a is greater than b (and how to fix it):

Dim a As Double
Dim b As Double
Dim factor As Double

factor = 25.4

a = 487.68
b = CDbl("19.2") * factor

If a > b Then
Debug.Print "greater"
End If
a should not be greater than b !


Serial # Problem For Doubles
I have a column where I enter my serial numbers for different products.

I am not sure if I have doubles or not unless I write a script to check the 1st with the entire column, and then the 2nd.. etc..
This is a very time consuming function.

Is there an easier way to do this without having to actually run the macro?
Such as when I "enter" a serial # that is the same as a previous "#" in that column... could I have it pop up a MSG BOX saying "Serial Number already exists"?


Scanner Doubles Output
I have an app that uses a scanner as an input device. Occasionally, each scanned number repeats itself, i.e. 1234 will scan 11223344. Can anyone please explain this? Thanks very much.


How Do I Stop Doubles In A List Box?
How do I stop the same thing being added to a list box twice?

Overflow Error With Doubles
Hey... I've got a double variable and it's being fed the following equation

35000 + 1700 * (nIndex + 1)

Now in this case, nIndex is 19, which gives us 69000. I was under the impression that doubles could store numbers greater than this. If I'm wrong, what is the maximum number a double can store?

Interesting Problem With Doubles
I have the following function:

Function triggerEncode(value As Double) As Byte
value = value / 5
Dim findVal As Double
findVal = value
Debug.Print findVal

'Select Case findVal
' Case 1: triggerEncode = &HF0
' Case 0.937: triggerEncode = &HE0
' Case 0.875: triggerEncode = &HD0
' Case 0.812: triggerEncode = &HC0
' Case 0.75: triggerEncode = &HB0
' Case 0.688: triggerEncode = &HA0
' Case 0.625: triggerEncode = &H90
' Case 0.562: triggerEncode = &H80
' Case 0.5: triggerEncode = &H70
' Case 0.437: triggerEncode = &H60
' Case 0.375: triggerEncode = &H50
' Case 0.312: triggerEncode = &H40
' Case 0.25: triggerEncode = &H30
' Case 0.188: triggerEncode = &H20
' Case 0.125: triggerEncode = &H10
' Case 0.062: triggerEncode = &H0
' Case 0.042: triggerEncode = &H14
' Case 0.021: triggerEncode = &H4
' Case Else: triggerEncode = &H4
'End Select
If findVal = 0.042 Then triggerEncode = &H14
If findVal = 0.021 Then triggerEncode = &H4

End Function
when i pass the function .105 nothing happens using either my select case or if statements! You can uncomment the select statements. Put a break on the first line of the function to see what I am talking about. when I step through it I get .105/5 = .021 and everything goes as normal but it compeletely skips over my case 0.021. Also if I pass it 0.21 which comes out to .042 it also fails to set the trigger encode properly. For all other correct values it works. Like .31 trigger the .062 event and such... Please help I don't konw what is going on !!

Remove Doubles In SQLDB
I have to work on a Customer database on SQL Server without an index on CustomerID. Unfortunatly I have found some doubles in the database. So I would like to add an index to ClientID but of course I have to get rid of the doubles first....
But there are about 14000 records in it so is there a way with the DELETE statetement to get rid of the doubles? I want to keep 1 of each clientID
Thanks in advance for any assistance

CopyMemory And Arrays Of Doubles
How come this works:

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Form_Load()

Dim strA As String, strB As String

Dim dblX(1) As Double, dblY(1) As Double

dblX(0) = 123.456
dblX(1) = 987.654

strA = Space$(16)

CopyMemory ByVal StrPtr(strA), dblX(0), 16

CopyMemory dblY(0), ByVal StrPtr(strA), 16

Debug.Print dblX(0); dblX(1)
Debug.Print dblY(0); dblY(1)

End Sub
It returns

123.456 987.654
123.456 987.654
But this does not.

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Form_Load()

Dim strA As String, strB As String

Dim dblX(1) As Double, dblY(1) As Double

dblX(0) = 123.456
dblX(1) = 987.654

strA = Space$(16)

CopyMemory ByVal StrPtr(strA), dblX(0), 16

Open "CM.Fil" For Random As #1 Len = 18

Put #1, 1, strA

Get #1, 1, strB

CopyMemory dblY(0), ByVal StrPtr(strB), 16

Debug.Print dblX(0); dblX(1)
Debug.Print dblY(0); dblY(1)

End Sub
it returns
123.456 987.654
1.72448571545298E-307 1.72448571545298E-307
The only difference is that the string is written to a file first - then gotten back into a different string variable and that new string variable is COPYMEMORY'd into DBLY() array.

I'm guessing it has to do somehow with how strings are unicode - but I'm just guessing here...

I'm also confused as to why I had to make the LEN of the RANDOM file 18 when the recordsize is 16. I thought these were fixed length records - is it really storing cr/lf delimiters internally?

Sorting Doubles *RESOLVED*
I have a text box for a user to input a series of numbers, e.g.



I then put these into an array with the Split function

VB Code:
Dim arrFreqs arrFreqs = Split(txtLine.Text, vbCrLf)

I realise that the Split function returns strings so I convert them all to doubles

VB Code:
Dim i As Integer For i = 0 To UBound(arrFreqs)    arrFreqs(i) = CDbl(arrFreqs(i))Next

Now whenever I pass this array to a sorting algorithm e.g copied from a post by IROY55

VB Code:
Public Sub SortArray(varArray As Variant)'=========================================Dim i As LongDim j As LongDim temp As Double     Debug.Print "Start Time: " & Format(Now, "hh:nn:ss")    For i = 0 To UBound(varArray) - 1        For j = i To UBound(varArray)            If varArray(i) > varArray(j) Then                temp = varArray(j)                varArray(j) = varArray(i)                varArray(i) = temp            End If        Next    Next    Debug.Print "End Time: " & Format(Now, "hh:nn:ss") End Sub

It sorts in this order



This seems to be either treating the values as strings or just sorting the first character.

I have looked up several other sorting algorithms, QSort, Shellsort etc
but they are more complicated and I'm not sorting many numbers.

Can someone explain where I am going wrong?

Find Doubles In A Accesstable
What is the best way to check if I have doubles in a accesstable. Also, if I have som fields in a table which are identical, and then show the find identical filed in a form.

Tree View.. Doubles... How Do I ...
How do i make sure the treeview control wont except doubles. LIke if i have the number 1 - 10 there i do i make sure they cant enter 1 there. BUt these are names not numbers like "Jason" how do i make sure they cant enter Jason any more ?

Removing Doubles In A List
I wrote some code to remove repeated items in a list, but the only way I could get it to work is very slow and ineffective for very large lists. Please let me know how I can make this one work better. BTW the variables X and Y were dimmed already in Generals

For X = 0 To frmMain.lstURLS.ListCount - 1
For Y = 0 To frmMain.lstURLS.ListCount - 1
If frmMain.lstURLS.List(X) = frmMain.lstURLS.List(Y) And X &lt;&gt; Y Then
frmMain.lstURLS.RemoveItem (Y)
GoTo Restart:
End If
Next Y
Next X

Thanks for taking a look.

Doubles And Ghost-Decimals
Compute the following (it's an example, other numbers do the same) :

346.48 - 346

The result may not compare with the user-input value 0.48 :

because 346.48 - 346 does not give 0.48 but something like 0.48000000000018 (not sure for how much zeros)...

any comment/idea ?

Rounding Doubles And Displaying...Help Needed

I'm writing a code where I have to display the calculated double in the form "X= ####", no decimals and rounded. Here is my questions:

I'm using

dim X as double
X = Round(my expression, 0)
X = Format(X,####)

1. For 2.46, It prints 2. But it should be 3 if the number is rounded correctly. Another value is 1230.6 is being printed as "1231" which is fine.

2. I want X be printed as:
" 2" if X = 2, three spaces before 2
" 30" if X=30, two spaces before 30
" 300" if X = 300, one space before 300
Can I do this using strings? or is there any way to format my double and keeping it in 4 character length/size? plz how?
If my double is 12366, How can I round it to 1237?

3. I'm using VB from Excel and using "Cells(row,column)" to output my values. Is there a way that I can specify the sheet or creat a new sheet(excel tap) and send all my values there?

Thanks in advance

Finding Doubles In Access-table
I have a database with about 70000 entries. Lots of them are double and I would like to clean the db from these doubles.
Now, does someone know how to check for double entries in an Access database and how to remove one of them?

2#, 180# Necessary For Precision?
I have an application which gets results which are not as precise as I expected.

I am looking for errors in some formulae, but I am wondering about some integer constants. For example, I have a formula like
Angle=Pi * Longitude / 180
Angle, Pi, & Longitude are Doubles. Should I use "180#" to force the integer constant to be a Double? Will VB get more precision if I do that?

How Do I Extract Doubles, Longs, Integers Etc, From A BYTE Array
OK, I have a byte array that contains a stream of binary data packets. Some are doubles(8bytes), some longs(4 bytes) etc. For this example lets say the first 8 bytes of my array is a binary double value the next 4 bytes are a binary long integer, the next 2 are a binary integer. How do I assign the first 8 bytes to my variable dDblx, the next four to my variable lLngx and the next two bytes

to my variable iIntx?

I can not use a Type struct because the bytes are not always in this order. Any suggestions?

Thanks, Terry

Extended Precision
Are extended precision variables avaliable in vb6.0, or only in .net?

What is the highest precision I can obtain using vb 6.0?

Precision Point
i want to display the value from corresponding changes in hscroll in text box

so i write

Text1.Text = (HScroll1.Value / 255) * 100

but the display value in text1, has too many precision point, how can i set it to display until 3 precision point, like 0.392 instead of 0.392156862.....

Decimal Precision
I'm creating a simple trig calculator. Because I can't extend pi to infinity, some of my results are a little inaccurate i.e.: 4.99999 should be 5.00.
How can I set the decimal precision to say.. round to the nearest 3 decimal places? Like, instead of 4.99999 round to 5.000?

I have tried selecting the text box and adjusting the format, but it has no effect. Where am I going wrong?

Thanks to all!


Precision Problem????
I have the following (I've simplified some things to demonstrate my point):

Dim LintI As Long
Dim Total As Long
Dim Count As Long
Dim fract As Double

Count = 0
Total = 1048576
For LintI = 0 To Total
Count = Count + 1
fract = CDbl(Count) / CDbl(Total)
next LintI

"fract" is obtaining strange values. Any ideas?

The Precision Is Invalid?!?!?!
Hello All

I am getting a rather frustrating error here. I am using MSDE, and VB 6.0. keep getting an error that the precision is invalid. I have done some research on this error, and have learned that it is something with the size or datatypes of the parameters being passed. However I can not see this error. My suspicion is that it is setting the Date to Null, or not passing the size of lt_descritpion which is a TEXT colum. I am really not too sure.

I have created the following stored procedure.

CREATE PROCEDURE dbo.insert_items
@vc_name VARCHAR(100),
@lt_description TEXT,
@vc_supplier VARCHAR(100),
@n_category_id BIGINT,
@n_our_price DECIMAL,
@n_sale_price DECIMAL,
@n_qoh INTEGER,
@n_on_order INTEGER,
@n_re_order_level INTEGER,
@d_last_ship_date DATETIME,
@d_next_ship_date DATETIME,
@vc_discontinued VARCHAR(5)
INSERT INTO tbl_items(vc_name, lt_description, vc_supplier, n_category_id,
n_our_price, n_sale_price, n_qoh, n_on_order, n_re_order_level,
d_last_ship_date, d_next_ship_date, vc_discontinued)
VALUES(@vc_name, @lt_description, @vc_supplier, @n_category_id,
@n_our_price, @n_sale_price, @n_qoh, @n_on_order, @n_re_order_level,
@d_last_ship_date, @d_next_ship_date, @vc_discontinued);
And I have the follwoing function that posts to the stored proecure.

Public Function DB_AddNewRecord() As Boolean
' *** Setup error hanlder
On Error GoTo errHand

o_Cmd.CommandText = "insert_items"
o_Cmd.CommandType = adCmdStoredProc

o_Cmd.Parameters.Append o_Cmd.CreateParameter("@vc_name", adVarChar, adParamInput, 100, F.txtName.Text)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@lt_description", adVarChar, adParamInput, Len(F.txtDescription.Text), F.txtDescription.Text)
' *** Other vendor
If Not F.cboVendorsList.Text = "OTHER" Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@vc_supplier", adVarChar, adParamInput, 100, GetVendorID(F.cboVendorsList.Text))
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@vc_supplier", adVarChar, adParamInput, 100, F.txtSupplierOther.Text)
End If
' *** No category
If (F.chkNoCategory.Value = 1) Or (F.cboCategoryList.Text = vbNullString) Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_category_id", adNumeric, adParamInput, , -1)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_category_id", adNumeric, adParamInput, , GetCategoryID(F.cboCategoryList.Text))
End If
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_our_price", adDecimal, adParamInput, , CDec(F.txtOurPrice.Text))
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_sale_price", adDecimal, adParamInput, , CDec(F.txtSalePrice.Text))
' *** QOH
If F.chkQOH_NA.Value = 1 Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_qoh", adNumeric, adParamInput, , -1)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_qoh", adNumeric, adParamInput, , CInt(F.txtQOH.Text))
End If
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_on_order", adNumeric, adParamInput, , CInt(F.lblQuantityOnOrder.Caption))
' *** Reorder level
If F.chkReOrderNA.Value = 1 Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_re_order_level", adNumeric, adParamInput, , -1)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@n_re_order_level", adNumeric, adParamInput, , CInt(F.txtReOrderLevel.Text))
End If
' *** Last shipment date
If F.chkLastShipDate.Value = 0 Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@d_last_ship_date", adDBDate, adParamInput, , Null)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@d_last_ship_date", adDBDate, adParamInput, , CDate(F.dtLastShipDate.Value))
End If
' *** Next shipment date
If F.chkNextShipDate.Value = 0 Then
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@d_next_ship_date", adDBDate, adParamInput, , Null)
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@d_next_ship_date", adDBDate, adParamInput, , CDate(F.dtNextShipDate.Value))
End If
o_Cmd.Parameters.Append o_Cmd.CreateParameter("@vc_discontinued", adVarChar, adParamInput, 5, F.optDisco(0).Value)

DB_AddNewRecord = True

Exit Function
DB_AddNewRecord = False
Call GLOBAL_ERROR_HANDLER(Err.Number, Err.Description, "modProdServ", Err.Source, "DB_AddNewRecord()", Erl(), True, True)
End Function
Do not let the F.ITEMNAMES confuse you. I have created a golobal form OBJECT for this module so that I can refer to the form as F instead of the big long form name. This is forsure not the problem. I have tried it with out and get the same error.

The parameters are all in the correct order, and their Sizes match that of the table definition. SO this is very strange to me to be getting this error.

Any ideas? Thanks in advance

Precision Problem

I have a formula in VB6:

60429780.5861 * rs("exp_rate") / 100 * 3 / 366

where exp_rate is a sql field of size Numeric(12,8) with value 0.25. When I select it and put it to a recordset for calculation. I got the result


If I replace the field with constant (0.25 in this case):

60429780.5861 * 0.25 / 100 * 3 / 366

I got 1238.31517594467 only.

I just like to know, why the precision is different?
Indeed the first case is in my program, and this cause precision error when I try to insert this value to sql table where the max. precision support is 28 only.


Precision Problems
I have 2 variables both set up as Singles.

otHoursWorked = 0
tempOTPremium = 8.07

These are displayed correctly everywhere. After this command, however:

otHoursWorked = otHoursWorked + tempOTPremium

I get the following:


What is happening here? I have all kinds of Singles added and subracted together and I don't get this problem - Only with this instruction and variable.

This is about the 4th time I hit this instruction with problems at all.

What could be happening here?



Precision Uptime
I've been looking around for a precision uptime [thingy], but I can't find one..
Basically I want a function/API Call (or whatever) that returns a value that surpasses Microsoft's stupid GetTickCount 49 day thing..

Variable Precision
I have a question that I cant seem to solve. using long is all fine and good for storing the smaller numbers that people normally use in their programs, but what if I want to store a number like 10^30? is there any way to specify in vb the number of bytes a variable occupies? is it easy to work with such things? or do I need to design a library that will do these things?

Currency Precision
Is there a way to set the currency precision level to 2 after the . I mean like everey time I would put something in a currency variable it would round up automaticly. EX:

VB Code:
Dim MoneyVariable as Currency MoneyVariable= 1.2345Print MoneyVariable '---> 1.23 MoneyVariable=1.2356 Print MoneyVariable '-----1.24  


Precision Error
Hi !

I'm using VB6 and SQL Server as databse. I have a stored procedure (called from a command object) that receives a Decimal(9,2) as a parameter.

From VB6 I write:


Set Param = CMND.CreateParameter("@Price", adDecimal, adParamInput, , FormatNumber(CDec(Val(txtPrice.Text)), 2))



Set Param = CMND.CreateParameter("@Price", adDecimal, adParamInput, , CDec(Val(txtPrice.Text)))

But when I execute the command object I get the error number -2147467259 ("Precision is not valid").

Please help me !

Precision Smaller Then 1/20 Sec
Here is my problem : I'm trying to make a control scroll smoothly on my form but the problem is that I can't get a delay with a precision smaller then 1/20 of a secs (20 trigger per seconds)

I've tried setting a timer interval to 1 without succes. Then I tried a loop as follow :


Dim dLast As Double
Dim dNow As Double
Const Interval As Double = 1

dLast = Devel.GetCTime()
While True
dNow = Devel.GetCTime()
If (bStopLoop) Then Exit Sub
If (dNow >= dLast + Interval) Then
dLast = dNow
' Code for my scrolling here
End If

Devel.GetCTime return the current time as Double with milliseconds precision using the GetSystemTime API. Now if I set interval to 1 I get the same result as with a timer, if I set it to 0, well it's going way too fast (2000 to 5000 triggers/sec). MY goal is to get 50 to 100 trigger / seconds and be configurable.

Any idea gurus ?


Copyright 2005-08, All rights reserved