Tracker
 ASP PHP JavaScripts MySQL MS Access MSSQL VISUAL BASIC Subjects/Titles Post Text/Content
HOME    TRACKER    Visual Basic

# Floating Point Type Overflows

## I'm writing a function that computes taxes, and when I input a number over 33,000 it gives me an overflow error. Obviously, it is because of the byte size of the datatype, but how do I allocate more memory for my double?-Steve

See Related Forum Messages: Follow the Links Below to View Complete Thread
(RESOLVED)Wat Data Type Can Represent Character And Floating Point?
I wish to have a data type which could represent both character and floating point?
How?

DIM A AS VARIANT?

Single And Double Data Type Floating-point Format
The range of the single data type is quoted in VB help as +/-3.402823e38 with a zero gap of +/-1.401298e-45. Similarly the double data type has a range of +/-1.79769313486231e308 and a zero gap of +/-4.94065645841247e-324.
I am aware of the IEEE754 32-bit standard with its sign bit, 8-bit exponent, implied one bit and binary point followed by a 23 bit fraction for single data types and similar 64-bit standard double data type with an 11-bit exponent and 52-bit fraction.
I am a part time lecturer on BND Computer Processes which covers binary systems. One of my students has asked me to demonstrate how to calculate the ranges for VB data types and I have done this for all data types except Single and Double. Does VB use a different format? Any suggestions?

Simple Math Using The Double-precision Floating-point Data Type
I'm just learning how to program using VB 6.0. I've built a calculator. Whenever numbers are entered (using either the keyboard or on-sceen buttons I've created), they appear in a label which serves as the display screen for the calculator. I'd like it to be capabale of dealing with large numbers, so I've declared all of my variables as double-precision floating-point data. The program is near finished and most mathematical functions seem to work quite well (adding, subtracting, dividing, multiplying, exponents, square roots, etc.) But for some strange reason, when I enter "650.2 - 650" (or 650.0), and hit the "=" key, I get 0.200000000000045. It not only appears in my display window, but even when I put a breakpoint next to the line of code that performs the operation and hover over each variable, the function looks like this: "0.200000000000045 = 650.2 - 650". So it doesn't seem to be an error with the manner in which I'm displaying the information, nor the way in which I'm transfering it to the display screen. It seems that the mathematical computation is actually a tiny bit inaccurate. Is that simply a negative characteristic of using double data types or is there something else that explains this occurrence? Should I be using another data type? If you need to, I can certainly include the code, but like I said, I think this has something to do with the way it's storing the variables, not my code. Thanks in advance for your help...

- Matt -

Beyond Floating Point
I have a 'program' written in Excel that uses cell formulas to come up with an answer. In itself, it isn't that exciting, the thing is that there are about 16000 rows and 50 some odd columns. Needless to say it takes a while to calculate. I have converted this to Visual Basic, however I get different answers, generally a couple of percent off. I think this may be because Excel stores 30 digits wheras VB only has 15. Is there any way I can store 30 digits in VB. If not, what language would you suggest I use that could handle this. Thanx.

Grover

Floating Point Numbers
I have a binary file containing vertex data. I need to read this information from the file as a 32 bit float for each x, y, and z position. No matter what I declare the variables I'm storing this information into, the program constantly returns the values as a Double.

Is there a way to do this in VB?

All help is greatly appreciated,

SoDA

String To Floating Point
Is there an easy way to convert a string = "3,5" to the floating point value 3,5 (or 3.5 for those outside of Norway?

Rounding Floating Point
how do you round single or doubles in visual basic?

Floating Point Variable To Hex Value
I am trying to show the hex values of a floating point number but having a problem.
My code:
Dim x As Single
x = CSng(Text1)
Label1 = Hex\$(x)
Label2 = CSng("&h" & Label1)
If I enter 1.1 then label1 shows 1 and label2 shows 1
if I enter -1.1 then label1 shows FFFFFFFF and label2 shows 1
I searched the site and ran into the copymemory api and tried it but it crashed the program. I guess it couldn't move it to a label.
Is there a way to get the full hex value of a floating point variable?

16-bit Floating Point Variables
Hi, does anyone know of a way to use 16-bit floating point variables/arrays in Visual Basic 6.0? I'm processing great amounts of data, so 32-bit floats are not an option (to memory intensive).

Perhaps there's a way to use integer variables/arrays to store it, but process them differently...

Floating Point Issues
I'm trying to work with a file that includes floating point information in 32 bit form. I need to read that information in and display it as a decimal number, allow the user to change that decimal number to another decimal number and then put it back into the file in 32bit form.

Getting it in and out of the file isn't a problem - I have it in a byte array , and once I've got it back into a byte array I don't have the slightest problem putting it back into the file.

So lets say that I have a 32 bit floating point number in a byte array as follows:
41 9E 66 66

Now I happen to know that converted to decimal that is 19.8; but I've no idea how to program a routine to make the change.

And equally, if my user enters "3" into a text box I have no idea how to turn that into the byte array 40 40 00 00.

Any help much appreciated,
Rob

Function To Cut The Floating Point Value
Hi, I am looking for the function that cuts the point in numbers.
EXAMPLE:

a = func("5.7")
a becomes 5

See what I mean, just want to cut the point in a real/floating point value.

I tried using an Integer but that just rounds it up (I do not want to do this)

3.9 becomes 4 (<-- NOT RIGHT)
3.9 becomes 3 (<-- RIGHT)

Thanks

Check Floating Point
I want to make an error checking to check whether the value in the textbox is a floating point or not. Please help me...........

How Can I Round A Floating Point In SQL ?
sql = "Select id, avg(delay) as avgdelay from tel group by id "

How can I make avgdelay a integer ?

Bytes To Floating Point
I want to convert 4 bytes in row to a single precision number. These are read from a file by means of Get commands but it's in big endian order so I must reverse them.

Specifically, I'd like to know if there is a similar function to ntohs and ntohl for single and double precision floating point numbers. Or is there any command to read a file backwards?

Floating Point Numbers
Admittedly this is a more general than just a vb-specific question: could someone give me a good address for a tutorial on floating point numbers, i.e. how single and double precision values are coded? I don't ask you guys to post an explanation as I guess it might take a good deal of writing.

Floating Point Numbers
Anyone know where I can get some information on Floating Point Numbers?

Like: What are they?
What do you use them for?

Thanks

Floating Point Numbers
How can I get the user to enter a floating point number? If I use a text box then the val function, I believe errors occur.

Floating Point Error
Hi VB Programmers,

Here is my short code

Code:

Dim x as Double
Dim y as Double
Dim z as Double

x = 10.0005
y = 10.0004
z = x - y

MsgBox z

When I execute it, it show unexpected 9.9999999999759E-05.

How can I get the expected result such 0.0001?

Thank You.

Floating Point Number
How can i minimize the error if i have to work with floating point number.

Floating Point Numbers
Hello,

I would like to return the floating point remainder of a calculation and use it for further calculation

For instance 60000/256 = 234.375 I would like to strip the 375 and add it to a variable any help will be very much appreciated, and thanks in advance fo any replies.

8 Byte Floating Point Variable
Hello everyone im taking a class were we are doing some programming and basically our 1st program is a simple interest calculation program, but he said we need to have double precision 8 byte floating point variable. I have no idea how to do this in visual basic 6.0 can someone tell me how to set this up so that all my numbers are in this format, thanks so much. Im really new at VB

To Transform IEEE-P 754 80-bit Floating Point
Hallo experts!

I import the 80bits as string to VB. Is there a routine/function to transform the values to VB type "double". (of course by loss of precision) and vis versa.

Thanks and regards

Hans

Floating Point Scientific Notation
Hi,
I'm designing a function shich returns a floating point value "single". how can I forca Visualbasic to print it normally without converting to the scientific notation.

MNT

Long 2 32bit Floating Point
I'm useing this bit of code to get values stored as 32bit floating point numbers in a file:

Private Sub Command1_Click()
Dim vals(3) As Long
Dim sngOut As Single
Dim lngIn As Long

vals(0) = Val("&H" & Text1.Text)
vals(1) = (Val("&H" & Text2.Text) * 256)
vals(2) = (Val("&H" & Text3.Text) * 65536)
vals(3) = (Val("&H" & Text4.Text) * 16777216)

Label1.Caption = vals(0) + vals(1) + vals(2) + vals(3)

lngIn = Label1.Caption
sngOut = 0
CopyMemory sngOut, lngIn, 4
Label2.Caption = sngOut

End Sub

It works fine for some values e.g the hex values 00,00,00,3F should retern 0.5 and they do, but some overflow the Long can anyone think of a way round this?

Invalid Floating Point Operation
Hello,

I am experiencing a hard to catch error in my VB6 application. After running for several hours, it stops working correctly (but does not crash), also a messagebox pops up, titled VB6, with in it the text: "Invalid floating point operation".

I have no clue what might be causing this error, and it does not seem to be reproducable. I just happens, after some hours of running. No other application is active, the PC, running Win 98, is as clean as it can be.

Maybe anyone recognizes this kind of problem and have a solution?

Regards,
Bas

Hex To 32bit Floating Point Numbers
E02F = 703.5
4041 = 773
C04C = 847

How is this conversion done?
or how can this conversion be done in VB?

Floating Point Over A Serial Port
I would like to send single precision floating point numbers over the RS232 buss. There should be a way to do this directly by sending 4 bytesper number. Does anyone have any examples of this. I've seen the LSetinstruction used to move data between different data types on a byte bybyte basis if the types are the same size. This implies that a floatingpoint single precision number could be converted into 4 characters andthese characters then could be sent over the serial port. There areexamples athttp://support.microsoft.com/support/kb/articles/ Q235/8/56.ASP?LN=EN-U&SD=gn&FR=0&qry=floati ng%20point&rnk=12&src=DHCS_MSPSS_gn_SRCHSPR= VBB but I can't seem to get it to work in my applications.

Precision Of The Floating-point Types?
I need to find out the precision of the floating-point data types viz. Single, Double, and Decimal i.e. the maximum number of decimal digits ( i.e. base 10 digits ) that may be specified in thier literals. As this is not documented I thought I'd keep on enterring ever larger numbers in the Immediate Window until I got an overflow error and then count the number of digits. However to my frustration, as soon as I exceed 15 9' i.e. 9999999999999999, it gets converted to 1E+16.

How can I find out the precision and scale of these three data types?

8-byte Floating Point Number
Hi All,

If I have an 8-byte floating point number in hex how do I convert it to a
double?

0 0 0 0 0 98 C1 3F this should be about .13746

Thanks

Dan...

Single Floating Point Conversion
I'm trying to convert a 4-byte Hex value which supposed to be represented as a single floating point value under the IEEE 754. The following is what I did, but the result is not correct.

Dim test as Single
test = &HC2934000

test is eqaul to -1.030537E+09, but it suppose to be 1.150390625E+06 which is -73.625

Any help is greatly appreciated. Thanks!

Floating Point In MSHFlexGrid Control
Hi you all,
How do i set the floating point precision of a numeric field in the recordset of a MSHFlexGrid control. I am activating a query that rounds the value by 2 and still on the MSHFlexGrid control it shows as 3.4599999999999999 for example.
How do i fix it ?

Convert Hexadecimal To Floating Point
Call me an idiot, but for the life in me i cannot figure out how to do a simple conversion......
I need to convert
FB7E0E41 to 8.9060
32 bit IEEE Hex to decimal
can ANYONE Help ME!!!!!

Thanks
Nick Stone

Changing A Floating Point Number To An Integer
I have an long equation that returns a value as a long floating point number my question is there a way to convert this to an integer value so that it can be compared to an integer number that is input by the user.

This is the equation.

Text7.Text = 90 - (Atn((32.2 * 180) / ((Text1.Text / 5280) * 3600) ^ 2) * (180 / 3.14159265359))

it usually returns a number like 1.45838568387579 I know converting it into an integer will lose accuracy but that doesnt matter.

Convert Bytes To IEEE-754 Floating Point
I have 4 ordered bytes values (0-255) which I receive from a device's output. These 4 bytes represent a 32bit floating point value in IEEE-754 format. I need a VB6 function which converts the 4 bytes into this value. Thanks in advance.

Simple Way To Generate A Floating Point Error
Here is something that might leave you with bald patches on your head at the end of the day. A simple decimal, less its integer portion, results in a floating point error in the 11th decimal place.

A = 2436116.81
B = Int(A)
C = A - B

Result:

C = 0.810000000055879

I mean seriously, I can do this in my head. My pocket calculator can do it too. Why can't Visual Basic 6.0 do it right?

Datatypes: Bitoperations With Floating Point Numbers
Hello,

from my point of view, I'm having a really tiny problem, but I am not able to solve it by myself. Is it really that hard?

My final goal is to be able to split 32bit floating point variables (as Sinlgle) binary into 4 seperate bytes (which is necessary to work with a given transmission protocol). Therefor I am trying to read and write single bits in a floating point variable (e.g. Single).
I already succeeded with 32bit variables of the type Long. So my problem would already be solved, if you can help me to write the 32 bits of the 32bit Single variable bitwise into the 32 bits of a 32bit Long variable.

Code:
dim s as Single
'dim s as Long
Dim bit(0 To 30) As Boolean

s = 12.34
's = 12

For j = 0 To 30
bit(j) = CBool(s And 2 ^ j)
Next j
This code works fine for the commented line, when the variable s is of type Long. When s is of type Single, VB automaticly casts the 12.34 to an Integervalue of 12.

So how can I split (and recombine) the 32bit Singlevalue into Bytes (8bit)?

Thanks in advance for your effort, Parity

How To Distinguish If The Numerical Value Is Integer Or Floating Point?
Hi, I'm a newbie of VB, I'm working on my VB assignment at this moment and encounter a problem.

In a part of my assignment, I need to make a Textbox and type a numerical value in. And I need to know if this number is Integer or floating point value.

Any VB build-In function can do that? or I should check if the string value contain "."?

Thanks for helping

Formatted Output Floating Point Numbers
Does anyone know how to output a floating point numbers in scientific notations ?

Such as: 1 will be displayed as 1.0E0
0.1 will be displayed as 1.0E-1

and so on.

Thanks!

Floating-point Inexact Result Error
I am running vb6 with sp5. I am making a simple text editor. The application runs perfect in the ide. however when I compile, run, end exit the program I get this error. I looked it up on MSDN and discovered this err was suposed to be fixed in sp4

I am using a coolbar controll (microsoft windows common controlls-3 6.0(sp5)

any ideas of how I can fix this?

Floating-point Inexact Result Error...
When compiled, I get the following:

Unhandled Exception at 0x7c81eb33 in <myprogram.exe> 0xC000008F, floating-point inexact result...

Is there anyway to track this down? It doesn't happen in the VB6 IDE at all, just from the compiled exe...

Data Types: Double And Floating Point
Hey guys,I was wondering is there any other vairable type to hold a fractional number other then Double?And what does "floating point" mean?Also what is "fixed point" then?
Misha

Floating Point Error Raised After END Statement.
I have an application which, in certain circumstances, unloads all forms and is then terminated with an END statement. However on some (but not all occasions) when this code is run on machines with a Win32 debugger installed I get the following error message.

---------------------------
Messenger:Messenger.exe - Application Error
---------------------------
The exception Floating-point inexact result.

(0xc000008f) occurred in the application at location 0x77e87e08.

Click on OK to terminate the program
Click on CANCEL to debug the program
---------------------------
OK Cancel
---------------------------

Does anyone have any ideas as to how I may resolve this?

Limiting Displayed Floating Point Numbers
I want all my displayed data to be limited to 4 decimal places. I was using the format function. However I have come across a problem

I have a small number e.g. 6.07222318649292E-07 If I limit this with the format function, I get 0., which is clearly wrong.
What I would like is 6.0722E-07.

Here is how I am using the format function

VB Code:
my_text = format(my_number,"0.##")

Any ideas?

Cheers

VBVM60 Error / Floating Point Exception
Hi All,

I've made an small application in which i'm using hook, unhook and WndProc. When unhooking at the exit (through exe), it shows an error in VBVM60 and if NT machine then error is Floating point exception. I'm using SetWindowLong for unhooking.

Thanks.

Byte Array To Floating Point Number
I have a binary array which I wish to convert to a 4-byte floating point number.
I have searched but can't find a way to do this. Any help would be appreciated.
Using VB6. Thanks.

Help - Floating Point Inexact Result - Not Coolbar.
I have a very simple form (simple now that I've stripped everything else out of it) that consists of a timer and a winsock control. The winsock listens for incoming connections and the timer fires to see how long ago the last connection was. There are no other controls. I had a tray icon that popped up a menu to end the program, but now (for debugging) you click the form to end it.

When the program ends, I get :

The exception Floating-point inexact result.
(0xc000008f) occurred in the application at location 0x7c59ba9d.

I have disabled the timer and removed all code from the Load event of the form so that the winsock is not active. Essentially there's nothing going on, but I always get this error.

In the development environment, issuing an End command either in code or from the immediate window hangs VB and I have to kill it form task manager. I have no float, single, or double data types in the program and no math.

I have used the winsock control for years and have never seen this. Any ideas?

Thanks,
Bob

Edited by - rlisak on 2/5/2004 8:17:23 AM

Floating Point Problem In SQL UPDATE String
I'm using the following string to update some records in a MSAccess table.

Code:"UPDATE DatiAnnuali SET saldatt = " & dati(dovesto, 2) & " WHERE disegno like '" & currentdisegno & "' AND anno = " & I

It works just fine until one of the variables contains a value with decimals. What follows is how the Local Variables window resolves the string when it stops for the syntax error.

"UPDATE DatiAnnuali SET saldatt = 512,5 WHERE disegno like '830847' AND anno = 2005"

By trial and error I realised it does not like the comma in "512,5". That number is coming from dati(x,y) which is a Single (Precision) array, and its value populates a Single Precision field in the db table.

How do I overcome this???

Thank you!
Cam

IEEE Format For FLoating Point Calculations
hii,
how can i convert Text1.Text to Floating Variable using IEEE-754 Floating Point Conversion,