reading a large json file which is really huge and it contains pair of id and URL in a array as shown below.. i need to get this all id and URL from this file in my java code and need to perform operations . how to read such a big file and store the contents in java code to process it further.
{
"Array": [
{
"id": "1test",
"URL": "http://servername/test/1test.html"
},
{
"id": "2test",
"URL": "http://servername/test/2test.html"
}
....................many id and URL in the same Array which makes the file Huge..
]
}
It seemed to me that the bufferSize is limited so that I can only have it less than a default one, how to increase it?
My question now is how to increase the size of buffer in order to read a larger InputStream ? BufferedInputStream() default size is private static int defaultCharBufferSize = 8192; private static int defaultExpectedLineLength = 80;
How to make it larger and working? I tried to increase the defaultCharBufferSize to 500000000 but it did not work!
Im trying to read a file called m1 and put that information into a 2-D array with the first two numbers being the size of the array i was able to set the size of the array but how do i populate array?
import java.util.Scanner; import java.io.*; public class test{ public static void main(String [] args)throws IOException{ Scanner fr = new Scanner(new File("m1.txt")); String line = fr.nextLine();
I would like some clarification on a project I am working on. I need to build a java application to query a xml and json file. I have started by parsing both documents and I am unsure if I am doing the next steps correct. I was thinking about dropping the txt file with both outcomes into a db and doing it that way, is this the right way or is there a better and more efficient way of doing things ?
I am trying to make a bank program that reads to .txt files, one that has the right format and one that has the wrong format. This is part of my problem I do not know how to do this. I also do not know why my code works if I input "account" rather than "account.txt" which is supposed to be the right thing.
Let me present my code that I have so far:
// File: BankAccount.java import java.util.Scanner; import java.io.IOException; public class BankAccount { private int accountNumber; private double balance;
I am advised to use a while loop the Scanner method hasNextLine() and in the while loop body, call the Scanner method nextLine(), and add the returned String to the ArrayList of Strings. what I have gotten so far:
Scanner input = new Scanner(new File("")); while(input.hasNextLine()) { String line = input.nextLine(); System.out.println(line);
We have an autosys job running in our production on daily basis. It calls a shell script which in turn calls a java servlet. This servlet reads these files and inserts the data into two different tables and then does some processing. Java version is 1.6 & application server is WAS7 and database is oracel-11g.
We get several issues with this process like it takes time, goes out of memory etc etc. Below are the details of the way we have coded this process.
1. When we read the file using BufferedReader, do we really get a lot of strings created in the memory as returned by readLine() method of BufferedReader? These files contain 4-5Lacs of line. All the records are separated by newline character. Is there a better way to read files in java to achieve efficiency? I couldnt find any provided the fact that all the record lines in the file are of variable length.
2. When we insert the data then we are doing a batch process with statement/prepared statement. We are making one batch containing all the records of the file. Does it really matter to break the batch size to have better performance?
3. If the tables has no indexes defined nor any other constraints and all the columns are VARCHAR type, then which operation will be faster:- inserting a new row or updating an existing row based upon some matching condition?
Our client has a user facing web application running on Jboss. There is a separate admin application (in its own ear) but deployed on same Jboss server on which user facing web application is running.
They need a screen to upload large amount of data into database. Their original files were in excel with size > 60 mb. We suggested following to them:
a. Change upload format to CSV - this brought down file sizes to 25-30 mb b. Upload process will be MDB - asynchronous processing of data so that admin web app does not stop responding
We also suggested following to them:
a. Host admin app on a different machine so that user facing site does not respond slow during data processing b. We can provide incremental upload feature and they should upload files in the chunks of 4-5 mb, specifically if they have user a web page to upload such files - they don't buy this argument though. c. Data processing can be a separate script instead of a part of admin web application. They can FTP files to a designated location and this script will process those files.
I have following questions:
Q1 - Have you seen upload of such large datafiles to a web application? I see sites like Zoho CRM or Salesforce do not support such data imports and mostly fail or not respond. Q2 - Is there a set of guidelines/best practices to upload large data files of this nature? How do insurance companies or others with enormous set of data accomplish such tasks (what is the architecture of such programs)?
I looked on other examples which have collection of object but I have this extra top level object "AvailableDeliveries" not sure how to handle that level.
I need to built a file upload service which should be memory effective. I should avoid loading the entire file into memory,Since I may have multiple http request which will pile up the Heap memory. Any effective way to upload a large file(For ex:1GB file) using http Streaming. I need to do the file upload on a single http call. Let's consider a scenario where 1 GB file to be uploaded using 512MB Heap memory. Not Sure If practically I can achieve this or not.
I have a requirement where a large file (100 -200MB) is uploaded from the client to a content management system. I am using a servlet with Apache Commons File Upload API. Apache FileUpload has 2 ways of handling files,
1) Non-Streaming 2) Streaming
Currently I use the Non-Streaming approach where the servlet stores the file in a temp location and upload the same into the content management system - This is taking lot of time so I am trying to implement Streaming API.
Content Management API supports streaming in 2 methods,
a) SetContent - Takes the file's ByteArrayOutputStream as input -> This gives OutOfMemoryException because the file being large b) AppendContent - Takes the file's ByteArrayOutputStream as input -> This method can be called multiple times to upload the large file but I dont know how to do this. The Apache File Upload gives InputStream of the file and I need to split that into chuncks and append into the content management system.
How to convert InputStream to 4KB ByteArrayOutputStream so that I can use the AppendContent method in content management API?
I have a large text file of 1 GB size. I need to print the line when a matching word is found in a particular line. Below is the code I am using. But if there are many lines that has the matching word, it's taking lot of time. Any solution to print the lines much faster.
Scanner scanner = new Scanner(file); while (scanner.hasNextLine()) { String line = scanner.nextLine(); if(line.contains("xyz")) { System.out.println(line); } }
I am working on a project that requires me to build a database with random access file, representing products, the base product contains a name (about 30 characters), a price (double), and a quantity (integer). I have worked on this project for probably 15+ hours and have tried so many things and feel like I've barley made any progress...
The part i am really struggling with is taking the data from the text file and creating an object array with it using the product class. Once ive accomplished that, i have to use that data to create a random access file with the data. Here is the base Product class that must be used to create the objects for the array.
public class Product { public String pName; public String stringName; public double price; public int quanity;
[Code]...
these continue for about 40-50 entries, they are not seperated by a blank line though i had to add those so it would display correctly, each entry is on its own line with name seperated with spaces, then price after a comma, then quanity after the second comma.....
I am working on a project that requires me to build a database with random access file, representing products, the base product contains a name (about 30 characters), a price (double), and a quantity (integer). I have worked on this project for probably 15+ hours and have tried so many things and feel like I've barley made any progress...
The part i am really struggling with is taking the data from the text file and creating an object array with it using the product class. Once ive accomplished that, i have to use that data to create a random access file with the data.
Here is the base Product class that must be used to create the objects for the array.
public class Product { public String pName; public String stringName; public double price; public int quanity; //Constructor public Product( String pName, double price, int quanity )
[code]....
and then here is the data from the text file that i must extract to use to create product objects.
Dill Seed,938,34
Mustard Seed,100,64
Coriander Powder,924,18
Turmeric,836,80
Cinnamon (Ground Korintje),951,10
Cinnamon (Ground) Xtra Hi Oil (2x),614,31
Cinnamon (Ground) High Oil (1X),682,19
these continue for about 40-50 entries, they are not separated by a blank line though i had to add those so it would display correctly, each entry is on its own line with name separated with spaces, then price after a comma, then quanity after the second comma.....
i have to write more than 100000 rows in a excel sheet (file size more than 20 MB) via java.
when i use XSSF, i am getting below Error.
java.lang.OutOfMemoryError: Java heap space at org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1592) at org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1223) at org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1144)
[Code]....
when i use HSSF , i am getting the below Error. java.lang.OutOfMemoryError: Java heap space
I have tried increasing the java heap size , by giving upto -Xms1500m -Xmx2048m
I develop a finite element code at java. I am looking for efficiency solver and fast for large, sparse , symmetric and positive define matrices .
i used in jblas but i encounter in problem when the matrix has high condition number(ill condition number) and i get error for singular matrix while in mathematica i succeed to solve that system without problems...
Any good solver and fast solver package in java can i use for solving that system?
I have a CSV file with 16K entries of a data table. Does Java work well with CSV file? So I found this code. And it seems its quite easy to read in the data I need. Say for example if I wanted a loop to randomly pick the first field of a specific line in the CSV data table. How would i go about coding that??????
The CSV looks like the above. and I basically would like to read in the Hand to get it to show in a text box and then randomly have the program ask me to correctly identify the True/False return for one of the SB/BB/UG/MP/CO/BN columns.
how to get the first few hex symbols of a file in java, for example if i input a pdf into my coding i want my program to output, e.g "25 46 44 38" ....
I have been able to print out the hex of a whole file but not managed to set a maximum read limit so that my code only takes a certain amount of values ....
I am implementing a recursive descent parser that recognizes strings in the language below. The input should be from a file "input.txt" and output should be to the console.
The grammar:
A -> I = E | E E -> T + E | T - E | T T -> F * T | F / T | F F -> P ^ F | P P -> I | L | UI | UL | (A) U -> + | - | ! I -> C | CI C -> a | b | ... | y | z L -> D | DL D -> 0 | 1 | ... | 8 | 9
An example session might look like this:
String read from file: a=a+b-c*d
The string "a=a+b-c*d" is in the language.
String read from file: a=a**b++c
The string "a=a**b++c" is not in the language.
Java Code: /**
* The Grammar * A -> I = E | E *E -> T + E | T - E | T *T -> F * T | F / T | F *F -> P ^ F | P *P -> I | L | UI | UL | (A) *U -> + | - | !
[code]....
My current output looks like this:
Java Code: The string read from file: a=a+b-c*d
The string "" is not in the language. mh_sh_highlight_all('java');
So it seems to be reading the input file correctly. My error seems to be on this part
