Posts: 96
Threads: 10
Joined: Feb 2024
How to open large(+100gb) csv or sql files on a decent laptop? Is it possible to search in the files without unzipping them?
I tried to write a python code that unzips parts of the file then readlines and searchs for the word part by part but it take a lot of time.
I dont have enough storage to unzip it and I only have 8 ram btw This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: fed
Posts: 21
Threads: 1
Joined: Apr 2024
it is 100gb of single or multiple files?
btw, on windows I use EmEditor to open large text files.
Posts: 343
Threads: 66
Joined: Mar 2024
Quote: Is it possible to search in the files without unzipping them?
Not really unfortunately..
Quote:How to open large(+100gb) csv or sql files on a decent laptop?
EmEditor, LTFViewer, Lister.
LTF https://www.portablefreeware.com/?id=693
Lister https://www.ghisler.com/lister/
EmEditor https://www.emeditor.com/
Reply if you need something alternative to these
Quote:I dont have enough storage to unzip it
Time to buy a larger harddrive
Posts: 5
Threads: 2
Joined: Apr 2024
Some text editors are capable of handling large plain-text files, but performance may vary depending on the size of the file and the capabilities of the editor.I was used "emeditor" to open large file,but,over 100gb is indeed too large, and the process of waiting for loading should be uncomfortable.
Of course, the performance of your machine needs to keep up
Posts: 30
Threads: 1
Joined: Jun 2023
Apr 19, 2024, 03:40 AM
(This post was last modified: Apr 19, 2024, 03:42 AM by pyr.)
Learn how to use ripgrep, it will shred thru large files and rip out whatever you are searching for
rg ' [email protected]' -i
^ search for a single thing
rg 'joe biden|poop| [email protected]|555-555-5555' -i
^ search for multiple thangs at once
GL yung jedi on your future adventures
Posts: 55
Threads: 6
Joined: Jul 2023
mm I think you can do it with commands, you use zcat to unzip the file and grep to search inside it
Posts: 96
Threads: 10
Joined: Feb 2024
Apr 19, 2024, 08:32 AM
(This post was last modified: Apr 19, 2024, 08:34 AM by dehumanized.)
(Apr 18, 2024, 04:45 PM)DevEye Wrote: Quote: Is it possible to search in the files without unzipping them?
Not really unfortunately..
Quote:How to open large(+100gb) csv or sql files on a decent laptop?
EmEditor, LTFViewer, Lister.
LTF https://www.portablefreeware.com/?id=693
Lister https://www.ghisler.com/lister/
EmEditor https://www.emeditor.com/
Reply if you need something alternative to these
Quote:I dont have enough storage to unzip it
Time to buy a larger harddrive
Thanks a lot!
(Apr 18, 2024, 03:55 PM)ayaweah Wrote: it is 100gb of single or multiple files?
btw, on windows I use EmEditor to open large text files.
Its a single file, I think I have to cut it to parts first. Thanks for the advice 
(Apr 19, 2024, 05:54 AM)Wanderlust Wrote: mm I think you can do it with commands, you use zcat to unzip the file and grep to search inside it
I've just tried grep and its so good. Thanks This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: fed
Posts: 21
Threads: 1
Joined: Apr 2024
(Apr 19, 2024, 08:32 AM)dehumanized Wrote: I've just tried grep and its so good. Thanks
try awk, it faster than grep.
Posts: 96
Threads: 10
Joined: Feb 2024
(Apr 19, 2024, 09:56 AM)ayaweah Wrote: (Apr 19, 2024, 08:32 AM)dehumanized Wrote: I've just tried grep and its so good. Thanks
try awk, it faster than grep.
Will do, Thanks! This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: fed
Read, learn and stay hard.
Posts: 503
Threads: 15
Joined: Sep 2023
Use emeditor. Requires windows by doing it from an VM/RDP would suffice for the work.
|
