Alex Rivera | Logout

Python efficient way to check if very large string contains a substring

Asked 2011-08-24T11:57:18.663
12

Python is not my best language, and so I'm not all that good at finding the most efficient solutions to some of my problems. I have a very large string (coming from a 30 MB file) and I need to check if that file contains a smaller substring (this string is only a few dozen characters). The way I am currently doing it is:

if small_string in large_string:
    # logic here

But this seems to be very inefficient because it will check every possible sequence of characters in the file. I know that there will only be an exact match on a newline, so would it be better to read the file in as a list and iterate through that list to match?

EDIT: To clear up some confusion on "matching on a newline only", here's an example:

small_string = "This is a line"
big_string = "This is a line\nThis is another line\nThis is yet another"

If I'm not mistake, the in keyword will check all sequences, not just every line.

Edit
Report

1 Answer

-1

I would rely on fast implementation by someone else:

import subprocess
from subprocess import STDOUT
import os

...
with open(os.devnull, 'w') as devnull:
    if subprocess.call('grep %s "%s"' % (smallstring, file), shell=True, stdout=devnull, stderr=STDOUT) == 0:
        pass #do stuff

Won't work on windows.

edit: I'm worried taht grep returns 0 wheter it finds something or not. But I don't have any shell available to me now so I can't test it.

answered 2011-08-24T12:12:01.287

Your Answer