Why is the same SQLite query being 30 times slower when fetching only twice as many results?

I have been working on speeding up a query I'm using for about a week now and asked several questions about it here ( How can I speed up fetching the results after running an sqlite query?, Is it normal that sqlite.fetchall() is so slow?, How to use min() and max() in an efficient way?).

With the very useful help from the answers given there I managed to get the time down to the sqlite query taking 100.95 seconds and fetchall taking: 1485.43. This was still not enough, so after trying out some different indexes I managed to get the query time down to 0.08 seconds for one sample and the fetchall time down to 54.97 seconds. So I thought I finally managed to speed things up enough.

Then the query runs for the next sample, taking 0.58 seconds, and the fetchall taking 3952.80 seconds. For the third sample the query took 1.01 seconds and took 1970.67 seconds to fetchall.

The first sample fetched 12951 rows, the second sample 24972 rows and the third 6470 rows. I'm very curious why the first sample was so much faster to fetch the rows, when it had only about half the amount to fetch as the second example.

Code (spectrumFeature_inputValues is (1,), (2,) and (3,), from the 3 samples used.):

self.cursor.execute('begin')
self.cursor.execute("EXPLAIN QUERY PLAN "+
                    "SELECT precursor_id, feature_table_id "+
                    "FROM `MSMS_precursor` "+
                    "INNER JOIN `spectrum` ON spectrum.spectrum_id = MSMS_precursor.spectrum_spectrum_id "+