Edit Question - KnowledgeHub

Title Body

I have a database table with a unique string field and a couple of integer fields. The string field is usually 10-100 characters long. Once every minute or so I have the following scenario: I receive a list of 2-10 thousand tuples corresponding to the table's record structure, e.g. [("hello", 3, 4), ("cat", 5, 3), ...] I need to insert all these tuples to the table (assume I verified neither of these strings appear in the database). For clarification, I'm using InnoDB, and I have an auto-incremental primary key for this table, the string is not the PK. My code currently iterates through this list, for each tuple creates a Python module object with the appropriate values, and calls ".save()", something like so: @transaction.commit_on_success def save_data_elements(input_list): for (s, i1, i2) in input_list: entry = DataElement(string=s, number1=i1, number2=i2) entry.save() This code is currently one of the performance bottlenecks in my system, so I'm looking for ways to optimize it. For example, I could generate SQL codes each containing an INSERT command for 100 tuples ("hard-coded" into the SQL) and execute it, but I don't know if it will improve anything. Do you have any suggestion to optimize such a process? Thanks

Tags (comma-separated) Cancel