如何快速搜索1500万条记录

I have excel file of 600 records, which I am going to compare with 15 million records from the database. For each excel record I need to find matches from 15 million records. currently it takes about 4 hours to complete the process. I want to minimize it at least up to 2 hours.

  1. Export excel to CSV
  2. Import CSV into mysql (LOAD)
  3. Create index on the key column (the one you want to compare with the 15M records)
  4. Make sure that the 15M records table has the proper index set up on the comparison key!
  5. Write a simple query that joins the two table with the criteria you need.

All these points are fairly trivial and left as an exercise to the reader.

Try

if indexing not done : Do indexing across the fields you are comparing, the result would be magical.

if possible avoid using LIKE queries and change the sequence of WHERE query.

eg: WHERE city = "New York" AND name LIKE "XYZ" AND mobile=7777777777 AND status=1

it should be like

WHERE status=1 AND mobile=7777777777 AND city = "New York" AND name LIKE "XYZ"

SEQUENCE of where flags then ints then chars then varchars then like in last, it matters a lot.