I need to create a cron script to process products (add, delete, update) when I get them from a raw data file from a third party firm which sends a new file (data separated with ;) whenever a product is changed (added, deleted, updated). My script actually works, it deletes, updates and adds products according to the file information I am getting.
My problem is that I have more than 70 000 products to add for the first rawdata that I am getting (the entire catalog from the provider), which creates a sql error :
Warning: mysqli_connect(): (08004/1040): Too many connections in D:\prestashop\crontasks\config\Database.inc.php on line 20
The problem is that I am using the Product class methods add, delete and update, and apparently the connections from the database don't get closed.
I would like to know if there is a solution to that, if I have to manually create a function to do the insert of all the products at once, or if there already is one. I tried to look into AdminImportController, but I can't find where the insert is made, and it does not solve the update and delete case. I am a little affraid to create functions for it, I don't know if having very long queries would not slow the website a lot while adding the products, nor the effect on the database.
The script in itself is very simple, it only explodes the rawdata line on the character ; and gets the important informations (it has more than 280 columns).
The big amount of products (or lines) is also not a one time thing, as there will be one (or more) for winter-products, or just to update prices.
I have heard about augmenting the amount of connections allowed, but I don't believe it's a good solution, as it might slow the website a lot during the processing of the products.
If anybody has a solution, I will gladly hear it.