Thank you for opening this question.
I am building a scraper, py script scraps things, py put results in json, php read it and display them in view. when i try to choose some record to save in my database it results in error
Object not found!
that error happened for most of the data, some others are saved fine into my database.
the error is just random, I don't think there is some pattern from what I found. Object not found issue usually related with path, route, httacess, but in my case, the path is correct (some data can be saved fine).
What I tried to find the problem :
result : no pattern, from 30 data i scrapped, only data no 3,4,5,15,16,17,21,22,23,24,27,28 and 29 worked
result: no pattern, for some data, to whatever order a data is moved, that data is still can/cannot be saved, so an order is not affecting. sometimes order effects
it exists and located correctly
located at : C:\xampp\htdocs\scraper
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L]
My MVC Snippet if it's help. (I'm using Codeigniter)
controller :
function saveProduct($URLEncoded){
$this->load->model('modelhere');
$json = json_decode(file_get_contents('C:\xampp\htdocs\selenium\scrap-product.json'), true);
$URLdecoded=base64_decode(urldecode($URLEncoded));
$savedData;
//GET THE URL FROM CHOSEN DATA, THEN GET DATA THAT MATCH WITH CHOSEN DATA URL IN JSON FILE
// DATA CONTAINS PRODUCT AND SELLER INFORMATION
for ($x = 0; $x < sizeof($json) ; $x++) {
if ($json[$x]['URL'] == $URLdecoded) {$savedData = $json[$x];}
}
// CHECK IF SELLER EXIST IN DB
$result = $this->penjual_model->cekPenjualExist($savedData);
//CHECK IF PRODUCT EXIST IN DB
$result2 = $this->produk_model->cekProdukExist($savedData);
// IF !EXIST(SELLER) : SAVE THAT SELLER
if (empty($result)){
$this->penjual_model->tambahPenjual($savedData);
}
// IF EXIST(SELLER) AND !EXIST(PRODUK) : SAVE THAT PRODUK
if (!empty($result) && empty($result2)){
$this->produk_model->setDataProduk2($savedData, $result);
}
// IF BOTH EXIST DO NOTHING
else if (!empty($result) && !empty($result2)){
}
//REDIRECT
$this->index();
}
Model :
function setDataProduk2($temp, $temp2){
//isi produk
$uname['productname'] = $temp['productname'];
$uname['website'] = $temp['marketplace'];
$uname['weight'] = $temp['weight'];
$uname['desc'] = $temp['desc'];
$uname['URL'] = $temp['URL'];
$uname['idseller'] = $temp2['idSeller'];
$this->db->insert('produk', $uname);
}
View :
//it's the submit button that encode and sends selected product url to controller
$i=0;
if(isset($data)){
foreach ($data as $list){
echo "<tr>";
echo "<td>".'
<a class="btn btn-success btn-sm" href='.base_url('product_controller/saveProduct/'.urlencode(base64_encode($list['URL']))).'>save</button>'.
"</td>";
$i++;
}
}
?>
example of URL that worked fine :
raw URL :
https://mywebsite/p/sitepath/sitepath2/sitepath3/6epkxb-chocolate-aura?from=omnisearch&product_owner=normal_seller&search%5Bkeywords%5D=chocolate
// i don't think that i can show the raw URL due to some reasons
encoded :
http://localhost/scraper/produk_controller/saveresult/aHR0cHM6Ly93d3cuYnVrYWxhcGFrLmNvbS9wL3BlcmxlbmdrYXBhbi1rYW50b3IvYWxhdC10dWxpcy1rYW50b3IvbGFrYmFuLWxlbS82ZXBreGItanVhbC1sYWtiYW4tY29rbGF0LW1lcmstYXVyYT9mcm9tPW9tbmlzZWFyY2gmcHJvZHVjdF9vd25lcj1ub3JtYWxfc2VsbGVyJnNlYXJjaCU1QmtleXdvcmRzJTVEPWNva2xhdA%3D%3D
example of not working :
raw:
https://mywebsite/p/sitepath/sitepath2/sitepath3/el6k85-jual-squishy-chocolate?from=omnisearch&product_owner=normal_seller&search%5Bkeywords%5D=chocolate
encoded:
http://localhost/skripsi-scraper/produk_controller/simpanPencarian/aHR0cHM6Ly93d3cuYnVrYWxhcGFrLmNvbS9wL2hvYmkta29sZWtzaS9tYWluYW4vbWFpbmFuLWxhaW5ueWEvZWw2azg1LWp1YWwtc3F1aXNoeS1jb2tsYXQ%2FZnJvbT1vbW5pc2VhcmNoJnByb2R1Y3Rfb3duZXI9bm9ybWFsX3NlbGxlciZzZWFyY2glNUJrZXl3b3JkcyU1RD1jb2tsYXQ%3D
So, my questions are :
any idea what caused it and maybe the solution?
are there any ways to debug this ?
My JSON file if it's help
JSON - it has 30 row data