scrape google api return

Language: PHP, NO FRAMEWORKS

Database: mysql

API used: [login to view URL]

Table A (named rfpsites_mainlist)

name varchar

state (varchar) 2 letter us state code

type varchar(city, county, school district, university, college)

Table B (named serpurl)

id (int auto increment)

name varchar

url

state (varchar) 2 letter us state code

type varchar(city, county, school district, university, college)

createddate (auto populate on insert)

key

dataforseo_task_id

runid

Table C (named dataforseoloc)

import of csv file from [login to view URL]

Testing:

You must be able to run without arror against 100 random test search queries of your choice, twice. Simply populate your test search queries into the name column on table A, as well as two letter US state names in the state column in table B. Half way through your run of 100, you will do a ctrl-C to force close the process, and rerun the script. It must only retrieve the remaining portion from where it left off. Once that is complete, do one additional run of the script against the complete 100 records. You should have 10,000 - 15,000 rows in table B at this point.

The script will:

1

Determine the current run id. This will either allow a full run, or a partial run to complete, if it was interrupted. Look at table B for the highest runid value (select runid from tableB order by runid DESC limit 1). Loop through all of the concatted ("[login to view URL] [login to view URL] [login to view URL]") in table A, to see if that value exists in table [login to view URL] with the same runid value. If they all exist, increment the runid upward one, and start an entire new run. If they partially exist, skip past any key values that exist in both tables for that runid, and complete the data being gathered into table B. Each key will need to be in table B a minimum of one time, and a maximum of 100 for the given runid.

If there are no records in table B, set the runid to 1.

For the concatted values being skipped, skip that one and check the next match.

For each of the concatted values that are not being skipped, proceed with step 2

2

identify one of the location id to use for the query

For entries in table A with a state value, convert the 2 digit state to full name, and prepare a query as follow:

"Select loc_id from tableC where loc_name_canonical like "*StateName, United States" and loc_country_iso_code limit 1"

For entries in table A without a state value, we will just leave loc_id undefined

3

for each for of the records in tableA, run a query against the serp api from dataforseo

do a multi_curl call to the url (enough to try and saturate 100 megabit)

set key to the value of these fields concatted together with a space in between "[login to view URL] [login to view URL] [login to view URL]"

set loc_id to the one retrieved in step 2

set se_name to "[login to view URL]"

4

parse the results and keep track of which key resulted in which multi_curl return

skip the paid results

skip any url with these words in the domain name - "weather", "twitter", "glassdoor, "census", "zillow", [login to view URL], mapquest, [login to view URL], [login to view URL], casetext, or "google" in it

5

for each of the results remaining from step 4, check to make sure a record does not already exist in tableB for that url. you can consider it a match if it is simply http vs https, but the frst of the url is identical. the domain portion is not case sensitive, but the path is case sensitive. If it does already exist, you can skip the insert.

6

insert a record into table B, with :

populate [login to view URL] to concat ([login to view URL], [login to view URL], [login to view URL])

url = from dataforsea return

dataforseo_task_id = from dataforsea return

runid = genereted in step 1

7

repeat steps 5-6 for every url returned from each api call, not skipped in step 4-5

8

repeat steps 2-7

PHP MySQL Nxjerrje të dhënash nga interneti cURL

ID Projekti: #20157634

Rreth projektit