Go例程中的频道使用情况

I have a Golang application which stores urls (provided as a query parameter) in a database. Storing the urls is done using the following method:

func AddArticle(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        queryParam := r.FormValue("url")

        insertedId := dao.SaveArticle(db, queryParam)
        glog.Infof("add hostname %s, id: %d", getHostnameFromUrl(queryParam), insertedId)

        // start routine which scrapes url
        go dao.ScrapeArticle(db, insertedId)

        // finally output confirmation page
        renderObject := map[string]string{"hostname": getHostnameFromUrl(queryParam)}
        render.DisplayPage(w, r, renderObject, "confirmation.html")
    }
}

Now this is working fine, but for some reason when a lot of new urls are saved, the sqlite database is telling me that a deadlock occurred. So I thought I want to make use of channels, just to prevent those dead locks. But unfortunately I'm not sure how to start with this.

Basically, what I want is:

  • for each added url, start the dao.ScrapeArticle function (pass the insertedID as the article reference. Immediately show the confirmation page (a.k.a. url added)
  • When scraping of the artcle is done, the function adds the scraped information to the sqlite database (by updating on ID)

I'm not sure how to start and need some help. F.e. where do I define the channel?

My scraper function looks like this:

func ScrapeArticle(db *sql.DB, id int64) {
    // time function duration
    start := time.Now()

    // storedArticle contains information stored in db which need to be updated through scraping
    storedArticle := getArticleById(db, id)

    // init goquery
    doc, err := goquery.NewDocument(storedArticle.Url.String)
    glog.Info("scraping article with url --> ", storedArticle.Url.String)
    if err != nil {
        glog.Error("error while scraping article with id %d -- > ", storedArticle.ID, err)
        return
    }

    // start scraping page title
    doc.Find("head").Each(func(i int, s *goquery.Selection) {
        pageTitle := s.Find("title").Text()
        storedArticle.Name = sql.NullString{String: strings.TrimSpace(pageTitle), Valid: true}
    })

    // now get meta description field
    doc.Find("meta").Each(func(i int, s *goquery.Selection) {
        if name, _ := s.Attr("name"); strings.EqualFold(name, "description") {
            description, _ := s.Attr("content")
            storedArticle.Description = sql.NullString{String: strings.TrimSpace(description), Valid: true}
        }
    })

    // if unable to scrape title, then use url
    if len(storedArticle.Name.String) == 0 {
        storedArticle.Name.String = storedArticle.Url.String
    }

    // if unable to scrape description, then use default text
    if len(storedArticle.Description.String) == 0 {
        storedArticle.Description.String = noDescription
    }

    // debugging info
    glog.Infof("scraped title --> %s (length: %d)", storedArticle.Name.String, len(storedArticle.Name.String))
    glog.Infof("scraped description --> %s (length: %d)", storedArticle.Description.String, len(storedArticle.Description.String))

    // after succesfull scraping, add page title (and more?) to article in db
    updateArticle(db, storedArticle)

    elapsed := time.Since(start)
    glog.Infof("scraping article %d completed in %s", storedArticle.ID.Int64, elapsed.String())
}

For basic and advanced usages of channels, see this presentation: http://talks.golang.org/2013/advconc.slide#1