Depoly your flask app into Heroku Fix shit IE code manually ISBN to Book Category by Scraping DangDang A Generic Makefile for C/C++ Program Configure Raspberry pi Remove watermark with PyPDF2 tips for docker Anaconda+TensorFlow+CUDA Snippets Configure Remote Mathematica Kernel Build your own ngrok server Access Array SSL VPN 使用Rstudio制作html5幻灯片 tips for Mac OS X system Tips for ipython notebook 配置Ubuntu server + Openbox (Obuntu) tips for Vimperator tips for Vim 安装CUDA My First Jekyll Blog rsync常见选项 在Linux中读取Ipod touch的文件 tip for texmacs 在VPS上建站的一些tip Gnuplot绘图札记 Samba系统和autofs自动挂载 Linux中alsamixer声卡无法录音 搭建自己的RSS订阅器——Tiny Tiny RSS Grub2引导安装Ubuntu awk tips 将Ubuntu系统装入U盘 The Great Rtorrent 编译GCC 再这样剁手!!!该死的libgd 使用ulimit进行资源限制 使用SSH代理上IPV6 使用RCurl抓取网页数据 修复Ubuntu Grub记 openbox中的文件关联 在Ubuntu 12.04下编译qtiplot 处理BCM4312网卡驱动纪实 配置我的Ubuntu Server记 Cygwin杂记 Linux 使普通用户具有以超级权限执行脚本 让firefox自定义地处理文件类型 WordPress优秀主题及插件 在phpcloud上搭建wordpress UBUNTU下用pptpd做VPN server ubuntu升级内核过后的一些问题 安装telnet服务 kubuntu札记 64位kubuntu札记 统计软件R virtualbox stardict星际译王 Ubuntu重装windows系统后的grub引导修复 SSH服务及花生壳域名解析 采用cbp2make工具由code::blocks工程创建makefile文件 UBUNTU 札记

ISBN to Book Category by Scraping DangDang


Just a very simple script to provide an API to scrape the book’s category from DangDang and publish it with Heroku.


The following code is quite ugly but I tend not to update it, just put it here for recording.

import requests
from bs4 import BeautifulSoup
from flask import Flask, Response, jsonify
import os

class JSONResponse(Response):
    def force_type(cls, response, environ=None):
        if isinstance(response, (list, dict, bool)):
            response = jsonify(response)
        return super(Response, cls).force_type(response, environ)

app = Flask(__name__)
app.response_class = JSONResponse
app.config['JSON_AS_ASCII'] = False

    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36',
    'DNT': '1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',

SEARCH_URL = '{isbn}'
requests_session = requests.session()

def get_class(isbn):
    # Search the isbn first
    r = requests_session.get(SEARCH_URL.format(isbn=isbn))
    soup = BeautifulSoup(r.text, 'lxml')

    # Assuming always find at least one and the first is what we want
    item = soup.select_one('a[name="itemlist-title"]')
    if not item:
        return None

    # Find the product page and extract the book category 
    item_url = item.get('href')
    r = requests_session.get(item_url)
    soup = BeautifulSoup(r.text, 'lxml')
    return [c.get_text() for c in soup.select_one('#detail-category-path > label').next_sibling()]

if __name__ == '__main__':'', port=os.environ.get("PORT", 6500))


Build the docker image with following Dockerfile and command docker build -t${APP_NAME}/web:latest, push to heroku with docker push${APP_NAME}/web:latest and then release with heroku container:release web -a ${APP_NAME}.

FROM vaeum/alpine-python3-pip3

RUN apk add --update --no-cache g++ gcc libxslt-dev && pip3 install requests flask beautifulsoup4 lxml


CMD ["python3", ""]

BTW, Google books also provide an API to check the books’ details.