百度街景圖片存MySQL

1 引言

本文記錄百度街景圖片的存儲過程，主要邏輯是：讀取本地圖片，存mysql，代碼直接下載稍作修改就可運行，存儲相關使用的是Java原生接口。更多代碼以及百度街景相關背景參考另一篇博客-----百度地圖街景圖片爬取並存庫。

2 建表

表字段主要分爲兩部分：當前街景部分和歷史街景部分，每部分都包含4大屬性：id、拍攝日期、圖片url(4張)、圖片二進制流。爲了更好的搜索，建立相關索引

CREATE TABLE `test`.`baidu_pano_pics`  (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增主鍵',
  `cur_pano_id` char(30) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '當前街景位置ID',
  `cur_pano_date` varchar(6) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '當前街景的拍攝日期',
  `cur_pic_url_1` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '圖片1url',
  `cur_pic_url_2` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '圖片2url',
  `cur_pic_url_3` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '圖片3url',
  `cur_pic_url_4` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '圖片4url',
  `cur_pic_1` mediumblob NULL COMMENT '圖片1(從左往右)',
  `cur_pic_2` mediumblob NULL COMMENT '圖片2',
  `cur_pic_3` mediumblob NULL COMMENT '圖片3',
  `cur_pic_4` mediumblob NULL COMMENT '圖片4',
  `his_pano_id` char(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史街景位置ID',
  `his_pano_date` varchar(6) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史街景的拍攝日期',
  `his_pic_url_1` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史圖片1url',
  `his_pic_url_2` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史圖片2url',
  `his_pic_url_3` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史圖片3url',
  `his_pic_url_4` char(128) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '歷史圖片4url',
  `his_pic_1` mediumblob NULL COMMENT '歷史街景位置圖片1',
  `his_pic_2` mediumblob NULL COMMENT '歷史街景位置圖片2',
  `his_pic_3` mediumblob NULL COMMENT '歷史街景位置圖片3',
  `his_pic_4` mediumblob NULL COMMENT '歷史街景位置圖片4',
  `timestamp` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '入庫時間戳',
  PRIMARY KEY USING BTREE (`id`),
  UNIQUE INDEX `pano_index` USING BTREE(`cur_pano_id`)
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8 COLLATE = utf8_general_ci COMMENT = '百度街景圖片存儲表' ROW_FORMAT = Compact;

3 代碼

先封裝一個數據庫連接的工具類DbUtil，ConfigParser類源碼移步至作者之前的爬蟲相關文章：

/**
 * @author [email protected]
 * @since 0.1.0
 **/
public class DbUtil {

    private final static Logger LOGGER  =  Logger.getLogger(DbUtil.class);

    private DbUtil(){}

    private static class DbUtilHolder{
        private static DbUtil singleton = new DbUtil();
    }

    public static DbUtil getInstance(){
        return DbUtilHolder.singleton;
    }


    // 獲取連接
    public Connection getConn() throws SQLException {
        ConfigParser parser = ConfigParser.getInstance();
        String dbAlias = "mysql-data";
        Map<String, Object> dbConfig = parser.getModuleConfig("database");
        Map<String, Object> mysqlConfig = (Map)parser.assertKey(dbConfig, dbAlias, "database");
        String url = (String)parser.assertKey(mysqlConfig, "url", "database." + dbAlias);
        String username = (String)parser.assertKey(mysqlConfig, "username", "database." + dbAlias);
        String password = (String)parser.assertKey(mysqlConfig, "password", "database." + dbAlias);
        return DriverManager.getConnection(url, username, password);
    }

    // 關閉連接
    public void closeConn(Connection conn) {
        if (null != conn) {
            try {
                conn.close();
            } catch (SQLException e) {
                LOGGER.error("關閉連接失敗！");
                e.printStackTrace();
            }
        }
    }
}

建立百度街景圖片的java bean，通過該類實現相關屬性的封裝或存取，PanoPic的屬性填充詳見前文，這裏不再贅述。

import java.util.List;

/**
 * @author [email protected]
 * @since 0.1.0
 **/
public class PanoPic {

    /**
     * 當前街景ID
     */
    private String curPanoId;

    /**
     *當前街景拍攝時間
     */
    private String curShootDate;

    /**
     * 當前街景本地存儲路徑
     */
    private List<String> curPicPath;

    /**
     * 歷史街景ID
     */
    private String hisPanoId;

    /**
     * 歷史街景拍攝時間
     */
    private String hisShootDate;

    /**
     * 歷史街景本地存儲路徑
     */
    private List<String> hisPicPath;

    public PanoPic(String curPanoId,String curShootDate,List<String> curPicPath){
        this.curPanoId = curPanoId;
        this.curShootDate = curShootDate;
        this.curPicPath = curPicPath;
    }

    public String getCurPanoId() {
        return curPanoId;
    }

    public void setCurPanoId(String curPanoId) {
        this.curPanoId = curPanoId;
    }

    public String getCurShootDate() {
        return curShootDate;
    }

    public void setCurShootDate(String shootDate) {
        this.curShootDate = shootDate;
    }

    public List<String> getCurPicPath() {
        return curPicPath;
    }

    public void setCurPicPath(List<String> curPicPath) {
        this.curPicPath = curPicPath;
    }

    public String getHisPanoId() {
        return hisPanoId;
    }

    public void setHisPanoId(String hisPanoId) {
        this.hisPanoId = hisPanoId;
    }

    public String getHisShootDate() {
        return hisShootDate;
    }

    public void setHisShootDate(String hisShootDate) {
        this.hisShootDate = hisShootDate;
    }

    public List<String> getHisPicPath() {
        return hisPicPath;
    }

    public void setHisPicPath(List<String> hisPicPath) {
        this.hisPicPath = hisPicPath;
    }
}

街景圖片存儲的工具類封裝如下：

/**
 * @author [email protected]
 * @since 0.1.0
 **/
public class BlobInsertUtils {

    private final static Logger LOGGER  =  Logger.getLogger(BlobInsertUtils.class);
    private final String SQL_NO_HIS;
    private final String SQL_WITH_HIS;

    public BlobInsertUtils(String tableName){
        SQL_NO_HIS = getSQLWithoutHis(tableName);
        SQL_WITH_HIS = getSqlWithHis(tableName);
    }

    /**
     * 將當前街景圖片插入數據庫
     */
    public void readImage2DB(String tableName,String panoId, List<String> picPath) {
        Connection conn = null;
        PreparedStatement ps = null;
        ImagePersistence imagePersistence = new ImagePersistence();
        List<FileInputStream> fileInputStreamsList = new ArrayList<>(4);
        FileInputStream fileInputStream;
        try {
            for (String pic:picPath){
                fileInputStreamsList.add(imagePersistence.readImage(pic));
            }
            conn = DbUtil.getInstance().getConn();
            String sql = "insert into "+ tableName +
                    "(pano_id,pic_url_1,pic_url_2,pic_url_3,pic_url_4,pic_1,pic_2,pic_3,pic_4,timestamp)values(?,?,?,?,?,?,?,?,?,?)";
            ps = conn.prepareStatement(sql);
            ps.setString(1, panoId);
            for (int i = 0; i < picPath.size(); i++){
                fileInputStream = fileInputStreamsList.get(i);
                ps.setString(2+i, picPath.get(i));
                ps.setBinaryStream(6+i, fileInputStream, fileInputStream.available());
            }
            ps.setTimestamp(10,new Timestamp(System.currentTimeMillis()));
            int count = ps.executeUpdate();
            if (count > 0) {
                LOGGER.info(panoId + "插入成功！");
            } else {
                LOGGER.info(panoId + "插入失敗！");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            DbUtil.getInstance().closeConn(conn);
            if (null != ps) {
                try {
                    ps.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                }
            }
        }
    }

    private String getSql(String tableName){
        return "insert into " + tableName +
                "(cur_pano_id,cur_pano_date,cur_pic_url_1,cur_pic_url_2,cur_pic_url_3,cur_pic_url_4,cur_pic_1,cur_pic_2,cur_pic_3,cur_pic_4,"+
                "(his_pano_id,his_pano_date,his_pic_url_1,his_pic_url_2,his_pic_url_3,his_pic_url_4,his_pic_1,his_pic_2,his_pic_3,his_pic_4,"+
                "timestamp)"+
                "values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"+
                "on duplicate key update"+
                "cur_pano_id=?,cur_pano_date=?,cur_pic_url_1=?,cur_pic_url_2=?,cur_pic_url_3=?,cur_pic_url_4=?,cur_pic_1=?,cur_pic_2=?,cur_pic_3=?,cur_pic_4=?"+
                "his_pano_id=?,his_pano_date=?,his_pic_url_1=?,his_pic_url_2=?,his_pic_url_3=?,his_pic_url_4=?,his_pic_1=?,his_pic_2=?,his_pic_3=?,his_pic_4=?";
    }

    private String getSqlWithHis(String tableName){
        return "insert into " + tableName +
                "(cur_pano_id,cur_pano_date,cur_pic_url_1,cur_pic_url_2,cur_pic_url_3,cur_pic_url_4,cur_pic_1,cur_pic_2,cur_pic_3,cur_pic_4,"+
                "his_pano_id,his_pano_date,his_pic_url_1,his_pic_url_2,his_pic_url_3,his_pic_url_4,his_pic_1,his_pic_2,his_pic_3,his_pic_4,"+
                "timestamp)"+
                "values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
    }

    private String getSQLWithoutHis(String tableName){
        return "insert into " + tableName +
                "(cur_pano_id,cur_pano_date,cur_pic_url_1,cur_pic_url_2,cur_pic_url_3,cur_pic_url_4,cur_pic_1,cur_pic_2,cur_pic_3,cur_pic_4,"+
                "timestamp)"+
                "values(?,?,?,?,?,?,?,?,?,?,?)";
    }

    /**
     * 將當前及歷史街景圖片插入數據庫
     *
     * @param panoPic bean
     */
    public void insertAllImage2DBWithNoCheck(PanoPic panoPic) {
        Connection conn = null;
        PreparedStatement ps = null;
        ImagePersistence imagePersistence = new ImagePersistence();
        int allPicNum;
        boolean isHisEmpty = true;
        List<String> curPicPath = panoPic.getCurPicPath();
        List<String> hisPicPath = null;
        if (null==panoPic.getHisPicPath()){
            allPicNum = curPicPath.size();
        }else {
            isHisEmpty = false;
            hisPicPath = panoPic.getHisPicPath();
            allPicNum = curPicPath.size() + panoPic.getHisPicPath().size();
        }
        List<FileInputStream> fileInputStreamsList = new ArrayList<>(allPicNum);
        FileInputStream fileInputStream;
        try {
            //加載當前街景輸入流
            for (String curPic:curPicPath){
                fileInputStreamsList.add(imagePersistence.readImage(curPic));
            }
            conn = DbUtil.getInstance().getConn();
            //加載歷史街景輸入流
            if (!isHisEmpty){
                for (String hisPic:panoPic.getHisPicPath()){
                    fileInputStreamsList.add(imagePersistence.readImage(hisPic));
                }
                ps = conn.prepareStatement(SQL_WITH_HIS);
            }else {
                ps = conn.prepareStatement(SQL_NO_HIS);
            }
            ps.setString(1, panoPic.getCurPanoId());
            ps.setString(2, panoPic.getCurShootDate());
            Timestamp timestamp = new Timestamp(System.currentTimeMillis());
            for (int i = 0; i < curPicPath.size(); i++){
                fileInputStream = fileInputStreamsList.get(i);
                ps.setString(3+i, curPicPath.get(i));
                ps.setBinaryStream(7+i, fileInputStream, fileInputStream.available());
            }
            if (!isHisEmpty){
                ps.setString(11, panoPic.getHisPanoId());
                ps.setString(12, panoPic.getHisShootDate());
                for (int j = 0; j < hisPicPath.size(); j++){
                    fileInputStream = fileInputStreamsList.get(4+j);
                    ps.setString(13+j, hisPicPath.get(j));
                    ps.setBinaryStream(17+j, fileInputStream, fileInputStream.available());
                }
                ps.setTimestamp(21,timestamp);
            }else {
                ps.setTimestamp(11,timestamp);
            }
            int count = ps.executeUpdate();
            if (count > 0) {
                LOGGER.info(panoPic.getCurPanoId() + "插入成功!");
            } else {
                LOGGER.info(panoPic.getCurPanoId() + "插入失敗!");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            DbUtil.getInstance().closeConn(conn);
            if (null != ps) {
                try {
                    ps.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                }
            }
        }
    }


    /**
     * 將當前及歷史街景圖片插入數據庫
     * 已有則更新
     * @param panoPic bean
     */
    public void readAllImage2DB(PanoPic panoPic) {
        Connection conn = null;
        PreparedStatement ps = null;
        ImagePersistence imagePersistence = new ImagePersistence();
        int allPicNum;
        boolean isHisEmpty = true;
        List<String> curPicPath = panoPic.getCurPicPath();
        List<String> hisPicPath = null;
        if (null==panoPic.getHisPicPath()){
            allPicNum = panoPic.getCurPicPath().size();
        }else {
            isHisEmpty = false;
            hisPicPath = panoPic.getHisPicPath();
            allPicNum = panoPic.getCurPicPath().size() + panoPic.getHisPicPath().size();
        }
        List<FileInputStream> fileInputStreamsList = new ArrayList<>(allPicNum);
        FileInputStream fileInputStream;
        try {
            //加載當前街景輸入流
            for (String curPic:curPicPath){
                fileInputStreamsList.add(imagePersistence.readImage(curPic));
            }
            //加載歷史街景輸入流
            if (!isHisEmpty){
                for (String hisPic:panoPic.getHisPicPath()){
                    fileInputStreamsList.add(imagePersistence.readImage(hisPic));
                }
            }
            conn = DbUtil.getInstance().getConn();
            ps.setString(1, panoPic.getCurPanoId());
            ps.setString(2, panoPic.getCurShootDate());
            Timestamp timestamp = new Timestamp(System.currentTimeMillis());
            for (int i = 0; i < curPicPath.size(); i++){
                fileInputStream = fileInputStreamsList.get(i);
                ps.setString(3+i, curPicPath.get(i));
                ps.setBinaryStream(7+i, fileInputStream, fileInputStream.available());
            }
            if (!isHisEmpty){
                ps.setString(11, panoPic.getHisPanoId());
                ps.setString(12, panoPic.getHisShootDate());
                for (int j = 0; j < hisPicPath.size(); j++){
                    fileInputStream = fileInputStreamsList.get(4+j);
                    ps.setString(13+j, hisPicPath.get(j));
                    ps.setBinaryStream(17+j, fileInputStream, fileInputStream.available());
                }
                ps.setTimestamp(21,timestamp);
                //on duplicate key update 後續參數拼接
                ps.setString(22, panoPic.getCurPanoId());
                ps.setString(23, panoPic.getCurShootDate());
                for (int i = 0; i < curPicPath.size(); i++){
                    fileInputStream = fileInputStreamsList.get(i);
                    ps.setString(24+i, curPicPath.get(i));
                    ps.setBinaryStream(28+i, fileInputStream, fileInputStream.available());
                }
                ps.setString(32, panoPic.getHisPanoId());
                ps.setString(33, panoPic.getHisShootDate());
                for (int j = 0; j < hisPicPath.size(); j++){
                    fileInputStream = fileInputStreamsList.get(4+j);
                    ps.setString(34+j, hisPicPath.get(j));
                    ps.setBinaryStream(38+j, fileInputStream, fileInputStream.available());
                }
            }else {
                ps.setTimestamp(21,timestamp);
                //on duplicate key update 後續參數拼接
                ps.setString(22, panoPic.getCurPanoId());
                ps.setString(23, panoPic.getCurShootDate());
                for (int i = 0; i < curPicPath.size(); i++){
                    fileInputStream = fileInputStreamsList.get(i);
                    ps.setString(24+i, curPicPath.get(i));
                    ps.setBinaryStream(28+i, fileInputStream, fileInputStream.available());
                }
            }
            ps.setTimestamp(42,timestamp);
            int count = ps.executeUpdate();
            if (count > 0) {
                LOGGER.info(panoPic.getCurPanoId() + "插入成功!");
            } else {
                LOGGER.info(panoPic.getCurPanoId() + "插入失敗!");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            DbUtil.getInstance().closeConn(conn);
            if (null != ps) {
                try {
                    ps.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

這裏有個輸入流的封裝(線程安全)，傳入文件路徑參數，用於讀取本地圖片，源碼如下：

/**
 * @author [email protected]
 * @since 0.1.0
 **/
public class ImagePersistence {

    // 讀取本地圖片獲取輸入流
    public FileInputStream readImage(String path) throws IOException {
        return new FileInputStream(new File(path));
    }

    // 讀取表中圖片獲取輸出流
    public void readBin2Image(InputStream in, String targetPath) {
        File file = new File(targetPath);
        String path = targetPath.substring(0, targetPath.lastIndexOf("/"));
        if (!file.exists()) {
            new File(path).mkdir();
        }
        FileOutputStream fos = null;
        try {
            fos = new FileOutputStream(file);
            int len = 0;
            byte[] buf = new byte[1024];
            while ((len = in.read(buf)) != -1) {
                fos.write(buf, 0, len);
            }
            fos.flush();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (null != fos) {
                try {
                    fos.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

最後封裝整個街景圖片存儲邏輯示例調用如下：

private void storePanoPicsToDb(PanoPic panoPic,String tableName){
    //讀本地圖片存數據庫
    BlobInsertUtils blobInsertUtils = new BlobInsertUtils(tableName);
    blobInsertUtils.insertAllImage2DBWithNoCheck(panoPic);
}

3.1 注意

BlobInsertUtils類中有些插入方法是沒有用到的，本文只調用了insertAllImage2DBWithNoCheck方法。readAllImage2DB方法和insertAllImage2DBWithNoCheck方法區別是考慮到爬蟲邏輯一般爲週期性的計劃任務（街景可能每週或每個月爬取1次），爲了實現插入時判斷數據庫表中是否已有記錄，readAllImage2DB採用on duplicate key update關鍵字建立插入語句（無則插入，有則更新），但本方法由於字段過多（超過了41個），調用該方法插入會報錯，解決辦法：重新建表，刪除不必要的字段（如：圖片路徑），重新封裝sql參數即可。

參考

blob類型數據存儲參考：
https://blog.csdn.net/qq_38785658/article/details/75009700
https://www.cnblogs.com/warrior4236/p/5682830.html

百度街景圖片存MySQL

1 引言

2 建表

3 代碼

3.1 注意

參考

使用neovim打造go ide(支持代碼跳轉, 代碼補全, 實時語法檢查)

挑戰程序設計競賽 2.3章習題 poj 3046 Ant Counting

Shell/Python中的用戶名獲取

MySQL和PostgreSQL設計規範

微服務後端接口開發及返回值規範札記

IntelliJ Idea遠程調試容器中的應用

okhttp3之java.net.SocketTimeoutException: timeout 異常處理

項目構建打包與容器部署

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結