梯度下降解決線性迴歸，以及自適應學習率設置

線性迴歸一般用於預測：比如股票漲停
梯度下降是機器學習中最核心的優化算法

函數解析：
np.random.normal(size,loc,scale):
給出均值爲loc，標準差爲scale的高斯隨機數
1.參數loc(float)：正態分佈的均值，對應着這個分佈的中心。loc=0說明這一個以Y軸爲對稱軸的正態分佈，
2.參數scale(float)：正態分佈的標準差，對應分佈的寬度，scale越大，正態分佈的曲線越矮胖，scale越小，曲線越高瘦。
3.參數size(int 或者整數元組)：輸出的值賦在shape裏，默認爲None。

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
#生成100個隨機數點
#方程爲 y=0.1*x+0.2
#權重 0.1 偏差0.2
points = 100
vectors = []
for i in range(points):
    x1 = np.random.normal(0.0,0.66)
    y1 = x1*0.1 +0.2 +np.random.normal(0.0,0.04)
    vectors.append([x1,y1])
x_data = [v[0] for v in vectors]
y_data = [v[1] for v in vectors]
plt.plot(x_data,y_data,'r*',label="Original data")
# plt.title('Interesting Graph',color='blue')
plt.legend()
plt.show()

此時畫出來的圖像如下：

接下來需要通過訓練得出擬合直線，越接近 y=0.1x+0.2最好
w爲-1到1的單隨機數組
b爲初始化爲[0]的數組
loss爲損失函數，均方差

梯度下降法是一個一階最優化算法，通常也稱爲最速下降法。要使用梯度下降法找到一個函數的局部極小值，必須向函數上當前點對於梯度（或者是近似梯度）的反方向的規定步長距離點進行迭代搜索。所以梯度下降法可以幫助我們求解某個函數的極小值或者最小值。對於n維問題就最優解，梯度下降法是最常用的方法之一。
關於梯度下降學習率的設置參考：
https://segmentfault.com/a/1190000011994447
https://blog.csdn.net/william_hehe/article/details/78658421?locationNum=9&fps=1
https://vimsky.com/article/3788.html
https://cloud.tencent.com/developer/ask/116497/answer/209443

w = tf.Variable(tf.random_uniform([1],-1.0,1.0))
b = tf.Variable(tf.zeros([1]))
y= w*x_data+b
#損失函數
loss=tf.reduce_mean(tf.square(y-y_data))
#梯度下降優化
optimizer=tf.train.GradientDescentOptimizer(0.5)
train=optimizer.minimize(loss)

sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
for step in range(20):
    sess.run(train)
    print("step:%d,loss:%f,weight:%f,bias:%f"%(step ,sess.run(loss),sess.run(w),sess.run(b)))
    
plt.plot(x_data,y_data,'r*',label="Original data")
plt.title('Interesting Graph',color='blue')
plt.plot(x_data,x_data*sess.run(w)+sess.run(b),label="Fittedd line")
plt.xlabel('x')
plt.ylabel('y')
plt.show()
sess.close()

畫出的圖像如下：

下面給出一種自適應學習率的方法：
主要用到的函數如下
learning_rate = tf.train.exponential_decay(
0.1, # Base learning rate. 基礎的學習率
global_steps, # Current index into the dataset.用於迭代
10, # Decay step.衰減步長
0.95, # Decay rate.這兩個參數的意思是每隔10次迭代就把基礎學習率*0.95
staircase=True)

w = tf.Variable(tf.random_uniform([1],-1.0,1.0))
b = tf.Variable(tf.zeros([1]))
y= w*x_data+b
loss=tf.reduce_mean(tf.square(y-y_data))

global_steps = tf.Variable(0, trainable=False)

learning_rate = tf.train.exponential_decay(
  0.1,                # Base learning rate.
  global_steps,  # Current index into the dataset.
  10,          # Decay step.
  0.95,                # Decay rate.
  staircase=True)

optimizer=tf.train.GradientDescentOptimizer(learning_rate)
train=optimizer.minimize(loss,global_step=global_steps)
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
for step in range(40):
    sess.run(train)
    print("step:%d,loss:%f,weight:%f,bias:%f"%(step ,sess.run(loss),sess.run(w),sess.run(b)))
    print(sess.run(learning_rate))
    
plt.plot(x_data,y_data,'r*',label="Original data")
plt.title('Interesting Graph',color='blue')
plt.plot(x_data,x_data*sess.run(w)+sess.run(b),label="Fittedd line")
plt.xlabel('x')
plt.ylabel('y')
plt.show()
sess.close()

打印結果如下：
step:0,loss:0.063258,weight:0.449633,bias:0.038551
0.1
step:1,loss:0.050180,weight:0.428209,bias:0.069480
0.1
step:2,loss:0.040675,weight:0.408036,bias:0.094305
0.1
step:3,loss:0.033597,weight:0.389057,bias:0.114242
0.1
step:4,loss:0.028194,weight:0.371216,bias:0.130264
0.1
step:5,loss:0.023971,weight:0.354453,bias:0.143150
0.1
step:6,loss:0.020598,weight:0.338712,bias:0.153524
0.1
step:7,loss:0.017851,weight:0.323938,bias:0.161883
0.1
step:8,loss:0.015578,weight:0.310076,bias:0.168626
0.1
step:9,loss:0.013672,weight:0.297074,bias:0.174075
0.095
step:10,loss:0.012133,weight:0.285492,bias:0.178262
0.095
step:11,loss:0.010808,weight:0.274598,bias:0.181697
0.095
step:12,loss:0.009660,weight:0.264353,bias:0.184518
0.095
step:13,loss:0.008661,weight:0.254720,bias:0.186841
0.095
step:14,loss:0.007788,weight:0.245663,bias:0.188758
0.095
step:15,loss:0.007024,weight:0.237150,bias:0.190343
0.095
step:16,loss:0.006353,weight:0.229147,bias:0.191658
0.095
step:17,loss:0.005764,weight:0.221626,bias:0.192752
0.095
step:18,loss:0.005246,weight:0.214558,bias:0.193666
0.095
step:19,loss:0.004790,weight:0.207915,bias:0.194432
0.09025
step:20,loss:0.004407,weight:0.201985,bias:0.195045
0.09025
step:21,loss:0.004068,weight:0.196395,bias:0.195567
0.09025
step:22,loss:0.003767,weight:0.191126,bias:0.196014
0.09025
step:23,loss:0.003500,weight:0.186159,bias:0.196399
0.09025
step:24,loss:0.003263,weight:0.181478,bias:0.196732
0.09025
step:25,loss:0.003053,weight:0.177065,bias:0.197020
0.09025
step:26,loss:0.002866,weight:0.172906,bias:0.197272
0.09025
step:27,loss:0.002700,weight:0.168986,bias:0.197493
0.09025
step:28,loss:0.002553,weight:0.165291,bias:0.197688
0.09025
step:29,loss:0.002422,weight:0.161809,bias:0.197860
0.0857375
step:30,loss:0.002312,weight:0.158691,bias:0.198005
0.0857375
step:31,loss:0.002213,weight:0.155743,bias:0.198136
0.0857375
step:32,loss:0.002125,weight:0.152957,bias:0.198254
0.0857375
step:33,loss:0.002046,weight:0.150323,bias:0.198361
0.0857375
step:34,loss:0.001975,weight:0.147833,bias:0.198458
0.0857375
step:35,loss:0.001912,weight:0.145479,bias:0.198547
0.0857375
step:36,loss:0.001856,weight:0.143254,bias:0.198628
0.0857375
step:37,loss:0.001806,weight:0.141150,bias:0.198703
0.0857375
step:38,loss:0.001761,weight:0.139162,bias:0.198772
0.0857375
step:39,loss:0.001721,weight:0.137282,bias:0.198835
0.08145062