Android CTS 測試研究之二
作者： 宋立新
Email ： [email protected]
前言
繼續較深入瞭解 CTS 的細節
先研究一下具體用法
命令行下敲 cts 進入命令行 ,
先看一下幫助
Help 一下
cts_host > help
Usage: command options
Avaiable commands and options:
Host:
help: show this message
exit: exit cts command line
Plan:
ls --plan: list available plans
ls --plan plan_name: list contents of the plan with specified name
add --plan plan_name: add a new plan with specified name
add --derivedplan plan_name -s/--session session_id -r/--result result_type: derive a plan from the given session
rm --plan plan_name/all: remove a plan or all plans from repository
start --plan test_plan_name: run a test plan
start --plan test_plan_name -d/--device device_ID: run a test plan using the specified device
start --plan test_plan_name -t/--test test_name: run a specific test
start --plan test_plan_name -p/--package java_package_name: run a specific java package
start --plan test_plan_name -t/--test test_name -d/--device device_ID: run a specific test using the specified device
start --plan test_plan_name -p/--package java_package_name -d/--device device_ID: run a specific java package using the specified device
Package:
ls -p/--package: list available packages
ls -p/--package package_name: list contents of the package with specified name
add -p/--package root: add packages from root to repository
rm -p/--package package_name/all: remove a package or all packages from repository
Result:
ls -r/--result: list all result of sessions
ls -r/--result -s/--session session_id: list detail case result of a specified session
ls -r/--result [pass/fail/notExecuted/timeout] -s/--session session_id: list detail cases of a specified session by the specified result.
History:
history/h: list all commands in command history
history/h count: list the latest count records in command history
history/h -e num: run the command designated by 'num' in command history
Device:
ls -d/--device: list available devices
列出所有的 plan
cts_host > ls --plan
List of plans (8 in total):
CTS
Signature
AppSecurity
Performance
RefApp
Java
VM
Android
查看某 plan 的內容
cts_host > ls --plan Android
Packages of plan Android (29 in total):
=================================
android.apidemos.cts
android.accessibilityservice
android.accounts
android.app
android.bluetooth
android.content
android.database
android.dpi
android.dpi2
android.example
android.gesture
android.graphics
android.hardware
android.jni
android.location
android.media
android.net
android.os
android.permission2
android.permission
android.provider
android.speech
android.telephony
android.text
android.util
android.view
android.webkit
android.widget
android.tests.appsecurity
添加一個新的 plan
cts_host > add --plan marvell
[Choose package] SignatureTest: select[Y], reject[n], or choose suite in it[m]? [Y/n/m] Y
Invalid input. Please chose 'y', 'n', or 'm' (default is 'y')
[Choose package] SignatureTest: select[Y], reject[n], or choose suite in it[m]? [Y/n/m] y
[Choose package] android.accessibilityservice: select[Y], reject[n], or choose suite in it[m]? [Y/n/m] y
[Choose package] android.accounts: select[Y], reject[n], or choose suite in it[m]? [Y/n/m] y
[Choose package] android.apidemos.cts: select[Y], reject[n], or choose suite in it[m]? [Y/n/m]
[Choose package] android.app: select[Y], reject[n], or choose suite in it[m]? [Y/n/m] y
。。。。
[Choose package] android.widget: select[Y], reject[n], or choose suite in it[m]? [Y/n/m]
cts_host > ls --plan marvell
The following package(s) contained in plan marvell have been removed:
SignatureTest
Packages of plan marvell (55 in total):
=================================
android.accessibilityservice
android.accounts
android.apidemos.cts
android.app
。。。
android.widget
刪除一個 plan
cts_host > rm --plan marvell
執行一個 plan
cts_host > start --plan Performance
start test plan Performance
==============================================================
Test package: android.performance2
android.performance2.cts.AppStartup#testStartup........................................................................................................................................................(timeout)
==============================================================
CTS_INFO >>> Max ADB operations reached. Restarting ADB...
CTS_INFO >>> Restarting device ...
CTS_INFO >>> Restart complete.
==============================================================
Test package: android.performance3
android.performance3.cts.AppStartup#testStartup..............(pass)
CTS_INFO >>> Max ADB operations reached. Restarting ADB...
CTS_INFO >>> Restarting device ...
CTS_INFO >>> Restart complete.
==============================================================
Test package: android.performance4
android.performance4.cts.AppStartup#testStartup.......(pass)
CTS_INFO >>> Max ADB operations reached. Restarting ADB...
CTS_INFO >>> Restarting device ...
CTS_INFO >>> Restart complete.
==============================================================
Test package: android.performance5
android.performance5.cts.AppStartup#testStartup........(fail)
junit.framework.AssertionFailedError: App Took too long to startup: 715 650 at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)
at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)
at java.lang.reflect.Method.invokeNative(Native Method)
at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)
at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)
at java.lang.reflect.Method.invokeNative(Native Method)
at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)
at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:1404)
at android.app.Instrumentation.execStartActivity(Instrumentation.java:1378)
at android.app.ApplicationContext.startActivity(ApplicationContext.java:555)
at android.performance.cts.MultiAppStartupTest.launchActivity(MultiAppStartupTest.java:52)
at android.performance.cts.MultiAppStartupTest.testMultipleApps(MultiAppStartupTest.java:89)
at java.lang.reflect.Method.invokeNative(Native Method)
at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)
at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
Test summary: pass=2 fail=2 timeOut=1 notExecuted=0 Total=5
Time: 635.927s
查看有多少個包
cts_host > ls -p
Available packages (56 in total):
android.util
android.hardware
。。。
android.core.tests.luni.net
android.jni
android.core.vm-tests
查看某個包
cts_host > ls -p android.net
Test suites (3 in total):
android.net.cts
android.net.wifi.cts
android.net.http.cts
查看測試結果
cts_host > ls -r
List of all results:
Session Test result Start time End time Test plan name
Pass Fail Timeout NotExecuted
1 2 2 1 0 2010.06.01 16:18:03 2010.06.01 16:28:39 Performance
2 1 0 0 0 2010.06.01 16:39:38 2010.06.01 16:41:33 Signature
研究 cts 生成文件
cts 腳本文件位於： ~/mydroid/out/host/linux-x86/bin ，它是一個 100 行不到的腳本，最終執行：
java -Xmx512M
-cp ./../framework/ddmlib.jar:./../framework/junit.jar:./../framework/hosttestlib.jar:./../framework/cts.jar:./cts.jar
-DHOST_CONFIG=./../cts/android-cts/repository/host_config.xml
com.android.cts.TestHost
可見， CTS 的 Java 主程序爲： com.android.cts.TestHost ，同時使用了配置文件： host_config.xml.
兩個 cts.jar: ./out/host/linux-x86/cts/android-cts/tools/cts.jar ./out/host/linux-x86/framework/cts.jar 相同。
cts 使用的目錄則爲： ~/mydroid/out/host/linux-x86/cts
文件列表 1
songlixin @zjujoe-desktop:~/mydroid/out/host/linux-x86/cts$ tree -L 3
.
├── all_cts_core_files_stamp
├── all_cts_files_stamp
├── android-cts
│ ├── docs
│ ├── repository
│ │ ├── host_config.xml
│ │ ├── log_2010.06.01_16.02.21_.txt
│ │ ├── log_2010.06.01_16.12.22_.txt
│ │ ├── log_2010.06.01_16.38.30_.txt
│ │ ├── log_2010.06.01_16.59.50_.txt
│ │ ├── plans
│ │ ├── results
│ │ └── testcases
│ └── tools
│ ├── cts.jar
│ ├── hosttestlib.jar
│ ├── junit.jar
│ └── startcts
├── android-cts.zip
└── temp
└── description.xml
我們關心的主要是 repository 目錄。
文件列表 2
songlixin @zjujoe-desktop:~/mydroid/out/host/linux-x86/cts$ tree android-cts/repository/
android-cts/repository/
├── host_config.xml
├── plans
│ ├── Android.xml
│ ├── AppSecurity.xml
│ ├── CTS.xml
│ ├── Java.xml
│ ├── Performance.xml
│ ├── RefApp.xml
│ ├── Signature.xml
│ └── VM.xml
├── results
│ ├── 2010.06.01_16.18.03
│ │ ├── cts_result.css
│ │ ├── cts_result.xsl
│ │ ├── logo.gif
│ │ ├── newrule-green.png
│ │ └── testResult.xml
│ ├── 2010.06.01_16.18.03.zip
│ ├── 2010.06.01_16.39.38
│ │ ├── cts_result.css
│ │ ├── cts_result.xsl
│ │ ├── logo.gif
│ │ ├── newrule-green.png
│ │ └── testResult.xml
│ ├── 2010.06.01_16.39.38.zip
│ ├── cts_result.css
│ ├── cts_result.xsl
│ ├── logo.gif
│ └── newrule-green.png
└── testcases
├── android.core.tests.annotation.apk
├── android.core.tests.annotation.xml
├── android.core.tests.archive.apk
。。。。。。。。。。。。。。。。。。
├── CtsWidgetTestCases.xml
├── SignatureTest.apk
├── SignatureTest.xml
└── TestDeviceSetup.apk
5 directories, 156 files
可見 plans 目錄下爲以 xml 格式定義的測試方案
results 目錄爲 測試結果。
testcases 目錄則爲一個個測試用例包
host_config.xml 爲配置文件
研究 cts 源文件
從根目錄開始
Cts 的 source 包位於 android 根目錄下：
songlixin @zjujoe-desktop:~/mydroid/cts$ tree -L 2
.
├── Android.mk
├── tests
│ ├── accessibilityservice
│ ├── AndroidManifest.xml
│ ├── Android.mk
│ ├── ApiDemosReferenceTest
│ ├── appsecurity-tests
│ ├── assets
│ ├── config_demo
│ ├── core
│ ├── ProcessTest
│ ├── res
│ ├── SignatureTest
│ ├── src
│ ├── tests
│ └── vm-tests
└── tools
├── Android.mk
├── annotation-helper
├── cts-reference-app-lib
├── dasm
├── device-setup
├── dex-tools
├── dx-tests
├── host
├── signature-tools
├── spec-progress
├── test-progress
├── test-progress-new
├── utils
└── vm-tests
根目錄下的 make file 非常簡單：
songlixin @zjujoe-desktop:~/mydroid/cts$ cat Android.mk
include $(call all-subdir-makefiles)
看來主要內容在兩個子目錄下。
子目錄 1 ： tools
根目錄下的 Android.mk 與其父目錄相同。
各子目錄代表不同的工具，比如 dasm 編譯後的位置是： ./out/host/linux-x86/bin/dasm
這裏就不仔細分析這些工具了。
子目錄 2 ： tests
看一下根目錄下的 Android.mk:
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := tests
LOCAL_SRC_FILES := $(call all-java-files-under, src)/
$(call all-java-files-under, core/runner/src)/
src/android/app/cts/ISecondary.aidl/
src/android/os/cts/IEmptyService.aidl
LOCAL_JAVA_LIBRARIES := android.test.runner
# Resource unit tests use a private locale and some densities
LOCAL_AAPT_FLAGS = -c xx_YY -c cs -c 32dpi -c 240dpi -c 160dpi
LOCAL_PACKAGE_NAME := CtsTestStubs
include $(BUILD_PACKAGE)
# Build the test APK using its own makefile, and any other CTS-related packages
include $(call all-makefiles-under,$(LOCAL_PATH))
所有 src 及 core/runner/src 下的 java 文件被編譯成：
./out/host/linux-x86/cts/android-cts/repository/testcases/CtsTestStubs.apk
剩下的就是很多子目錄以及子目錄的子目錄會被編譯成 *.apk
Tests 目錄下每個子目錄包括一個 test suite.
songlixin @zjujoe-desktop:~/mydroid/cts/tests/tests$ tree -L 1
.
├── accessibilityservice
├── accounts
├── Android.mk
。。。
├── webkit
└── widget
Core 目錄下每個子目錄包含一個 test suite, 但是他們的代碼在其他地方
songlixin @zjujoe-desktop:~/mydroid/cts/tests/core$ tree -L 2
.
├── Android.mk
├── annotation
│ ├── AndroidManifest.xml
│ └── Android.mk
├── archive
│ ├── AndroidManifest.xml
│ └── Android.mk
。。。
├── text
│ ├── AndroidManifest.xml
│ └── Android.mk
├── xml
│ ├── AndroidManifest.xml
│ └── Android.mk
└── xnet
├── AndroidManifest.xml
└── Android.mk
appsecurity-tests/test-apps 目錄下每個子目錄包括一個 test case. 都特別簡單 .
songlixin @zjujoe-desktop:~/mydroid/cts/tests/appsecurity-tests/test-apps$ tree -L 1
.
├── Android.mk
├── AppAccessData
├── AppWithData
├── InstrumentationAppDiffCert
├── PermissionDeclareApp
├── SharedUidInstall
├── SharedUidInstallDiffCert
├── SimpleAppInstall
├── SimpleAppInstallDiffCert
├── TargetInstrumentationApp
└── UsePermissionDiffCert
總結
Cts 其實就是基於 instrumentation test, Test case 有很多類型，不同的實現方式。 對我們來說，需要：
1 ） 能夠執行現有的 test case
2 ） 能夠模仿 api demo test case 寫一些比較簡單的：
可以參考： http://www.netmite.com/ /
android/mydroid/cupcake/development/pdk/docs/instrumentation_testing.html