iOS Vision framework x WWDC 24 Discover Swift enhancements in the Vision framework Session
Published in
45 min readAug 13, 2024
Vision framework 功能回顧 & iOS 18 新 Swift API 試玩
主題
Vision framework
Vision framework 是 Apple 整合機器學習的圖像辨識框架，讓開發者可以簡單快速地實現常見的圖像辨識功能；Vision framework 早在 iOS 11.0+ (2017/ iPhone 8) 就已推出，期間不斷地迭代優化，並完善與 Swift Concurrency 的特性整合提升執行效能，並且從 iOS 18.0 提供全新的 Swift Vision framework API 發揮 Swift Concurrency 最大效果。
Vision framework 特色
- 內建眾多圖片辨識、動態追蹤方法 (iOS 18 為止一共 31 種)
- On-Device 單純使用手機晶片運算，辨識過程不依賴雲端服務，快速又安全
- API 簡單好操作
- Apple 全平台均支援 iOS 11.0+, iPadOS 11.0+, Mac Catalyst 13.0+, macOS 10.13+, tvOS 11.0+, visionOS 1.0+
- 已發布多年 (2017~今) 且不斷更新
- 整合 Swift 語言特性提升運算效能
6 年前曾經小玩過：Vision 初探 — APP 頭像上傳 自動識別人臉裁圖 (Swift)
這次搭配 WWDC 24 Discover Swift enhancements in the Vision framework Session 重新回顧並結合新的 Swfit 特性再玩一次。
CoreML
Apple 還有另外一個 Framework 叫 CoreML，也是基於 On-Device 晶片的機器學習框架；但他可以讓你自己訓練想辨識的物件、文件模型，並將模型放到 App 中直接使用，有興趣的朋友也可以玩看看。(e.g. 即時文章分類、即時垃圾訊息檢測…)
p.s.
Vision：主要用於圖像分析任務，如臉部識別、條碼檢測、文本識別等。它提供了強大的 API 來處理和分析靜態圖像或視頻中的視覺內容。
VisionKit：專門用於處理與文件掃描相關的任務。它提供了一個掃描儀視圖控制器，可以用來掃描文檔，並生成高質量的 PDF 或圖像。
Vision framework 在 M1 機型上無法跑在模擬器，只能接實體手機測試；在模擬器環境執行會拋出 Could not create Espresso context Error，查官方論壇討論，沒找到解答。
因手邊沒有實體 iOS 18 裝置進行測試，所以本文中的所有執行結果都是使用舊的 (iOS 18 以前) 的寫法結果；如新寫法有出現錯誤再麻煩留言指教。
WWDC 2024 — Discover Swift enhancements in the Vision framework
本文是針對 WWDC 24 — Discover Swift enhancements in the Vision framework Session 的分享筆記，跟一些自己實驗的心得。
Introduction — Vision framework Features
人臉辨識、輪廓識別
圖像內容文字辨識
截至 iOS 18 為止，支援 18 種語言。
// 支援的語系列表
if #available(iOS 18.0, *) {
print(RecognizeTextRequest().supportedRecognitionLanguages.map { "\($0.languageCode!)-\(($0.region?.identifier ?? $0.script?.identifier)!)" })
} else {
print(try! VNRecognizeTextRequest().supportedRecognitionLanguages())
}
// 實際可用辨識語言以這為主。
// 實測 iOS 18 輸出以下結果：
// ["en-US", "fr-FR", "it-IT", "de-DE", "es-ES", "pt-BR", "zh-Hans", "zh-Hant", "yue-Hans", "yue-Hant", "ko-KR", "ja-JP", "ru-RU", "uk-UA", "th-TH", "vi-VT", "ar-SA", "ars-SA"]
// 未看到 WWDC 提到的 Swedish 語言，不確定是還沒推出還是跟裝置地區、語系有關聯動態動作捕捉
- 可以實現人、物件動態捕捉
- 手勢補捉實現隔空簽名功能
What’s new in Vision? (iOS 18)— 圖片評分功能 (品質、記憶點)
- 可對輸入圖片得計算出分數，方便篩選出優質照片
- 分數計算方式包含多個維度，不只是畫質，還有光線、角度、拍攝主體、是否有讓人感到的記憶點…等等
WWDC 中給了以上三張圖片做說明(相同畫質之下)，分別是：
- 高分的圖片：取景、光線、有記憶點
- 低分的圖片：沒有主體、像是隨手或不小心拍的
- 素材的圖片：技術上拍的很好但是沒有記憶點，像是作為素材圖庫用的圖片
iOS ≥ 18 New API: CalculateImageAestheticsScoresRequest
let request = CalculateImageAestheticsScoresRequest()
let result = try await request.perform(on: URL(string: "https://zhgchg.li/assets/cb65fd5ab770/1*yL3vI1ADzwlovctW5WQgJw.jpeg")!)
// 照片分數
print(result.overallScore)
// 是否被判定為素材圖片
print(result.isUtility)What’s new in Vision? (iOS 18) — 身體＋手勢姿勢同時偵測
以往只能個別偵測人體 Pose 和 手部 Pose，這次更新可以讓開發者同時偵測身體 Pose + 手部 Pose，合成同一個請求跟結果，方便我們做更多應用功能開發。
iOS ≥ 18 New API: DetectHumanBodyPoseRequest
var request = DetectHumanBodyPoseRequest()
// 一併偵測手部 Pose
request.detectsHands = true
guard let bodyPose = try await request.perform(on: image). first else { return }
// 身體 Pose Joints
let bodyJoints = bodyPose.allJoints()
// 左手 Pose Joints
let leftHandJoints = bodyPose.leftHand.allJoints()
// 右手 Pose Joints
let rightHandJoints = bodyPose.rightHand.allJoints()New Vision API
Apple 在這次的更新當中提供了新的 Swift Vision API 封裝給開發者使用，除了基本的包含原本的功能支援之外，主要針對加強 Swift 6 / Swift Concurrency 的特性，提供效能更優、寫起來更 Swift 的 API 操作方式。
Get started with Vision
這邊講者又重新介紹了一次 Vision framework 的基礎使用方式，Apple 已經封裝好了 31 種(截至 iOS 18)常見的圖像辨識請求「Request」與對應回傳的「Observation」物件。
- Request: DetectFaceRectanglesRequest 人臉區域識別請求
Result: FaceObservation
之前的文章「Vision 初探 — APP 頭像上傳 自動識別人臉裁圖 (Swift)」就是用這對請求。 - Request: RecognizeTextRequest 文字辨識請求
Result: RecognizedTextObservation - Request: GenerateObjectnessBasedSaliencyImageRequest 主體物件辨識請求
Result: SaliencyImageObservation
全部 31 種請求 Request：
- 前面補上 VN 就是舊的 API 寫法 (iOS 18 以前的版本)
講者提到了幾個常用的 Request，如下。
ClassifyImageRequest
辨識輸入的圖片，得到標籤分類與置信度。
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
let request = ClassifyImageRequest()
Task {
do {
let observations = try await request.perform(on: URL(string: "https://zhgchg.li/assets/cb65fd5ab770/1*yL3vI1ADzwlovctW5WQgJw.jpeg")!)
observations.forEach {
observation in
print("\(observation.identifier): \(observation.confidence)")
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let completionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNClassificationObservation] else {
return
}
observations.forEach {
observation in
print("\(observation.identifier): \(observation.confidence)")
}
}
let request = VNClassifyImageRequest(completionHandler: completionHandler)
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: URL(string: "https://zhgchg.li/assets/cb65fd5ab770/1*3_jdrLurFuUfNdW4BJaRww.jpeg")!, options: [:])
do {
try handler.perform([request])
}
catch {
print("Request failed: \(error)")
}
}
}分析結果：
• outdoor（戶外）: 0.75392926
• sky（天空）: 0.75392926
• blue_sky（藍天）: 0.7519531
• machine（機器）: 0.6958008
• cloudy（多雲）: 0.26538086
• structure（結構）: 0.15728651
• sign（標誌）: 0.14224191
• fence（柵欄）: 0.118652344
• banner（橫幅）: 0.0793457
• material（材料）: 0.075975396
• plant（植物）: 0.054406323
• foliage（樹葉）: 0.05029297
• light（光）: 0.048126098
• lamppost（燈柱）: 0.048095703
• billboards（廣告牌）: 0.040039062
• art（藝術）: 0.03977703
• branch（樹枝）: 0.03930664
• decoration（裝飾）: 0.036868922
• flag（旗幟）: 0.036865234
....略RecognizeTextRequest
辨識圖片中的文字內容。(a.k.a 圖片轉文字)
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = [.init(identifier: "ja-JP"), .init(identifier: "en-US")] // Specify language code, e.g., Traditional Chinese
Task {
do {
let observations = try await request.perform(on: URL(string: "https://zhgchg.li/assets/9da2c51fa4f2/1*fBbNbDepYioQ-3-0XUkF6Q.jpeg")!)
observations.forEach {
observation in
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "No text recognized")
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let completionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNRecognizedTextObservation] else {
return
}
observations.forEach {
observation in
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "No text recognized")
}
}
let request = VNRecognizeTextRequest(completionHandler: completionHandler)
request.recognitionLevel = .accurate
request.recognitionLanguages = ["ja-JP", "en-US"] // Specify language code, e.g., Traditional Chinese
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: URL(string: "https://zhgchg.li/assets/9da2c51fa4f2/1*fBbNbDepYioQ-3-0XUkF6Q.jpeg")!, options: [:])
do {
try handler.perform([request])
}
catch {
print("Request failed: \(error)")
}
}
}分析結果：
LE LABO 青山店
TEL:03-6419-7167
＊お買い上げありがとうございます*
No: 21347
日付：2023/06/10 14.14.57
担当：
1690370
レジ：008A 1
商品名
税込上代数量税込合計
カイアック 10 EDP FB 15ML
J1P7010000S
16,800
16,800
アナザー 13 EDP FB 15ML
J1PJ010000S
10,700
10,700
リップパーム 15ML
JOWC010000S
2,000
1
合計金額
（内税額）
CARD
2,000
3点御買上げ
29,500
0
29,500
29,500DetectBarcodesRequest
偵測圖片中的條碼、QRCode 數據。
let filePath = Bundle.main.path(forResource: "IMG_6777", ofType: "png")! // 本地測試圖片
let fileURL = URL(filePath: filePath)
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
let request = DetectBarcodesRequest()
Task {
do {
let observations = try await request.perform(on: fileURL)
observations.forEach {
observation in
print("Payload: \(observation.payloadString ?? "No payload")")
print("Symbology: \(observation.symbology)")
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let completionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNBarcodeObservation] else {
return
}
observations.forEach {
observation in
print("Payload: \(observation.payloadStringValue ?? "No payload")")
print("Symbology: \(observation.symbology.rawValue)")
}
}
let request = VNDetectBarcodesRequest(completionHandler: completionHandler)
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: fileURL, options: [:])
do {
try handler.perform([request])
}
catch {
print("Request failed: \(error)")
}
}
}分析結果：
Payload: 8859126000911
Symbology: VNBarcodeSymbologyEAN13
Payload: https://lin.ee/hGynbVM
Symbology: VNBarcodeSymbologyQR
Payload: http://www.hongthaipanich.com/
Symbology: VNBarcodeSymbologyQR
Payload: https://www.facebook.com/qr?id=100063856061714
Symbology: VNBarcodeSymbologyQRRecognizeAnimalsRequest
辨識圖片中的動物與置信度。
let filePath = Bundle.main.path(forResource: "IMG_5026", ofType: "png")! // 本地測試圖片
let fileURL = URL(filePath: filePath)
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
let request = RecognizeAnimalsRequest()
Task {
do {
let observations = try await request.perform(on: fileURL)
observations.forEach {
observation in
let labels = observation.labels
labels.forEach {
label in
print("Detected animal: \(label.identifier) with confidence: \(label.confidence)")
}
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let completionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNRecognizedObjectObservation] else {
return
}
observations.forEach {
observation in
let labels = observation.labels
labels.forEach {
label in
print("Detected animal: \(label.identifier) with confidence: \(label.confidence)")
}
}
}
let request = VNRecognizeAnimalsRequest(completionHandler: completionHandler)
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: fileURL, options: [:])
do {
try handler.perform([request])
}
catch {
print("Request failed: \(error)")
}
}
}分析結果：
Detected animal: Cat with confidence: 0.77245045其他：
- 偵測圖像中的人體：DetectHumanRectanglesRequest
- 偵測人、動物的 Pose 動作 (3D or 2D 都可以)：DetectAnimalBodyPoseRequest、DetectHumanBodyPose3DRequest、DetectHumanBodyPoseRequest、DetectHumanHandPoseRequest
- 檢測並追蹤物件的運動軌跡(在影片、動畫不同的偵中)：DetectTrajectoriesRequest、TrackObjectRequest、TrackRectangleRequest
iOS ≥ 18 Update Highlight:
VN*Request -> *Request (e.g. VNDetectBarcodesRequest -> DetectBarcodesRequest)
VN*Observation -> *Observation (e.g. VNRecognizedObjectObservation -> RecognizedObjectObservation)
VNRequestCompletionHandler -> async/await
VNImageRequestHandler.perform([VN*Request]) -> *Request.perform()WWDC Example
WWDC 官方影片以超市商品掃描器為例。
首先大多數的商品都有 Barcode 可供掃描
我們可以從 observation.boundingBox 取得 Barcode 所在位置，但不同於常見 UIView 座標系，BoundingBox 的相對位置起點是從左下角，值的範圍落在 0~1 之間。
let filePath = Bundle.main.path(forResource: "IMG_6785", ofType: "png")! // 本地測試圖片
let fileURL = URL(filePath: filePath)
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
var request = DetectBarcodesRequest()
request.symbologies = [.ean13] // 如果只要掃描 EAN13 Barcode，可直接指定，提升效能
Task {
do {
let observations = try await request.perform(on: fileURL)
if let observation = observations.first {
DispatchQueue.main.async {
self.infoLabel.text = observation.payloadString
// 標記顏色 Layer
let colorLayer = CALayer()
// iOS >=18 新的座標轉換 API toImageCoordinates
// 未經測試，實際可能還需要計算 ContentMode = AspectFit 的位移:
colorLayer.frame = observation.boundingBox.toImageCoordinates(self.baseImageView.frame.size, origin: .upperLeft)
colorLayer.backgroundColor = UIColor.red.withAlphaComponent(0.5).cgColor
self.baseImageView.layer.addSublayer(colorLayer)
}
print("BoundingBox: \(observation.boundingBox.cgRect)")
print("Payload: \(observation.payloadString ?? "No payload")")
print("Symbology: \(observation.symbology)")
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let completionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNBarcodeObservation] else {
return
}
if let observation = observations.first {
DispatchQueue.main.async {
self.infoLabel.text = observation.payloadStringValue
// 標記顏色 Layer
let colorLayer = CALayer()
colorLayer.frame = self.convertBoundingBox(observation.boundingBox, to: self.baseImageView)
colorLayer.backgroundColor = UIColor.red.withAlphaComponent(0.5).cgColor
self.baseImageView.layer.addSublayer(colorLayer)
}
print("BoundingBox: \(observation.boundingBox)")
print("Payload: \(observation.payloadStringValue ?? "No payload")")
print("Symbology: \(observation.symbology.rawValue)")
}
}
let request = VNDetectBarcodesRequest(completionHandler: completionHandler)
request.symbologies = [.ean13] // 如果只要掃描 EAN13 Barcode，可直接指定，提升效能
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: fileURL, options: [:])
do {
try handler.perform([request])
}
catch {
print("Request failed: \(error)")
}
}
}iOS ≥ 18 Update Highlight:
// iOS >=18 新的座標轉換 API toImageCoordinates
observation.boundingBox.toImageCoordinates(CGSize, origin: .upperLeft)
// https://developer.apple.com/documentation/vision/normalizedpoint/toimagecoordinates(from:imagesize:origin:)Helper:
// Gen by ChatGPT 4o
// 因為照片在 ImageView 是設定 ContentMode = AspectFit
// 所以要多計算上下因 Fit 造成的空白位移
func convertBoundingBox(_ boundingBox: CGRect, to view: UIImageView) -> CGRect {
guard let image = view.image else {
return .zero
}
let imageSize = image.size
let viewSize = view.bounds.size
let imageRatio = imageSize.width / imageSize.height
let viewRatio = viewSize.width / viewSize.height
var scaleFactor: CGFloat
var offsetX: CGFloat = 0
var offsetY: CGFloat = 0
if imageRatio > viewRatio {
// 圖像在寬度方向上適配
scaleFactor = viewSize.width / imageSize.width
offsetY = (viewSize.height - imageSize.height * scaleFactor) / 2
}
else {
// 圖像在高度方向上適配
scaleFactor = viewSize.height / imageSize.height
offsetX = (viewSize.width - imageSize.width * scaleFactor) / 2
}
let x = boundingBox.minX * imageSize.width * scaleFactor + offsetX
let y = (1 - boundingBox.maxY) * imageSize.height * scaleFactor + offsetY
let width = boundingBox.width * imageSize.width * scaleFactor
let height = boundingBox.height * imageSize.height * scaleFactor
return CGRect(x: x, y: y, width: width, height: height)
}輸出結果
BoundingBox: (0.5295758928571429, 0.21408638121589782, 0.0943080357142857, 0.21254415360708087)
Payload: 4710018183805
Symbology: VNBarcodeSymbologyEAN13部分商品無 Barcode，如散裝水果只有商品標籤
因此我們的掃瞄器也需要同時支援掃描純文字標籤。
let filePath = Bundle.main.path(forResource: "apple", ofType: "jpg")! // 本地測試圖片
let fileURL = URL(filePath: filePath)
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
var barcodesRequest = DetectBarcodesRequest()
barcodesRequest.symbologies = [.ean13] // 如果只要掃描 EAN13 Barcode，可直接指定，提升效能
var textRequest = RecognizeTextRequest()
textRequest.recognitionLanguages = [.init(identifier: "zh-Hnat"), .init(identifier: "en-US")]
Task {
do {
let handler = ImageRequestHandler(fileURL)
// parameter pack syntax and we must wait for all requests to finish before we can use their results.
// let (barcodesObservation, textObservation, ...) = try await handler.perform(barcodesRequest, textRequest, ...)
let (barcodesObservation, textObservation) = try await handler.perform(barcodesRequest, textRequest)
if let observation = barcodesObservation.first {
DispatchQueue.main.async {
self.infoLabel.text = observation.payloadString
// 標記顏色 Layer
let colorLayer = CALayer()
// iOS >=18 新的座標轉換 API toImageCoordinates
// 未經測試，實際可能還需要計算 ContentMode = AspectFit 的位移:
colorLayer.frame = observation.boundingBox.toImageCoordinates(self.baseImageView.frame.size, origin: .upperLeft)
colorLayer.backgroundColor = UIColor.red.withAlphaComponent(0.5).cgColor
self.baseImageView.layer.addSublayer(colorLayer)
}
print("BoundingBox: \(observation.boundingBox.cgRect)")
print("Payload: \(observation.payloadString ?? "No payload")")
print("Symbology: \(observation.symbology)")
}
textObservation.forEach {
observation in
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "No text recognized")
}
}
catch {
print("Request failed: \(error)")
}
}
} else {
// 舊的寫法
let barcodesCompletionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNBarcodeObservation] else {
return
}
if let observation = observations.first {
DispatchQueue.main.async {
self.infoLabel.text = observation.payloadStringValue
// 標記顏色 Layer
let colorLayer = CALayer()
colorLayer.frame = self.convertBoundingBox(observation.boundingBox, to: self.baseImageView)
colorLayer.backgroundColor = UIColor.red.withAlphaComponent(0.5).cgColor
self.baseImageView.layer.addSublayer(colorLayer)
}
print("BoundingBox: \(observation.boundingBox)")
print("Payload: \(observation.payloadStringValue ?? "No payload")")
print("Symbology: \(observation.symbology.rawValue)")
}
}
let textCompletionHandler: VNRequestCompletionHandler = {
request, error in
guard error == nil else {
print("Request failed: \(String(describing: error))")
return
}
guard let observations = request.results as? [VNRecognizedTextObservation] else {
return
}
observations.forEach {
observation in
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "No text recognized")
}
}
let barcodesRequest = VNDetectBarcodesRequest(completionHandler: barcodesCompletionHandler)
barcodesRequest.symbologies = [.ean13] // 如果只要掃描 EAN13 Barcode，可直接指定，提升效能
let textRequest = VNRecognizeTextRequest(completionHandler: textCompletionHandler)
textRequest.recognitionLevel = .accurate
textRequest.recognitionLanguages = ["en-US"]
DispatchQueue.global().async {
let handler = VNImageRequestHandler(url: fileURL, options: [:])
do {
try handler.perform([barcodesRequest, textRequest])
}
catch {
print("Request failed: \(error)")
}
}
}輸出結果：
94128s
ORGANIC
Pink Lady®
Produce of UShiOS ≥ 18 Update Highlight:
let handler = ImageRequestHandler(fileURL)
// parameter pack syntax and we must wait for all requests to finish before we can use their results.
// let (barcodesObservation, textObservation, ...) = try await handler.perform(barcodesRequest, textRequest, ...)
let (barcodesObservation, textObservation) = try await handler.perform(barcodesRequest, textRequest)iOS ≥ 18 performAll() 方法
前面的 perform(barcodesRequest, textRequest) 處理 Barcode 掃描跟文字掃描的方式需要等到兩個 Request 都完成才能繼續執行；iOS 18 開始提供新的 performAll() 方法，將回應方式改為串流，在收到其中一個 Reqeust 結果是就能做對應處理，例如掃描到 Barcode 就直接響應。
if #available(iOS 18.0, *) {
// 新的使用 Swift 特性的 API
var barcodesRequest = DetectBarcodesRequest()
barcodesRequest.symbologies = [.ean13] // 如果只要掃描 EAN13 Barcode，可直接指定，提升效能
var textRequest = RecognizeTextRequest()
textRequest.recognitionLanguages = [.init(identifier: "zh-Hnat"), .init(identifier: "en-US")]
Task {
let handler = ImageRequestHandler(fileURL)
let observation = handler.performAll([barcodesRequest, textRequest] as [any VisionRequest])
for try await result in observation {
switch result {
case .detectBarcodes(_, let barcodesObservation):
if let observation = barcodesObservation.first {
DispatchQueue.main.async {
self.infoLabel.text = observation.payloadString
// 標記顏色 Layer
let colorLayer = CALayer()
// iOS >=18 新的座標轉換 API toImageCoordinates
// 未經測試，實際可能還需要計算 ContentMode = AspectFit 的位移:
colorLayer.frame = observation.boundingBox.toImageCoordinates(self.baseImageView.frame.size, origin: .upperLeft)
colorLayer.backgroundColor = UIColor.red.withAlphaComponent(0.5).cgColor
self.baseImageView.layer.addSublayer(colorLayer)
}
print("BoundingBox: \(observation.boundingBox.cgRect)")
print("Payload: \(observation.payloadString ?? "No payload")")
print("Symbology: \(observation.symbology)")
}
case .recognizeText(_, let textObservation):
textObservation.forEach {
observation in
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "No text recognized")
}
default:
print("Unrecongnized result: \(result)")
}
}
}
}Optimize with Swift Concurrency
假設我們有一個圖片牆列表，每張圖片都需要自動裁切出物件主體；這時候可以善用 Swift Concurrency 增加載入效率。
原始寫法
func generateThumbnail(url: URL) async throws -> UIImage {
let request = GenerateAttentionBasedSaliencyImageRequest()
let saliencyObservation = try await request.perform(on: url)
return cropImage(url, to: saliencyObservation.salientObjects)
}
func generateAllThumbnails() async throws {
for image in images {
image.thumbnail = try await generateThumbnail(url: image.url)
}
}一次只執行一個，效率、效能緩慢。
優化 (1) — TaskGroup Concurrency
func generateAllThumbnails() async throws {
try await withThrowingDiscardingTaskGroup { taskGroup in
for image in images {
image.thumbnail = try await generateThumbnail(url: image.url)
}
}
}將每個 Task 都加入 TaskGroup Concurrency 執行。
問題：圖片辨識、截圖操作非常消耗記憶體性能，如果無節制狂加並行任務，可能造成使用者卡頓、OOM 閃退問題。
優化 (2) — TaskGroup Concurrency + 限制並行數量
func generateAllThumbnails() async throws {
try await withThrowingDiscardingTaskGroup {
taskGroup in
// 最多執行數量不得超過 5
let maxImageTasks = min(5, images.count)
// 先填充 5 個 Task
for index in 0..<maxImageTasks {
taskGroup.addTask {
image[index].thumbnail = try await generateThumbnail(url: image[index].url)
}
}
var nextIndex = maxImageTasks
for try await _ in taskGroup {
// taskGroup 裡 Task await 完成時...
// 檢查 Index 是否到尾部
if nextIndex < images.count {
let image = images[nextIndex]
// 繼續逐個填充 Task (將維持在最多 5 個)
taskGroup.addTask {
image.thumbnail = try await generateThumbnail(url: image.url)
}
nextIndex += 1
}
}
}
}Update an existing Vision app
- Vision 將在具備神經引擎的設備上移除對部分請求的 CPU 和 GPU 支持。在這些設備上，神經引擎是性能最好的選擇。
可以使用supportedComputeDevices()API 進行檢查 - 移除所有 VN 前綴
VNXXRequest,VNXXXObservation->Reqeust,Observation - 使用 async/await 取代原本的 VNRequestCompletionHandler
- 直接使用
*Request.perform()取代原本的VNImageRequestHandler.perform([VN*Request])
Wrap-up
- 為 Swift 語言特性新設計的 API
- 新的功能、方法都為 Swift Only, iOS ≥ 18 可用
- 新的圖片評分功能、身體＋手部動作追蹤
Thanks!
KKday 招募工商
👉👉👉本次讀書會分享源於 KKday App Team 組內每週技術分享活動，目前團隊也正在熱情招募 Senior iOS Engineer，有興趣的朋友歡迎投遞履歷。👈👈👈
參考資料
Discover Swift enhancements in the Vision framework
The Vision Framework API has been redesigned to leverage modern Swift features like concurrency, making it easier and faster to integrate a wide array of Vision algorithms into your app. We’ll tour the updated API and share sample code, along with best practices, to help you get the benefits of this framework with less coding effort. We’ll also demonstrate two new features: image aesthetics and holistic body pose.
Chapters
- 0:00 — Introduction
- 1:07 — New Vision API
- 1:47 — Get started with Vision
- 8:59 — Optimize with Swift Concurrency
- 11:05 — Update an existing Vision app
- 13:46 — What’s new in Vision?
Vision framework Apple Developer Documentation
-
有任何問題及指教歡迎與我聯絡。
