-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.xml
3314 lines (3288 loc) · 360 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
<channel>
<title>raikiriww's blog</title>
<link>https://blog.raikiriww.net/</link>
<description>Recent content on raikiriww's blog</description>
<generator>Hugo -- gohugo.io</generator>
<language>zh</language>
<lastBuildDate>Wed, 16 Oct 2024 20:10:39 +0800</lastBuildDate><atom:link href="https://blog.raikiriww.net/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Android BLE 长时间高频率扫描</title>
<link>https://blog.raikiriww.net/post/08c770aea2ea8fda/</link>
<pubDate>Wed, 16 Oct 2024 20:10:39 +0800</pubDate>
<guid>https://blog.raikiriww.net/post/08c770aea2ea8fda/</guid>
<description><p>最近在进行一些 Android app 蓝牙相关的开发，需要前台长时间高频率的扫描 BLE 设备，使用了 Android 提供的 <code>SCAN_MODE_LOW_LATENCY</code> 扫描模式，扫描 30 分钟后出现了扫描不到 BLE 设备的情况。</p></description>
<content:encoded><![CDATA[<p>最近在进行一些 Android app 蓝牙相关的开发，需要前台长时间高频率的扫描 BLE 设备，使用了 Android 提供的 <code>SCAN_MODE_LOW_LATENCY</code> 扫描模式，扫描 30 分钟后出现了扫描不到 BLE 设备的情况。</p>
<h2 id="问题">问题</h2>
<p>使用自己开发的 Android app ，蓝牙扫描代码为：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.BluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.ScanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.ScanSettings</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">BluetoothScanManager</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">BluetoothLeScanner</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">ScanCallback</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">ScanSettings</span><span class="w"> </span><span class="n">scanSettings</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="nf">BluetoothScanManager</span><span class="p">(</span><span class="n">BluetoothLeScanner</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">,</span><span class="w"> </span><span class="n">ScanCallback</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">bluetoothLeScanner</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">scanCallback</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">scanSettings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">Builder</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">setScanMode</span><span class="p">(</span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">SCAN_MODE_LOW_LATENCY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">setCallbackType</span><span class="p">(</span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">CALLBACK_TYPE_ALL_MATCHES</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">build</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">startScan</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">.</span><span class="na">startScan</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span><span class="w"> </span><span class="n">scanSettings</span><span class="p">,</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">stopScan</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">.</span><span class="na">stopScan</span><span class="p">(</span><span class="n">scanCallback</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>使用的测试手机的 Android 版本为 7.0 。程序开始扫描的前 30 分钟可以正常的高频扫描到 BLE 设备，30 分钟后一个 BLE 设备也扫描不到了。</p>
<h2 id="查找原因">查找原因</h2>
<p>根据搜索引擎检索到的相关信息查看 Android 蓝牙<a href="https://android.googlesource.com/platform/packages/apps/Bluetooth/+/android-7.0.0_r1/src/com/android/bluetooth/gatt/ScanManager.java#180">相关源码</a>，在 180 行找到了处理 BLE 扫描操作的类：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Handler class that handles BLE scan operations.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">class</span> <span class="nc">ClientHandler</span><span class="w"> </span><span class="kd">extends</span><span class="w"> </span><span class="n">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>其中用来接收消息的函数为：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="nd">@Override</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">handleMessage</span><span class="p">(</span><span class="n">Message</span><span class="w"> </span><span class="n">msg</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">ScanClient</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">ScanClient</span><span class="p">)</span><span class="w"> </span><span class="n">msg</span><span class="p">.</span><span class="na">obj</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">msg</span><span class="p">.</span><span class="na">what</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">MSG_START_BLE_SCAN</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">handleStartScan</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">MSG_STOP_BLE_SCAN</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">handleStopScan</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">MSG_FLUSH_BATCH_RESULTS</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">handleFlushBatchResults</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">MSG_SCAN_TIMEOUT</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mScanNative</span><span class="p">.</span><span class="na">regularScanTimeout</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">default</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Shouldn't happen.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">Log</span><span class="p">.</span><span class="na">e</span><span class="p">(</span><span class="n">TAG</span><span class="p">,</span><span class="w"> </span><span class="s">"received an unkown message : "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">msg</span><span class="p">.</span><span class="na">what</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>当收到开始 BLE 扫描的信号后会执行 <code>handleStartScan</code> 函数：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">handleStartScan</span><span class="p">(</span><span class="n">ScanClient</span><span class="w"> </span><span class="n">client</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">Utils</span><span class="p">.</span><span class="na">enforceAdminPermission</span><span class="p">(</span><span class="n">mService</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">logd</span><span class="p">(</span><span class="s">"handling starting scan"</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">isScanSupported</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">Log</span><span class="p">.</span><span class="na">e</span><span class="p">(</span><span class="n">TAG</span><span class="p">,</span><span class="w"> </span><span class="s">"Scan settings not supported"</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">return</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">mRegularScanClients</span><span class="p">.</span><span class="na">contains</span><span class="p">(</span><span class="n">client</span><span class="p">)</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">mBatchClients</span><span class="p">.</span><span class="na">contains</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">Log</span><span class="p">.</span><span class="na">e</span><span class="p">(</span><span class="n">TAG</span><span class="p">,</span><span class="w"> </span><span class="s">"Scan already started"</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">return</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Begin scan operations.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">isBatchClient</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mBatchClients</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mScanNative</span><span class="p">.</span><span class="na">startBatchScan</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mRegularScanClients</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mScanNative</span><span class="p">.</span><span class="na">startRegularScan</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">mScanNative</span><span class="p">.</span><span class="na">isOpportunisticScanClient</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mScanNative</span><span class="p">.</span><span class="na">configureRegularScanParams</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">mScanNative</span><span class="p">.</span><span class="na">isFirstMatchScanClient</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">Message</span><span class="w"> </span><span class="n">msg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mHandler</span><span class="p">.</span><span class="na">obtainMessage</span><span class="p">(</span><span class="n">MSG_SCAN_TIMEOUT</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">msg</span><span class="p">.</span><span class="na">obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">client</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Only one timeout message should exist at any time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mHandler</span><span class="p">.</span><span class="na">removeMessages</span><span class="p">(</span><span class="n">SCAN_TIMEOUT_MS</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mHandler</span><span class="p">.</span><span class="na">sendMessageDelayed</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span><span class="w"> </span><span class="n">SCAN_TIMEOUT_MS</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Update BatteryStats with this workload.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mBatteryStats</span><span class="p">.</span><span class="na">noteBleScanStarted</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="na">workSource</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">RemoteException</span><span class="w"> </span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="cm">/* ignore */</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>其中重要的是 // Begin scan operations 的 else 代码块中的第二个 if 代码块 ，首先会进行一个 <code>isFirstMatchScanClient</code> 函数的判断。函数在 531 行：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kt">boolean</span><span class="w"> </span><span class="nf">isFirstMatchScanClient</span><span class="p">(</span><span class="n">ScanClient</span><span class="w"> </span><span class="n">client</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="na">settings</span><span class="p">.</span><span class="na">getCallbackType</span><span class="p">()</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">CALLBACK_TYPE_FIRST_MATCH</span><span class="p">)</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>这个函数会将 client 的 CallbackType 和 <code>ScanSettings.CALLBACK_TYPE_FIRST_MATCH</code> 进行比对看是否一致，不一致的话会创建一个超时消息并使用 <code>sendMessageDelayed</code> 函数延迟 <code>SCAN_TIMEOUT_MS</code> 后发送。<code>SCAN_TIMEOUT_MS</code> 的值在 72 行：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// Maximum msec before scan gets downgraded to opportunistic</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">SCAN_TIMEOUT_MS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">30</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">60</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">1000</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>这个值就和遇到的 30 分钟后扫描不到 BLE 设备对上了。这个延迟 30 分钟后发送的消息会触发上面的 <code>handleMessage</code> 中的</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">MSG_SCAN_TIMEOUT</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">mScanNative</span><span class="p">.</span><span class="na">regularScanTimeout</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p><code>regularScanTimeout</code> 函数在 674 行：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">regularScanTimeout</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">ScanClient</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">mRegularScanClients</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">isOpportunisticScanClient</span><span class="p">(</span><span class="n">client</span><span class="p">)</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">isFirstMatchScanClient</span><span class="p">(</span><span class="n">client</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">logd</span><span class="p">(</span><span class="s">"clientIf set to scan opportunisticly: "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="na">clientIf</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">setOpportunisticScanClient</span><span class="p">(</span><span class="n">client</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="na">stats</span><span class="p">.</span><span class="na">setScanTimeout</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// The scan should continue for background scans</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">configureRegularScanParams</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">numRegularScanClients</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">logd</span><span class="p">(</span><span class="s">"stop scan"</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">gattClientScanNative</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">setOpportunisticScanClient</span><span class="p">(</span><span class="n">ScanClient</span><span class="w"> </span><span class="n">client</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// TODO: Add constructor to ScanSettings.Builder</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c1">// that can copy values from an existing ScanSettings object</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">Builder</span><span class="w"> </span><span class="n">builder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">Builder</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">ScanSettings</span><span class="w"> </span><span class="n">settings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="na">settings</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">setScanMode</span><span class="p">(</span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">SCAN_MODE_OPPORTUNISTIC</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">setCallbackType</span><span class="p">(</span><span class="n">settings</span><span class="p">.</span><span class="na">getCallbackType</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">setScanResultType</span><span class="p">(</span><span class="n">settings</span><span class="p">.</span><span class="na">getScanResultType</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">setReportDelay</span><span class="p">(</span><span class="n">settings</span><span class="p">.</span><span class="na">getReportDelayMillis</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">setNumOfMatches</span><span class="p">(</span><span class="n">settings</span><span class="p">.</span><span class="na">getNumOfMatches</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="na">settings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">builder</span><span class="p">.</span><span class="na">build</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>可以看到 <code>regularScanTimeout</code> 会把我们的蓝牙扫描模式设置为 <code>ScanSettings.SCAN_MODE_OPPORTUNISTIC </code>。Android developer 的<a href="https://developer.android.com/reference/android/bluetooth/le/ScanSettings#SCAN_MODE_OPPORTUNISTIC">文档</a>对这个扫描模式的描述为：</p>
<blockquote>
<p>A special Bluetooth LE scan mode. Applications using this scan mode will passively listen for other scan results without starting BLE scans themselves.</p>
</blockquote>
<p>根据文档，这种模式仅在其他应用已经启用了蓝牙扫描的情况下才会工作。也就是说，扫描不会主动进行，只会依赖其他应用的扫描行为。</p>
<p>至此，问题基本查清了。总结一下，当我们设置的 BLE 扫描的 <code>CallbackType</code> 不为 <code>ScanSettings.CALLBACK_TYPE_FIRST_MATCH</code> 时，BLE 的 ScanManager 会自动为我们 BLE 扫描的 client 增加一个 30 分钟后的超时消息发送，这个超时消息会改变我们的蓝牙扫描模式并把我们 BLE 扫描的 client 的状态设置为超时，导致我们搜索不到 BLE 设备。</p>
<h2 id="解决方式">解决方式</h2>
<p>由于这个 30 分钟的限制只针对我们 BLE 扫描的 client 。所以我们可以在开始扫描的同时创建一个定时任务，在一段时间（<30 分钟），比如 29 分钟后停止蓝牙扫描，等待一秒，然后再开始蓝牙扫描。大概代码为：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.BluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.ScanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.bluetooth.le.ScanSettings</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.os.Handler</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kn">import</span><span class="w"> </span><span class="nn">android.os.Looper</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">BluetoothScanManager</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">BluetoothLeScanner</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">ScanCallback</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">ScanSettings</span><span class="w"> </span><span class="n">scanSettings</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">Handler</span><span class="w"> </span><span class="n">handler</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">SCAN_DURATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">29</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">60</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">1000</span><span class="p">;</span><span class="w"> </span><span class="c1">// 29 minutes in milliseconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="nf">BluetoothScanManager</span><span class="p">(</span><span class="n">BluetoothLeScanner</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">,</span><span class="w"> </span><span class="n">ScanCallback</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">bluetoothLeScanner</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">scanCallback</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">handler</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">Handler</span><span class="p">(</span><span class="n">Looper</span><span class="p">.</span><span class="na">getMainLooper</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">BluetoothScanManager</span><span class="p">.</span><span class="na">scanSettings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">Builder</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">setScanMode</span><span class="p">(</span><span class="n">ScanSettings</span><span class="p">.</span><span class="na">SCAN_MODE_LOW_LATENCY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">build</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">startScan</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">.</span><span class="na">startScan</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span><span class="w"> </span><span class="n">scanSettings</span><span class="p">,</span><span class="w"> </span><span class="n">scanCallback</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">handler</span><span class="p">.</span><span class="na">postDelayed</span><span class="p">(()</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">stopScan</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">handler</span><span class="p">.</span><span class="na">postDelayed</span><span class="p">(</span><span class="n">BluetoothScanManager</span><span class="p">::</span><span class="n">startScan</span><span class="p">,</span><span class="w"> </span><span class="n">1000</span><span class="p">);</span><span class="w"> </span><span class="c1">// Wait 1 second before restarting scan</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="n">SCAN_DURATION</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">stopScan</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">bluetoothLeScanner</span><span class="p">.</span><span class="na">stopScan</span><span class="p">(</span><span class="n">scanCallback</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>修改后测试了两个小时，可以不间断的搜索到 BLE 设备。具体的 <code>SCAN_DURATION</code> 可能会根据不同厂商或不同安卓版本来修改。目前最新的 main 分支上的相关代码做了修改，但这个参数的值并没有变，详见 <a href="https://android.googlesource.com/platform/packages/apps/Bluetooth/+/refs/heads/main/src/com/android/bluetooth/btservice/AdapterService.java#3944">AdapterService.java</a> 。</p>
<h2 id="为什么会有这个限制">为什么会有这个限制</h2>
<p>AOSP code review system 上<a href="https://android-review.googlesource.com/c/platform/packages/apps/Bluetooth/+/215844">增加这个限制的 commit</a> 消息为：</p>
<blockquote>
<p>Add protection against LE scanning abuse</p>
<p>Added two checks to prevent abuse. The first check ensures that an app doesn’t scan too frequently in a certain time period. It is allowed to scan again after its oldest scan exceedes said time period. The second check ensures that an app doesn’t scan for too long. Upon starting a scan, this code waits a certain amount of time. If the app is still scanning by that point, this code stops the scan and forces the app to use opportunistic scanning instead.</p>
</blockquote>
<p>看来是为了防止 BLE 扫描的滥用。而且一开始的超时时间设置的为 5 分钟，后边的一次<a href="https://android-review.googlesource.com/c/platform/packages/apps/Bluetooth/+/231662">提交</a>才改为 30 分钟。但这些限制并没有在开发者文档中相关的地方标注出来。</p>
<h2 id="参考链接">参考链接</h2>
<ul>
<li><a href="https://juejin.cn/post/7046328465108402207">https://juejin.cn/post/7046328465108402207</a></li>
<li><a href="https://blog.classycode.com/undocumented-android-7-ble-behavior-changes-d1a9bd87d983">https://blog.classycode.com/undocumented-android-7-ble-behavior-changes-d1a9bd87d983</a></li>
</ul>]]></content:encoded>
</item>
<item>
<title>Windows 系统使用 Socat</title>
<link>https://blog.raikiriww.net/post/7f48681db0ca9a86/</link>
<pubDate>Sun, 03 Mar 2024 13:09:25 +0800</pubDate>
<guid>https://blog.raikiriww.net/post/7f48681db0ca9a86/</guid>
<description><p>最近在自己家中的迷你主机上部署了一些自己用的服务，使用 frp 做内网穿透将迷你主机的服务映射到公网供自己使用。frp 支持 SSH 的端口复用，但需要使用到 Socat 这个工具。Socat 在 linux 和 mac 环境下都可以使用命令直接下载使用，但 windows 没有现成的库，网上找到的都是别人自己编译，也不知道靠不靠谱，所以决定使用 Github action 全自动，透明的编译一版 windows 能直接用的 Socat 。目前已经做好了，项目地址：<a href="https://github.com/raikiriww/socat_windows">https://github.com/raikiriww/socat_windows</a> 。</p></description>
<content:encoded><![CDATA[<p>最近在自己家中的迷你主机上部署了一些自己用的服务，使用 frp 做内网穿透将迷你主机的服务映射到公网供自己使用。frp 支持 SSH 的端口复用，但需要使用到 Socat 这个工具。Socat 在 linux 和 mac 环境下都可以使用命令直接下载使用，但 windows 没有现成的库，网上找到的都是别人自己编译，也不知道靠不靠谱，所以决定使用 Github action 全自动，透明的编译一版 windows 能直接用的 Socat 。目前已经做好了，项目地址：<a href="https://github.com/raikiriww/socat_windows">https://github.com/raikiriww/socat_windows</a> 。</p>
<h2 id="项目介绍">项目介绍</h2>
<p>整个项目目前（2024-03-03）只有一个 readme 和 Github action 的执行文件。具体为：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span><span class="lnt">58
</span><span class="lnt">59
</span><span class="lnt">60
</span><span class="lnt">61
</span><span class="lnt">62
</span><span class="lnt">63
</span><span class="lnt">64
</span><span class="lnt">65
</span><span class="lnt">66
</span><span class="lnt">67
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Cygwin Build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Triggers the workflow on push or pull request events but only for the "main" branch</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"main"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">pull_request</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"main"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Allows you to run this workflow manually from the Actions tab</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">workflow_dispatch</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">build</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">permissions</span><span class="p">:</span><span class="w"> </span><span class="l">write-all</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">windows-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Install Cygwin</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> choco install cygwin
</span></span></span><span class="line"><span class="cl"><span class="sd"> choco install cyg-get
</span></span></span><span class="line"><span class="cl"><span class="sd"> cyg-get gcc-g++ gcc-core cygwin32-gcc-g++ cygwin32-gcc-core make gcc-fortran gcc-objc gcc-objc++ libkrb5-devel libkrb5_3 libreadline-devel libssl-devel libwrap-devel tcp_wrappers </span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Configure Environment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> echo "C:\tools\cygwin\bin" >> $GITHUB_PATH</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Download and Extract the archive</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> curl -O http://www.dest-unreach.org/socat/download/socat-1.8.0.0.tar.gz
</span></span></span><span class="line"><span class="cl"><span class="sd"> tar -xvzf socat-1.8.0.0.tar.gz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Execute Build Script in Cygwin</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> C:\tools\cygwin\bin\bash -lc "cd /cygdrive/d/a/socat_windows/socat_windows/socat-1.8.0.0 && ./configure && make && make install"
</span></span></span><span class="line"><span class="cl"><span class="sd"> </span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Copy Cygwin DLLs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygcrypto-3.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygwin1.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygssl-3.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygreadline7.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygwrap-0.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygncursesw-10.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Copy-Item "C:\tools\cygwin\bin\cygz.dll" -Destination "D:\a\socat_windows\socat_windows\socat-1.8.0.0"</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">shell</span><span class="p">:</span><span class="w"> </span><span class="l">pwsh</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Upload Artifacts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ success() }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/upload-artifact@v2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">socat-1.8.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">socat-1.8.0.0/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Archive production artifacts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ success() }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">7z a socat-1.8.0.0.zip D:\a\socat_windows\socat_windows\socat-1.8.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Upload Release</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">softprops/action-gh-release@v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ success() }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">tag_name</span><span class="p">:</span><span class="w"> </span><span class="m">1.8.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">files</span><span class="p">:</span><span class="w"> </span><span class="l">socat-1.8.0.0.zip</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>整个编译过程在 Github 的 windows runner 上执行。使用 Cygwin 作为编译工具。编译过程分为下面几步：</p>
<ol>
<li>初始化代码仓库。这步会将仓库的所有文件下载到 runner 中。</li>
<li>安装 Cygwin 和依赖的库。</li>
<li>将 Cygwin 的 bin 目录加到环境变量中。</li>
<li>下载 Socat 的源码。</li>
<li>使用 Cygwin 编译。</li>
<li>拷贝依赖的 dll 文件到编译好的程序目录。</li>
<li>打包编译好的程序作为 Artifact 。</li>
<li>将打包好的压缩包作为 Release 发布。</li>
</ol>
<h2 id="使用方法">使用方法</h2>
<ol>
<li>下载项目 Releases 页面的程序压缩包。解压到本地。</li>
<li>将解压好的文件夹路径添加到 windows 的 PATH 环境变量中。</li>
</ol>
<p>打开终端输入 socat 即可使用。编译好的程序在我自己的电脑上（windows11 22H2 22621.2506）运行良好。</p>]]></content:encoded>
</item>
<item>
<title>机器学习（3）非监督学习</title>
<link>https://blog.raikiriww.net/post/416ace227c5e27b8/</link>
<pubDate>Wed, 27 Dec 2023 21:40:51 +0800</pubDate>
<guid>https://blog.raikiriww.net/post/416ace227c5e27b8/</guid>
<description><p>非监督学习</p></description>
<content:encoded><![CDATA[<p>非监督学习</p>
<h1 id="聚类clustering">聚类（Clustering）</h1>
<p>聚类是一种非监督学习算法，可以自动寻找数据中的结构，将相似的数据自动归为一组。相比于分类算法，它不需要提供训练数据的真值 y 。聚类算法的应用有：</p>
<ul>
<li>合并同类新闻</li>
<li>市场划分</li>
<li>DNA 分析</li>
<li>天文数据分析</li>
</ul>
<h2 id="k均值聚类算法k-mean-algorithm">K均值聚类算法（K-mean algorithm）</h2>
<p>K 均值聚类算法是聚类算法的一种。它会初始化 K 个聚类中心（cluster centroids），然后对所有数据重复下面两步操作：</p>
<ul>
<li>求每个数据到所有聚类中心的距离，找到距离最小的那个，把这个数据分配给这个聚类中心</li>
<li>计算每个聚类中心的所有数据的均值，将这个均值指定为每个聚类中心新的位置。</li>
</ul>
<p>重复上面两步直至聚类中心的位置不再变化。</p>
<p>有特殊情况是有些聚类中心没有分配到数据。常见的做法是去除这个聚类中心，或者也可以重新初始化这个聚类中心，期待它重新计算后可以分配到数据。</p>
<h3 id="代价函数">代价函数</h3>
<p>第二步中我们会计算每个聚类中心与其被分配到的数据的距离，将这个距离平方，将所有距离的平方相加然后除以数据总数，我们就得到了 K 均值算法的代价函数。也被称为 Distortion 。上面一直重复的两步操作都在最小化这个代价函数。当代价停止减小时，我们就可以停止重复上面的两步并检查模型的拟合情况。</p>
<h3 id="初始值选择">初始值选择</h3>
<p>假设我们有 K 个聚类中心，一般的做法是从所有数据中随机选择 K 个数据，将它们作为 K 个聚类中心的初始值。但单次随机选择容易得到几个相同的聚类中心，达不到最好的效果。所以，我们会选择 50-1000 次 K 个初始值，然后对这么多个初始值分别应用聚类算法，但每个初始值只计算一次代价，并不重复上面提到的两步操作。然后，从所有这些代价中选择最小的那个作为我们真正的初始值来重复上面的两步。</p>
<h3 id="k-值选择">K 值选择</h3>
<p>选择一个合适的聚类中心的数量会让我们的 K 值聚类算法拟合的更好。但很多场景并没有一个明显的 K 值。一个方法是尝试多个不同的 K 值，画出 K 值和代价的曲线，找到拐点，也就是代价下降由快到慢的那个转折点的 K 值作为实际使用的 K 值。在有些情况下代价随着 K 值的增加是缓慢下降的，这样这个方法就无法确定 K 值了。</p>
<p>实际上我们训练 K聚类算法模型是为了后续的用途，所以，我们要根据后续实际用途的效果来权衡 K 值的选择。比如：我们训练一个 K 均值聚类算法模型来压缩图片，根据我们想要压缩后的图片是比较清晰还是比较小，我们可以需要不同的 K 值。</p>
<h1 id="异常检测anomaly-detection">异常检测（Anomaly detection）</h1>
<p>异常检测是不同于聚类的另一种算法，它用于检测异常的行为。比如：制造出的物品是否合格，用户在网站上的行为是否正常，是否被盗号。</p>
<p>实现异常检测算法的流程为：</p>
<ol>
<li>选择 \( n \) 个与异常行为有关的特征 \( x \)</li>
<li>在每个特征上计算平均数和方差</li>
<li>在新的样本上根据平均数和方差根据高斯分布计算每个特征的概率并相乘得到一个概率</li>
<li>将得到的概率与一个阈值比较，小于阈值会判定为异常行为</li>
</ol>
<p>总体来看，异常检测算法是在检测是否有一个或多个特征出现的概率很小，小就判定为异常。</p>
<h2 id="模型评估和优化">模型评估和优化</h2>
<p>当我们训练好一个异常检测算法模型后，我们怎么选择一个好的阈值呢？怎么去优化这个模型呢？这时候就需要进行模型的评估。虽然异常检测是非监督学习算法，但那是指训练的时候我们没有用到有标签的数据，在评估过程中我们可以找一些确认为异常的数据，将他们和部分正常数据混合成两部分：交叉验证集和测试集，这点和监督学习是一样的。我们可以在交叉验证集中评估我们的模型，根据结果调整这个阈值，然后将最终的模型在测试集上测试并得到我们模型最终的效果。</p>
<p>当我们用于评估的异常数据样本很少时，比如只有一两个，那我们可以部分成交叉验证集和测试集，而是将全部的异常样本和部分正常样本混合成交叉验证集，不要测试集。这也可以优化我们的模型，但有过拟合的风险。</p>
<h2 id="异常检测与监督学习">异常检测与监督学习</h2>
<p>之前提到，我们可以用一些有标签的数据来优化和评估我们的模型，那我们为什么不直接训练一个监督学习模型呢？🤔下面是异常检测和监督学习模式合适应用场景的比较：</p>
<p><strong>异常检测</strong>：</p>
<ul>
<li>当异常数据的量很少，正常数据的量很大时</li>
<li>当异常情况可能会与我们拥有的样本完全不同时</li>
</ul>
<p><strong>监督学习模型</strong>：</p>
<ul>
<li>当异常数据和正常数据的数量都很多时</li>
<li>当未来出来的异常情况都与我们的样本相似时</li>
</ul>
<p>造成这些不同适用条件的原因是：</p>
<ul>
<li>异常检测算法是计算数据是正常数据的概率，我们的模型在大量的正常数据样本上总结了经验，当新的数据与我们正常数据相差很多时，不管它是否出现在我们评估时用到的异常数据中，它都会被模型识别到出现的概率很小并判断为异常数据。比如诈骗，诈骗招数会与时俱进，我们的诈骗样本可能过几年就不适用了。</li>
<li>监督学习模型是既总结正常数据的经验，又总结异常数据的经验。所以，当我们的异常数据很少或我们的异常数据样本并没有包含大多数的异常情况时，监督学习模型的效果会很差。当我们有足够多的异常和正常数据时，它会表现的很好。比如垃圾邮件，垃圾邮件虽然有很多种，但基本都是相同的情况，要卖给你东西或者诱导你点击网站。</li>
</ul>
<h2 id="特征选择">特征选择</h2>
<p>在监督学习中，模型有数据的标签作为真值，它可以过滤掉没用的特征或调整调整的参数来适应特征。而在非监督学习中，我们没有真值给模型训练，所以，选择合适的特征尤为重要。</p>
<h3 id="高斯化">高斯化</h3>
<p>我们计算每个特征的概率时是根据高斯分布来计算的，加入我们原本特征的分布不是高斯分布，那我们计算的概率可能不会很好。所以，我们需要绘制每个特征的分布曲线，如果不是高斯分布，那就使用一些函数来将该特征转换为类似高斯分布的数据。比如使用 \( \log\left( x + C \right) \) ，取平方根，立方根等来将原来的数据转为新的数据。</p>
<h3 id="错误分析">错误分析</h3>
<p>当我们使用了交叉验证集评估了模型后，我们可以查看模型预测错的例子，找出模型为什么预测错了，尝试一些办法，比如增加新的特征来使模型可以正确预测这种数据。</p>
<h3 id="创建新的特征">创建新的特征</h3>
<p>有时，异常发生时，单个特征值会很大，但平常这个特征的值可能会和其他特征值一起变得很大，这时候就需要我们将几个特征结合起来组成新的特征。比如，正常服务器运行时 CPU 和网络 IO 都可能很大，当被黑客入侵时， CPU 负载可能会很大，网络 IO 不大。这时，我们就可以将 CPU 负载和网络 IO 结合起来组成新的特征：CPU负载/网络 IO。这样就可以区分这两种情况。</p>
<h1 id="推荐系统">推荐系统</h1>
<p>推荐系统在现实世界中的应用非常广泛，当我们访问例如淘宝，京东之类的网站，它们都会根据我们的喜好来推荐不同的商品。</p>
<h2 id="协同过滤collaborative-filtering"><strong>协同过滤</strong>（Collaborative Filtering）</h2>
<p>协同过滤算法的目的是生成两个向量，假设我们要针对电影来做推荐系统。那么我们生成的两个向量为：</p>
<ol>
<li>对于每个用户，生成能够表现这个用户电影品味（喜欢程度）的向量</li>
<li>对于每个电影，生成能够代表这个电影特征的向量</li>
</ol>
<p>这两个向量的点乘加上一个偏差值就代表我们的算法预测该用户会对该电影打多少分。</p>
<h3 id="代价函数-1">代价函数</h3>
<p><code>$$J({\mathbf{x}^{(0)},...,\mathbf{x}^{(n_m-1)},\mathbf{w}^{(0)},b^{(0)},...,\mathbf{w}^{(n_u-1)},b^{(n_u-1)}})= \left[ \frac{1}{2}\sum_{(i,j):r(i,j)=1}(\mathbf{w}^{(j)} \cdot \mathbf{x}^{(i)} + b^{(j)} - y^{(i,j)})^2 \right] + \underbrace{\left[\frac{\lambda}{2}\sum_{j=0}^{n_u-1}\sum_{k=0}^{n-1}(\mathbf{w}^{(j)}_k)^2 + \frac{\lambda}{2}\sum_{i=0}^{n_m-1}\sum_{k=0}^{n-1}(\mathbf{x}_k^{(i)})^2\right]}_{regularization}$$</code></p>
<p>其实协同过滤的代价函数和线性回归差不多，主要的区别是变量不仅是 \(w\) 和 \(b\)，还有 \(x\)。我们会同时训练 \(w\), \(x\), \(b\)</p>
<p>$$ w = w - \alpha \frac{\partial J(w,b,x)}{\partial w} $$</p>
<p>$$ b = b - \alpha \frac{\partial J(w,b,x)}{\partial b} $$</p>
<p>$$ x = x - \alpha \frac{\partial J(w,b,x)}{\partial x} $$</p>
<p>当我们的训练数据是二元的标签数据时，例如：喜欢/不喜欢。我们可以将 \(\mathbf{w}^{(j)} \cdot \mathbf{x}^{(i)} + b^{(j)}\) 通过 sigmod 函数处理，然后代价函数就变成了类似逻辑回归的代价函数。</p>
<h3 id="均值归一化mean-normalization">均值归一化（Mean normalization）</h3>
<p>当我们训练好协同过滤模型后，如果有一个新用户，他没有评价过任何一部电影，当我们初始的 \(b\) 为 0 时，那么我们的模型将会预测它会给所有电影打 0 分，这显然是不对的。我们可以用均值归一化来解决这个问题。</p>
<p>还是以电影评分为例。我们可以取每个电影的均分，然后将每个用户的评分减去这个均分。当我们计算这个用户对某个电影的评分时，我们需要再加上这个均分，也就是从 \(\mathbf{w}^{(j)} \cdot \mathbf{x}^{(i)} + b^{(j)}\) 改为 \(\mathbf{w}^{(j)} \cdot \mathbf{x}^{(i)} + b^{(j)} + u\) 。 \(u\) 指的就是这个均分。这样，当模型预测一个从没有评价过电影的新用户对某个电影的评分时，这个评分会时这个电影的平均评分，比之前的 0 合理。</p>
<h3 id="tensorflow-实现">TensorFlow 实现</h3>
<p>我们需要使用 TensorFlow 的 Custom Training Loop 功能来实现我们的算法。Custom Training Loop 可以让我们子集自定义计算代价的公式，TensorFlow 会自动帮助我们根据公式求偏导数，这一特性叫做 Auto Diff 。示例代码为：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">optimizer</span> <span class="o">=</span> <span class="n">keras</span><span class="o">.</span><span class="n">optimizers</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="mf">1e-1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">iterations</span> <span class="o">=</span> <span class="mi">200</span>
</span></span><span class="line"><span class="cl"><span class="n">lambda_</span> <span class="o">=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">iterations</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># Use TensorFlow’s GradientTape</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># to record the operations used to compute the cost </span>
</span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">GradientTape</span><span class="p">()</span> <span class="k">as</span> <span class="n">tape</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="c1"># Compute the cost (forward pass included in cost)</span>
</span></span><span class="line"><span class="cl"> <span class="n">cost_value</span> <span class="o">=</span> <span class="n">cofi_cost_func_v</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">Ynorm</span><span class="p">,</span> <span class="n">R</span><span class="p">,</span> <span class="n">lambda_</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="c1"># Use the gradient tape to automatically retrieve</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># the gradients of the trainable variables with respect to the loss</span>
</span></span><span class="line"><span class="cl"> <span class="n">grads</span> <span class="o">=</span> <span class="n">tape</span><span class="o">.</span><span class="n">gradient</span><span class="p">(</span> <span class="n">cost_value</span><span class="p">,</span> <span class="p">[</span><span class="n">X</span><span class="p">,</span><span class="n">W</span><span class="p">,</span><span class="n">b</span><span class="p">]</span> <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="c1"># Run one step of gradient descent by updating</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># the value of the variables to minimize the loss.</span>
</span></span><span class="line"><span class="cl"> <span class="n">optimizer</span><span class="o">.</span><span class="n">apply_gradients</span><span class="p">(</span> <span class="nb">zip</span><span class="p">(</span><span class="n">grads</span><span class="p">,</span> <span class="p">[</span><span class="n">X</span><span class="p">,</span><span class="n">W</span><span class="p">,</span><span class="n">b</span><span class="p">])</span> <span class="p">)</span>
</span></span></code></pre></td></tr></table>
</div>
</div><h3 id="找到相关的物品">找到相关的物品</h3>
<p>当我们训练好协同过滤模型后，我们会得到每个物品的特征向量。当我们需要找到与某个物品相关的物品时，我们可以将所有的特征向量与该物品的的特征向量做差值并。找到差值最小的即可。</p>
<h3 id="局限性">局限性</h3>
<ul>
<li>冷启动问题：当有一个全新的，没有人评过分的物品时，协同过滤算法的效果不好。当用户评过分的物品较少时，协同过滤算法的效果也不好。</li>
<li>对附带信息的使用不是很好：例如用户的性别，年龄，位置。物品的类别，流派等。</li>
</ul>
<h2 id="内容过滤content-based-filtering">内容过滤（Content-based filtering）</h2>
<p>相比于协同过滤，我们有用户的特征（年龄，性别等）和物品的特征（流派，平均评分等）。我们想基于这些特征来给用户推荐合适的物品。我们要做的就是计算用户的特征向量 v_u 和物品的特征向量 v_m ，并将这两个向量点乘来判断该物品和用户是否合适。</p>
<h3 id="理论实现">理论实现</h3>
<p>我们可以构建两个神经网络：用户神经网络和物品神经网络来分别从用户和物品的特征中生成对应的特征向量。因为我们最终要点乘这两个神经网络的输出，所以这两个神经网络拥有相同结构的输出层，隐藏层可以不同。然后将这两个神经网络合并成一个来作为我们最终的神经网络。当我们想要的结果是评估用户是否想要这个物品时，我们可以将点乘的结果套上一个 sigmod 函数来使其输出 0/1 。</p>
<h3 id="性能优化">性能优化</h3>
<p>内容过滤推荐系统的运行方式可以分为两步：检索和排序</p>
<ol>
<li>
<p>检索：检索的功能是生成一个用户可能会喜欢的物品的列表。例如：</p>
<ul>
<li>查看用户最新浏览的十个物品，找到十个相似的物品加入列表</li>
<li>统计用户浏览过的物品，找到出现率最高的三个类别，将这些类别的 top 10 物品加入列表</li>
<li>将用户所属国家的 top 30 加入列表</li>
</ul>
<p>将上述所有步骤生成的列表整合到一个列表中，去除重复的和用户已经有的。</p>
</li>
<li>
<p>排序：将检索生成的列表中的所有物品和用户特征放入神经网络进行推理，得到用户对这些物品的可能喜爱程度并按照从大到小的顺序进行排序再展示给用户。</p>
</li>
</ol>
<p>这之中存在一个权衡，当我们检索功能输出的列表很大时，我们根据这个列表排序就会很慢。当我们降低检索功能输出的列表大小时，我们又担心检索范围太小，无法检索到用户真正喜欢的物品。所以，我们需要进行离线测试，尝试不同的检索功能输出的列表大小来找到一个满足我们需求的点。</p>
<h3 id="道德问题">道德问题</h3>
<p>构建推荐系统时，应考虑在道德方面的影响。尽量构建符合道德的，对社会和人类有益的系统。</p>
<h1 id="主成分分析principal-component-analysis">主成分分析（Principal Component Analysis）</h1>
<p>当我们有一个比较复杂的数据集，它拥有十几个或者成百上千个特征时，我们很难去可视化这样的数据集。主成分分析可以帮助我们将大量的特征转化为两个或三个特征来方便我们可视化数据。比如：它可以删除变化不大的特征，将某几个相关特征整合成一个等。这样，我们就可以通过可视化更好的理解数据。</p>
<h2 id="具体做法">具体做法</h2>
<p>假设我们有一组数据，x1 作为横轴， x2 作为纵轴。如果他们的取值范围相差很大，需要先进行特征缩放。接下来，PCA 算法会尝试找到一个新的轴，从每个点到这个新的轴作垂线，相交的点就看作该点在这个新的轴上的映射。然后计算所有点在这个轴上的方差，这时的方差可以看作保留原本信息的程度，越大越好。重复上述步骤，找到方差最大的那个轴作为原来的 x1 和 x2 的替代。比如一个有一个点（2， 3）。原来的 x1, x2 两个轴的（1，1）在新的轴上的坐标为（0.7， 0.7）那么这个点（2， 3）在新的轴上的坐标就是 2 * 0.7 + 3 * 0.7 = 3.5 。这样确定好第一个轴后，第二个轴就是跟第一个轴垂直的那条直线，第三个轴就是和第一，第二轴组成的平面垂直的那条直线。</p>
<h2 id="与线性回归的区别">与线性回归的区别</h2>
<p>看起来主成分分析的做法与线性回归差不多，都是拟合一条直线。但它们还是有区别的。</p>
<ul>
<li>线性回归计算的是每个点的 y 与拟合的直线的距离，这个距离是从该点做一条垂直于 x 轴的垂线与拟合的直线相交，取这条线段的距离。线性回归使用 y 的距离作为调整拟合的直线的依据。</li>
<li>主成分分析是从 x1, x2 这个点做一条垂直于拟合的轴的垂线，取这个线段作为距离。它是使用 x1, x2 两个特征作为调整轴的依据的。</li>
</ul>
<h2 id="反推原来点的坐标">反推原来点的坐标</h2>
<p>在我们使用新的轴替代了老的两个轴后，我们已经不可能通过新的轴上的点的坐标反推出旧的点的精确坐标的。但我们可以得到一个近似值，将新的坐标轴的点乘上原来坐标轴的（1，1）点在新的轴上所对应的坐标就行。比如 3.5 = （3.5 * 0.7， 3.5 * 0.7） = （2.45， 2.45）。相较于原来的（2，3）可以看作一个近似值。</p>
<h1 id="强化学习">强化学习</h1>
<p>当我们想要使用代码自动控制一个机器人或者飞行器时，我们有这个物体的状态（坐标，动力大小，方向等），想要训练一个监督学习模型来根据这些状态来自动决定下一步干什么是很困难的，因为正确的下一步动作是模棱两可的，你可以加油门，也可以调整倾角。这时，我们需要强化学习来解决这个事。</p>
<p>强化学习的关键是奖励函数（reward function） ，这个函数可以判断算法模型的输出是否是好的，满足我们期望的输出。如果是，那就奖励算法，如果不是，那就惩罚算法。通过这个奖励函数来让算法自己发现怎么做才能取得好的结果。强化学习与其他算法的区别就在这里，我们不会教算法怎么去取得好的结果，而是评判它每次做的好不好从而让算法自己找到方法去取得好的结果。有点像训练狗狗，当狗狗表现的好时，我们就奖励狗狗吃的或者夸它，当狗狗表现的不好时，我们就批评它。通过奖惩来训练它称为满足我们要求的好狗狗🐶。</p>
<p>为了实现这个奖励机制，我们需要计算模型每个动作能够获得的回报（return），这个回报可以为正数，零或者负数。每进行一个动作，我们会将该动作获得的奖励乘上一个折扣系数（discount factor）的 n 次方，n 为步数。比如：一开始的奖励为 0 ，我们第一个动作做完后，本应获得的奖励为 10 ，我们设定的折扣系数为 0.9 。那么实际获得的奖励就是 10 * 0.9 = 9 。接下来我们执行第二个动作，第二个动作本来给的奖励为 20 ，乘上折扣系数，实际的奖励为 20 * （0.9 ** 2） = 16.2 。假设第二个动作完成后我们达到了结束状态（terminal state）。那么，我们总共获得的奖励为 0 + 9 + 16.2 = 25.2 。</p>
<p>我们还需要一个策略（policy）\(\pi\) 来将状态与实际行动通过上面提到的奖励函数结合起来。我们给这个策略一个确定的状态，策略可以根据这个状态计算接下来的回报，比较不同的回报来输出一个行动，然后实际执行这个行动。\(action = \pi(state) \)</p>
<p>总结一下，我们上面提到的所有步骤整合起来就是强化学习的运作形式，这个形式有一个名字：<strong>马尔可夫决策过程（Markov Decision Process）</strong> ，简称 MDP 。马尔可夫指的是：未来的状态仅取决于现在的状态，与当前状态的之前的所有事都无关。</p>
<h2 id="状态价值函数state-action-value-functionq-function">状态价值函数（State-action value function）（Q-function）</h2>
<p>状态价值函数，简称 Q 函数，这个函数用于帮助我们计算当前状态 s 时，我们采取行动 a 能取得的回报是多少。通过在当前状态尝试多种不同的 a ，取其中回报最大的 a 来当作我们当前状态的 \( \pi(state)\) 所输出的 a 。</p>
<h2 id="贝尔曼方程bellman-equation">贝尔曼方程（Bellman Equation）</h2>
<p>$$ Q(s, a) = R(s) + \gamma \max_{a’} Q(s’, a’) $$</p>
<p>上面就是贝尔曼方程，它就是我们用来实现状态价值函数的方法。其中，\(R(s)\) 是当前状态的回报，\(\gamma\) 是折扣系数，\(s’\) 是我们执行完动作 a 后进入到的下一个状态，\(a’\) 是 \(s’\) 状态所能执行的动作。方程等号的右边可以分为两部分：</p>
<ul>
<li>\(R(s)\) 代表即时奖励</li>
<li>\( \gamma \max_{a’} Q(s’, a’)\) 代表后续获得的奖励</li>
</ul>
<h2 id="神经网络架构">神经网络架构</h2>
<p>我们可以使用神经网络来实现对函数 \(Q(s, a)\) 的模拟。神经网络的输入为当前的状态 s ，由多个参数构成，例如x, y, v_x, x_y 等。输出层的神经元个数等同于所有的动作个数。假设我们对于每个状态都有三个动作可以选择：向左，向右，什么也不做。那么我们要构建的神经网络的输出层的神经元个数就是 3 。这样可以一次推理就计算出所有可能动作的回报，使我们的计算更有效率。</p>
<p>完整的过程为：</p>
<ul>
<li>随机初始化神经网络的参数</li>
<li>通过实验收集各种各样的所需数据：当前状态 s，选择的动作 a，当前回报 R(s)，执行完动作所进入到的下一个状态 s’。存储最新的大概 10000 个数据</li>
<li>然后将每组数据的 s 作为 x ，\(R(s) + \gamma \max_{a’} Q(s’, a’)\) 作为 y 来训练神经网络，使用新的神经网络替换原有神经网络</li>
<li>重复收集数据和训练神经网络的步骤</li>
</ul>
<h2 id="在学习过程中选择合适的动作">在学习过程中选择合适的动作</h2>
<p>上面提到我们要收集训练数据来训练我们的神经网络，如果我们全部随机选择下一步动作，训练的效果不会很好，而我们又不知道最合适的动作是什么。我们该如何选择下一步执行的动作呢？常见的做法是：</p>
<ul>
<li>大概率（比如：95%）的情况下，选择能够最大化 \(Q(s’, a’)\) 的动作</li>
<li>小概率（比如：5%）的情况下，选择一个随机的动作</li>
</ul>
<p>这种方法叫做：\(\epsilon\)-greedy policy 。\(\epsilon\) 指的是选择随机动作的概率，在这里是 5%。还有一个常见的技巧是：开始时选择比较大的 \(\epsilon\) ，随着训练的进行减小 \(\epsilon\) 。</p>
<h2 id="优化">优化</h2>
<h3 id="小批量学习mini-batch">小批量学习（Mini-batch）</h3>
<p>当我们 的训练数据量很大时，每一次我们的梯度下降都会求所有训练数据的导数，这会导致我们的训练进行的很慢。为了加快训练，我们可以每次训练不使用全部的数据，而是使用其中一部分数据。比如我们总共有一亿个训练数据，每次训练时，我们都使用不同的 1000 组数据来训练。这样可以加快我们的训练速度。当然，相比于使用全部数据来训练，这样会使我们的梯度下降变得不是很稳定，偶尔会朝着最小梯度以外的方向下降，但总体上还是朝着正确的方向下降。</p>
<h3 id="软更新soft-update">软更新（Soft Update）</h3>
<p>之前提到的训练过程中会使用新训练好的神经网络来直接替换掉原来的神经网络，但有时我们新训练好的神经网络可能还不如原来的那个，这时替换的话会导致我们的拟合的更差。为了应对这种情况，我们可以采用软更新的方法：假设我们神经网络的参数是 W， B 那么，我们每次更新参数时不是直接使用 new_W ， new_B 替换 W， B 。而是 W = 0.99W + 0.01new_W, B = 0.99B + 0.01new_B 。0.99 和 0.01 是可调的两个参数，要求相加等于 1 。这样可以使我们的神经网络拟合的更好。</p>]]></content:encoded>
</item>
<item>
<title>机器学习（2） 神经网络, 决策树和优化</title>
<link>https://blog.raikiriww.net/post/4f14d32a25efe612/</link>
<pubDate>Wed, 06 Dec 2023 20:33:49 +0800</pubDate>
<guid>https://blog.raikiriww.net/post/4f14d32a25efe612/</guid>
<description><p>神经网络，决策树和优化</p></description>
<content:encoded><![CDATA[<p>神经网络，决策树和优化</p>
<h2 id="神经网络neural-network">神经网络（Neural network）</h2>
<p>神经网络是模仿人类的神经元来设计的，虽然我们对人类大脑的具体原理还不清楚，但模仿神经元设计的神经网络的功能是非常强大的。</p>
<h3 id="结构">结构</h3>
<p>神经网络的结构是层级的，每一层（Layer）包含一个或多个神经元（Neuron）。每一个神经元都有自己的参数（\(w\),\(b\) ）,从前一层获取值并经过自己的参数计算，传递给激活函数（Activation function），最终输出一个激活值。每层的所有神经元的激活值合并到一起就构成了这一层的输出。图例为两层的神经网络，第一层有三个神经元，第二层为输出层，有一个神经元。</p>
<center> <img src="/images/neural-network-model.png" width="200" /> </center>
<p>神经网络模型模型的输入会从左到右通过所有的层，得到一个输出，即神经网络的推断结果。这一过程也成为前向传播（Forward propagation）</p>
<h3 id="代码表示">代码表示</h3>
<p>使用目前比较流行的 Tensorflow 库来构建神经网络。代码示例：</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tensorflow.keras.models</span> <span class="kn">import</span> <span class="n">Sequential</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tensorflow.keras.activations</span> <span class="kn">import</span> <span class="n">linear</span><span class="p">,</span> <span class="n">relu</span><span class="p">,</span> <span class="n">sigmoid</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">X_train</span>
</span></span><span class="line"><span class="cl"><span class="n">y_train</span>
</span></span><span class="line"><span class="cl"><span class="c1"># 构建模型</span>
</span></span><span class="line"><span class="cl"><span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="n">Dense</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">'relu'</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">"L1"</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="n">Dense</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">'linear'</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">"L2"</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 指定损失函数和优化方式</span>
</span></span><span class="line"><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">loss</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">losses</span><span class="o">.</span><span class="n">BinaryCrossentropy</span><span class="p">(</span><span class="n">from_logits</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="n">optimizer</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">optimizers</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="mf">0.001</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 训练模型</span>
</span></span><span class="line"><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">epochs</span><span class="o">=</span><span class="mi">200</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 使用训练好的模型来预测</span>
</span></span><span class="line"><span class="cl"><span class="n">predictions</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_testn</span><span class="p">)</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>Sequential 用于指定神经网络模型的架构（有多少层，每层的类型），Dense 是一种层的类型，它可以指定该层的神经元个数，激活函数种类。compile 函数用于指定模型的代价函数和优化器类型。最后调用 fit 函数使用数据训练模型，epochs 用于指定训练数据训练多少次。Tensorflow 会在训练时将训练数据分为 batches 。每个 batch 大小为 32 。</p>
<h3 id="激活函数">激活函数</h3>
<p>目前有三种常用的激活函数：</p>
<ul>
<li><strong>Linear</strong>：activation = \( f(x) \)。可以输出正负都很大的值。</li>
<li><strong>sigmoid</strong>： activation = \( \frac{1}{1+e^{-f(x)}} \)。输出的值的范围在 0 ~ 1 之间。</li>
<li><strong>ReLU</strong>：activation = \(max(0,f(x)) \)。输出的值大于等于 0 。</li>
</ul>
<p><img loading="lazy" src="/images/activation_functions_plot.png" alt="激活函数对比图" />
</p>
<p>一般建议在所有的非输出层，即隐藏层使用 ReLU 激活函数，它相比于 sigmoid 函数计算更快。如果在所有层使用 Linear 激活函数（或者不使用激活函数），则整个神经网络就退化为一个线性回归模型了。</p>
<h3 id="softmax">Softmax</h3>
<p>softmax 是一种回归的类型，它可以从几个特定的种类中输出一个种类。类似逻辑回归，但逻辑回归只能输出 0 或 1， softmax 的结果可以是很多种类中的一个。借用吴恩达老师课程中的图片：</p>
<p><img loading="lazy" src="/images/softmax.png" alt="激活函数对比图" />
</p>
<p>用在神经网络中，最后的输出层输出的结果是一个向量，向量中的每个元素对应结果是该种类的概率值。该向量中值最大的元素的索引就是神经网络最终预测的种类。</p>
<p><strong>代价函数</strong></p>
<p>$$
\mathbf{1}\{y == n\} = =\begin{cases}
1, & \text{if $y==n$}.\\
0, & \text{otherwise}.
\end{cases}
$$</p>
<p>$$
J(\mathbf{w},b) = -\frac{1}{m} \left[ \sum_{i=1}^{m} \sum_{j=1}^{N} 1\left\{y^{(i)} == j\right\} \log\frac{e^{z^{(i)}_j}}{\sum_{k=1}^N e^{z^{(i)}_k} } \right]
$$</p>
<p><strong>代码表示</strong></p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="n">Dense</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">'relu'</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">"L1"</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="n">Dense</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">'linear'</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">"L2"</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">loss</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">losses</span><span class="o">.</span><span class="n">SparseCategoricalCrossentropy</span><span class="p">(</span><span class="n">from_logits</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="n">optimizer</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">optimizers</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="mf">0.01</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">epochs</span><span class="o">=</span><span class="mi">200</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>这样可以构建一个解决多个类别选一个问题的神经网络。最后一层使用 linear 激活函数而不是 softmax，在代价函数中加入 <code>from_logits=True</code> ，这样可以提高训练的精度。</p>
<h3 id="高级优化法">高级优化法</h3>
<p>相比于梯度下降，一个更高级的优化方法是 Adam 法，相较于梯度下降，它的学习率 \(\alpha\) 是不固定的，可以在训练过程中根据训练情况实时自动调整，避免学习率过大和过小导致的问题。使用方法很简单，上面的代码已经用到了 Adam 。只需在 compile 函数中指定 <code>optimizer=tf.keras.optimizers.Adam(0.01)</code> 即可。0.01为初始的学习率。</p>
<h3 id="其他类型的层">其他类型的层</h3>
<p>目前用到的层都是 Dense 层，它的每个神经元都会接受前一层的所有数据作为输入。还有一种叫做卷积（convolutional layer）的层，它的每个神经元接受前一层的特定位置的值作为输入。比如前一层输出了十个数据，这一层的第一个神经元只接收前三个，第二个神经元只接收第四到第六个作为输入。如果神经网络中由多个卷积层，这样的神经网络也叫卷积神经网络。组合多种不同的层可以构成不同的神经网络模型。</p>
<h3 id="反向传播">反向传播</h3>
<p>反向传播是神经网络训练过程的方式，由于神经网络由很多参数，很多层级，单纯的从输入开始计算偏导数到输出的计算量会很大。因为会涉及重复的运算。使用反向传播的方式，从输出开始向输入的方向计算每个参数的偏导数，使用链式法则将偏导数结合起来可以一次遍历计算完所有需要的导数。</p>
<h2 id="模型评估和优化">模型评估和优化</h2>
<p>当模型在实际数据上的预测表现不佳时，有很多可以考虑做的优化方法，例如：收集更多的数据，尝试减少特征的数量，尝试寻找新的特征，尝试增加多项式特征，增大正则化系数，减小正则化系数等。这时候就需要正确评估什么方法是有效的。</p>
<h3 id="模型选择">模型选择</h3>
<p>选择使用什么样的模型来解决问题是很重要的一步，需要从多种模型（多项式，神经网络）中挑选最合适的那个。所以，我们将数据集分为三部分：</p>
<ul>
<li>训练集（training set），占总数据集的 60%，用于训练模型</li>
<li>交叉验证集（cross validation set），占总数据集的 20%，用于评估各个模型的效果</li>
<li>测试集（test set），占总数据集的 20%，用于测试交叉验证集选出的最优模型的泛化程度</li>
</ul>
<p>流程为：先构建多个不同的模型，在训练集上将所有模型都训练好，然后计算所有模型在交叉验证集上的代价（cost），挑选出代价最小的那个模型，最后，计算挑选出的模型在测试集上的代价，用这个代价来评估模型的泛化程度。</p>
<h3 id="bias-and-variance-tradeoff">Bias and Variance tradeoff</h3>
<p>评估 Bias 和 Variance 需要一个基准，这个基准可以为：</p>
<ul>
<li>人类水平</li>
<li>竞争算法的水平</li>
<li>凭借经验的猜测</li>
</ul>
<p>当模型的表现不佳时，一般分为两种情况，High bias(欠拟合) 或 High variance(过拟合)。两者都有的情况很少见。</p>
<ul>
<li>High bias 表现为在训练集上的准确率和基准相差很大，在交叉验证集上也和基准相差很大。</li>
<li>High variance 表现为在训练集上的准确率和基准近似，在交叉验证集上和基准相差很大。</li>
</ul>
<p>根据这两种情况应该做不同的事情来优化模型：</p>
<p><strong>High bias:</strong></p>
<ul>
<li>增加多项式特征</li>
<li>尝试增加特征的数量</li>
<li>降低正则化参数大小</li>
</ul>
<p><strong>High variance:</strong></p>
<ul>
<li>增加正则化参数大小</li>
<li>尝试减少特征的数量</li>
<li>获取更多训练数据</li>
</ul>
<p>神经网络被认为是 low bias model 。所以经常要处理的是 high variance 问题。</p>
<h3 id="错误分析error-analysis">错误分析（Error analysis）</h3>
<p>错误分析（Error analysis）是指查看部分模型输出结果不对的数据，分析这些数据的类型。比如，有一个模型，它的输出结果是十个种类中的一个。查看100个模型输出错误的例子，其中，1个是类型1，20个是类型2，50个是类型3，等等。找到这之中占比比较大的结果来进行优化。在这个例子中就是类型3和类型2，根据这两个结果来优化模型会取得较大的效果。可以尝试多收集一些类型3和类型2的数据。或者调整模型参数（架构，正则化参数等等）。</p>
<h3 id="数据增强data-argumentation">数据增强（Data argumentation）</h3>
<p>数据增强是一种用来生成更多数据的方法。在输入的数据为图像时，可以对图像进行旋转，镜像，缩放，扭曲等操作来生成新的数据。如果输入是声音数据，可以对声音增加不同的背景噪音来生成新的数据。这些生成的新的数据可以让我们的模型训练的更好。</p>
<h3 id="转移学习transfer-learning">转移学习（Transfer learning）</h3>
<p>在神经网络中，一般会存在很多的层，当我们要训练的神经网络的输入和别人训练好的神经网络的输入相同时，比如：都是相同大小的图片。可以直接下载别人训练好的神经网络参数，把这些参数作为我们训练神经网络的初始参数。然后有两种做法：</p>
<ul>
<li>根据这些初始参数在我们的数据上重新训练所有的参数。</li>
<li>只根据我们的数据训练输出层的参数，其余参数不变。</li>
</ul>
<p>别人训练好参数的步骤成为监督式预训练（supervised pretraining），我们根据别人训练好的参数来训练我们的模型的过程成为微调（fine tuning）。这种方式可以让我们不需要很多的数据就可以训练出我们自己的模型，还可以减少我们的训练时间。</p>
<h3 id="机器学习项目的完整循环">机器学习项目的完整循环</h3>
<p>借用吴恩达老师课程的图片：</p>
<p><img loading="lazy" src="/images/Full_cycle_of_a_machine_learning_project.png" alt="机器学习项目的完整循环" />