-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
8763 lines (5867 loc) · 505 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[TsaiKoga Blog]]></title>
<link href="https://TsaiKoga.github.com/atom.xml" rel="self"/>
<link href="https://TsaiKoga.github.com/"/>
<updated>2020-03-15T23:06:52+08:00</updated>
<id>https://TsaiKoga.github.com/</id>
<author>
<name><![CDATA[TsaiKoga]]></name>
</author>
<generator uri="http://octopress.org/">Octopress</generator>
<entry>
<title type="html"><![CDATA[浏览历史数据库表设计与缓存设计]]></title>
<link href="https://TsaiKoga.github.com/blog/2020/03/15/liu-lan-li-shi-shu-ju-ku-biao-she-ji-yu-huan-cun-she-ji/"/>
<updated>2020-03-15T14:55:00+08:00</updated>
<id>https://TsaiKoga.github.com/blog/2020/03/15/liu-lan-li-shi-shu-ju-ku-biao-she-ji-yu-huan-cun-she-ji</id>
<content type="html"><![CDATA[<h3>目录</h3>
<h4><a href="#1">1. 功能需求</a></h4>
<h4><a href="#2">2. 实现方案</a></h4>
<h4><a href="#3">3. 数据库设计</a></h4>
<h5><a href="#3.1">3.1 预测数据量</a></h5>
<h5><a href="#3.2">3.2 按用户区间分表</a></h5>
<h5><a href="#3.3">3.3 定时任务数据归档</a></h5>
<h4><a href="#4">4. 缓存设计</a></h4>
<h5><a href="#4.1">4.1 怎么存</a></h5>
<h5><a href="#4.2">4.2 哪里存</a></h5>
<h5><a href="#4.3">4.3 缓存时间设置</a></h5>
<h1 id='1'>1 功能需求 </h1>
<p>最近,公司 app 需要增加一个新的功能是历史浏览记录的页面。</p>
<p>说道 “浏览历史”,大家应该都不陌生,跟浏览器的浏览历史功能基本一致。浏览历史的功能,仅限于用户浏览文章页面,
记录用户浏览过的文章,一方面方便用户下次再次查找,另一方面,也对我们将来对用户画像设计有所帮助。</p>
<p><img src="https://TsaiKoga.github.com/images/posts/2020-03-15/history.png" title="历史记录" alt="历史记录图片" /></p>
<br/>
<h1 id='2'>2 实现方案</h1>
<p>为了记录用户浏览过的文章,我们只需要在用户访问文章详情的页面进行记录即可。</p>
<p>可以采用“异步方式”,将用户浏览过的文章记录。</p>
<p>但是目前用户上百万,每个用户一天浏览量不很清楚,但是随时间递增,这个记录一定很大;所以我们采用新建一个 RDS 实例,
专门用来存放历史记录。</p>
<p>面对庞大的数据量我们应该如何进行分表呢?</p>
<br/>
<h1 id='3'>3 数据库分表</h1>
<p>分表前,我们先思考影响这个表的有哪些变量?</p>
<p>一般都离不开 “时间” 和 “用户数”。
所以我们到底是要采用按“时间分表”还是“用户分表”呢?</p>
<br/>
<h2 id='3.1'>3.1 预测数据量</h2>
<p>我们可以到服务器上看文章详情请求数。</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>cat 20200214.log | grep -E <span class="s2">"articles/[0-9]+"</span> | wc -l
</span></code></pre></td></tr></table></div></figure>
<p>将一天文件中的文章详情数统计出来,也可以通过脚本获取一个月的文件进行统计。</p>
<p>然后我们可以根据我们的分表方式来对数据进行预测:</p>
<p>1) <strong>通过时间-按月分表</strong></p>
<p>这时候我们需要统计一个月的请求数,大致推测一个月数据 600w 左右的浏览量。</p>
<p>2) <strong>通过用户-按用户id分表</strong></p>
<p>这个时候统计的时候要考虑 user_id 的值,通过脚本给了个平均预测,一个用户一个月 50*30 = 1500。
(较大较理想情况,因为我们就是要防止一些大用户的操作)</p>
<br/>
<p>根据数据和预测情况,我们可以简单看出其中的<strong>优缺点</strong>:</p>
<p>1) <strong>按月分表</strong></p>
<p><strong>优点</strong>:各个表数据量分布均匀</p>
<p><strong>缺点</strong>:</p>
<ol>
<li><p>由于需求要展示最近三个月的数据,那么读取数据的时候比较麻烦,需要将多个月表的数据 <code>union</code>
后,进行排序和分页,<code>union</code> 后的数据不走索引,况且分页效率有点低。</p></li>
<li><p>如果用户突然增长,月表数据可能突破限制。这时候只能再通过月表再次分表。</p></li>
</ol>
<p>2)<strong>按用户分表</strong></p>
<p><strong>优点</strong>:因为查询只针对个别用户可以充分利用索引,查询将十分简单高效,用户之间独立。</p>
<p><strong>缺点</strong>:</p>
<ol>
<li><p>由于每个用户属性不一样,阅读数不等,造成数据量不平均。</p></li>
<li><p>用户数量级庞大,分表数量庞大</p></li>
</ol>
<p>我们可以通过这两个优缺点进行取长补短,查询问题必须选用 “按用户分表”,那么如何解决用户表数量过多问题?</p>
<p>一般会有 <strong>按用户id取模分表</strong>,但是这种分表有缺点:</p>
<ol>
<li><p>数量再次增大,想要再进行扩展分表比较困难;</p></li>
<li><p>取模运算如果是通过 mysql,取模将无法走索引;</p></li>
</ol>
<br/>
<h2 id='3.2'>3.2 按用户区间分表</h2>
<p>所以我们可以选择 <strong>按用户区间分表</strong>,它呢补了上面取模的缺点,对后期单表中数据增大可以很方便继续再分;
例如我们 1w 个用户一张表,如果将来数据上亿,可以考虑变成 5k 个用户一张表,也就是在原来的基础上分成两张表,
通过脚本即可创建并迁移数据;</p>
<p>那么我们可以通过前面获得的访问量来确定大概要多少用户为一个区间;那么剩下一个问题了,当时间不短推移,
数据量还是很大,难道我们还要对再对用户分表吗?</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="nv">$suffix</span> <span class="o">=</span> <span class="nb">strval</span><span class="p">(</span><span class="nx">floor</span><span class="p">(</span><span class="nv">$user_id</span> <span class="o">/</span> <span class="nx">History</span><span class="o">::</span><span class="na">SPLIT_NUM</span><span class="p">));</span>
</span><span class='line'><span class="nv">$tablename</span> <span class="o">=</span> <span class="s1">'history'</span><span class="o">.</span><span class="nv">$suffix</span><span class="p">;</span>
</span><span class='line'><span class="nv">$this</span><span class="o">-></span><span class="na">setTable</span><span class="p">(</span><span class="nv">$tablename</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>
<br/>
<h2 id='3.3'>3.3 定时任务数据归档</h2>
<p>不,因为只展示用户最近三个月数据,所以我们可以通过定时任务<strong>对旧数据进行归档处理</strong>。
这样我们的表就一直控制在那个数据级;唯一可能上升的情况就是平均用户浏览数增多,不过影响不大,前面已经说了,
再次对用户进行划分也很简单。</p>
<p>注意,由于数据一直在这张表,所以主键就不要再用自增 id 了,因为日积月累的数据可能撑爆最大整型。可以用 <strong>UUID</strong> 或其他字符串类型等;</p>
<br/>
<hr />
<p>就此存储方案就好了,剩下显示方案;</p>
<p>关于显示,我们可以在表增加查询需要的所以,直接从中获取到我们的数据,速度很快;但是还有一个问题,就是
文章查询问题。</p>
<p>由于每篇文章可能根据运营人员更改而变动,所以一开始我们保存在浏览历史表中的数据将是关联id,例如此处是文章id;</p>
<p>这时候一种方法就是通过 <code>join</code> 文章表进行显示,但是这样三个月浏览记录去 JOIN 文章,可想而知这是一个噩梦;</p>
<p>那么我们还有另一个办法,通过缓存减少对文章的查询,也舍弃 join 文章的方法。</p>
<br/>
<h1 id="4">4 缓存设计</h1>
<h2 id="4.1">4.1 怎么存</h2>
<p>如何设计缓存呢?和分表一样,我们也需要考虑几个变量。</p>
<p>这里我们要显示的是用户某一天查询的文章的分页,所以变量将是:</p>
<ul>
<li>用户id</li>
<li>日期</li>
<li>分页页码</li>
<li>每页数量</li>
</ul>
<p>如果用结果缓存,这个组合将非常巨大,并且 redis 存储不适合一次性存储超过 150kb 的数据;</p>
<p>所以我们可以通过 <strong>片段缓存</strong> 的方式来存储;</p>
<p>1)<strong>优点</strong>:多处地方都可以使用该片段缓存,缓存利用率高;</p>
<p>2)<strong>缺点</strong>:还是需要消耗一点点性能;</p>
<p>每个文章进行缓存,这样从性能上说,每次只是在浏览历史数据库将数据分页排序后取出,</p>
<p>对一页中的数据进行循环,每次循环直接内存中读取缓存即可。</p>
<p>最坏情况,循环每页数量去数据库搜索每篇文章(这个问题可以通过缓存存储触发位置来优化)。</p>
<br/>
<h2 id="4.2">4.2 哪里存</h2>
<p>通过从产品上分析,用户是大多都是从文章列表点击文章详情进入,所以,我们可以在文章详情处就设置这个
片段缓存;当然,浏览历史的地方也要,只是浏览历史处命中率就变得非常高了,几乎已经先在文章详情处
进行了存储,所以循环取文章就变得非常少了;</p>
<br/>
<h2 id='4.3'>4.3 缓存时间设置</h2>
<p>TODO</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Laravel 的数据库事务源码分析]]></title>
<link href="https://TsaiKoga.github.com/blog/2019/05/09/laravel-de-shu-ju-ku-shi-wu-yuan-ma-fen-xi/"/>
<updated>2019-05-09T17:54:00+08:00</updated>
<id>https://TsaiKoga.github.com/blog/2019/05/09/laravel-de-shu-ju-ku-shi-wu-yuan-ma-fen-xi</id>
<content type="html"><![CDATA[<p>在批发档口记账 app 中,由于用户数量越来越多,有几个老客户了反馈单据丢失的问题;</p>
<p><strong>打个比方:</strong></p>
<p>我们开单会去创建开单表 <code>order_create</code>,开单表会生成详情 <code>order_create_goods_sku</code>,然后生成欠货表 <code>order_owe_goods_sku</code>;</p>
<p>然后接着是发货请求,会生成发货主表 <code>order_delivery</code> 和 发货详情 <code>order_delivery_goods_sku</code>;发货会生成库存变动记录;</p>
<br>
<p>这时候已经有了开单和开单详情,证明已经开单请求成功;</p>
<p>但是库存变动记录有,发货单却找不到;这时候我去查找了日志,发现也有发货请求;
然而通过库存变动记录找不到相应的发货单了,这是怎么回事,难道是被人撤销了吗?</p>
<br>
<p>全局查找了整个项目,没有强删除的接口,难道是代码有问题?</p>
<p>那么我开始定位代码,因为事务包裹着整个”发货的创建流程” 和 “库存变动记录的生成”;
所以要么就全部回滚,那么不太可能是回滚问题?</p>
<br>
<h1>跟踪数据库操作:</h1>
<hr />
<p>那是不是有人从数据库删除了?</p>
<p>我写了个触发器,只要有人强删记录就会记录到一张跟踪表:</p>
<p>1.首先创建跟踪表:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='sql'><span class='line'><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="o">`</span><span class="n">juniu_track_delivery</span><span class="o">`</span> <span class="p">(</span>
</span><span class='line'> <span class="o">`</span><span class="n">track_delivery_id</span><span class="o">`</span> <span class="nb">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="n">AUTO_INCREMENT</span> <span class="k">COMMENT</span> <span class="s1">'订单ID'</span><span class="p">,</span>
</span><span class='line'> <span class="o">`</span><span class="n">order_delivery_id</span><span class="o">`</span> <span class="nb">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="mi">0</span> <span class="k">COMMENT</span> <span class="s1">'发货单ID'</span><span class="p">,</span>
</span><span class='line'> <span class="o">`</span><span class="n">order_id</span><span class="o">`</span> <span class="nb">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="mi">0</span> <span class="k">COMMENT</span> <span class="s1">'订单ID'</span><span class="p">,</span>
</span><span class='line'> <span class="o">`</span><span class="n">deliver_timestamp</span><span class="o">`</span> <span class="k">timestamp</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="k">CURRENT_TIMESTAMP</span> <span class="k">COMMENT</span> <span class="s1">'发货时间'</span><span class="p">,</span>
</span><span class='line'> <span class="o">`</span><span class="n">deleted_at</span><span class="o">`</span> <span class="k">timestamp</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="k">CURRENT_TIMESTAMP</span> <span class="k">COMMENT</span> <span class="s1">'被删除时间'</span><span class="p">,</span>
</span><span class='line'> <span class="o">`</span><span class="n">deleted_by</span><span class="o">`</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="k">DEFAULT</span> <span class="s1">''</span> <span class="k">COMMENT</span> <span class="s1">'删除的用户ip'</span><span class="p">,</span>
</span><span class='line'> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="o">`</span><span class="n">track_delivery_id</span><span class="o">`</span><span class="p">)</span>
</span><span class='line'> <span class="p">)</span> <span class="n">ENGINE</span><span class="o">=</span><span class="n">InnoDB</span> <span class="n">AUTO_INCREMENT</span><span class="o">=</span><span class="mi">0</span> <span class="k">DEFAULT</span> <span class="n">CHARSET</span><span class="o">=</span><span class="n">utf8</span><span class="p">;</span>
</span></code></pre></td></tr></table></div></figure>
<br>
<p>2.建立触发器 <code>track_delete_delivery</code> 跟踪:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='sql'><span class='line'><span class="k">DELIMITER</span> <span class="o">//</span>
</span><span class='line'><span class="k">CREATE</span> <span class="k">TRIGGER</span> <span class="n">track_delete_delivery</span>
</span><span class='line'><span class="k">AFTER</span> <span class="k">DELETE</span>
</span><span class='line'> <span class="k">ON</span> <span class="n">juniu_order_delivery</span> <span class="k">FOR</span> <span class="k">EACH</span> <span class="k">ROW</span>
</span><span class='line'><span class="k">BEGIN</span>
</span><span class='line'> <span class="k">DECLARE</span> <span class="n">vUser</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span>
</span><span class='line'> <span class="c1">-- Find username of person performing the DELETE into table</span>
</span><span class='line'> <span class="k">SELECT</span> <span class="k">USER</span><span class="p">()</span> <span class="k">INTO</span> <span class="n">vUser</span><span class="p">;</span>
</span><span class='line'> <span class="c1">-- Insert record into audit table</span>
</span><span class='line'> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">juniu_track_delivery</span>
</span><span class='line'> <span class="p">(</span> <span class="n">order_delivery_id</span><span class="p">,</span>
</span><span class='line'> <span class="n">order_id</span><span class="p">,</span>
</span><span class='line'> <span class="n">deliver_timestamp</span><span class="p">,</span>
</span><span class='line'> <span class="n">deleted_at</span><span class="p">,</span>
</span><span class='line'> <span class="n">deleted_by</span><span class="p">)</span>
</span><span class='line'> <span class="k">VALUES</span>
</span><span class='line'> <span class="p">(</span> <span class="k">OLD</span><span class="p">.</span><span class="n">order_delivery_id</span><span class="p">,</span>
</span><span class='line'> <span class="k">OLD</span><span class="p">.</span><span class="n">order_id</span><span class="p">,</span>
</span><span class='line'> <span class="k">OLD</span><span class="p">.</span><span class="n">deliver_timestamp</span><span class="p">,</span>
</span><span class='line'> <span class="n">SYSDATE</span><span class="p">(),</span>
</span><span class='line'> <span class="n">vUser</span> <span class="p">);</span>
</span><span class='line'><span class="k">END</span><span class="p">;</span> <span class="o">//</span>
</span><span class='line'><span class="k">DELIMITER</span> <span class="p">;</span>
</span></code></pre></td></tr></table></div></figure>
<p>观察一段时间,仍有数据丢失,但是并没有跟踪到,初步判断不是人为删除;</p>
<br>
<h1>Mysql 事务特性:</h1>
<hr />
<p>Mysql 事务具有四大特性:A(原子性)C(一致性)I(隔离性)D(持久性);</p>
<p>事务有五个级别:</p>
<blockquote><ol>
<li><p>TRANSACTION_NONE 不使用事务。</p></li>
<li><p>TRANSACTION_READ_UNCOMMITTED 未提交读,允许脏读。</p></li>
<li><p>TRANSACTION_READ_COMMITTED 提交读,防止脏读,最常用的隔离级别,并且是大多数数据库的默认隔离级别</p></li>
<li><p>TRANSACTION_REPEATABLE_READ 重复读,可以防止脏读和不可重复读,</p></li>
<li><p>TRANSACTION_SERIALIZABLE 序列化,可以防止脏读,不可重复读取和幻读,(事务串行化)会降低数据库的效率</p></li>
</ol>
</blockquote>
<p>1、<strong>脏读:</strong> 事务A读取了事务B更新的数据,然后B回滚操作,那么A读取到的数据是脏数据</p>
<p>2、<strong>不可重复读:</strong> 事务 A 多次读取同一数据,事务 B 在事务A多次读取的过程中,对数据作了更新并提交,导致事务A多次读取同一数据时,结果不一致。</p>
<p>3、<strong>幻读:</strong> 事务 A 修改了表中所有数据,但是事务 B 插入了一条数据,当事务 A 查询数据发现还有一条记录没有改过来,就好像发生了幻觉一样,这就叫幻读。</p>
<p><strong>小结:</strong> 不可重复读的和幻读很容易混淆,不可重复读侧重于修改,幻读侧重于新增或删除。解决不可重复读的问题只需锁住满足条件的行,解决幻读需要锁表</p>
<br>
<p>最后我们来查看一下 Mysql 数据库事务配置,看是不是事务配置错误:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='sql'><span class='line'><span class="k">select</span> <span class="o">@@</span><span class="n">tx_isolation</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">select</span> <span class="o">@@</span><span class="k">session</span><span class="p">.</span><span class="n">tx_isolation</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">select</span> <span class="o">@@</span><span class="k">global</span><span class="p">.</span><span class="n">tx_isolation</span><span class="p">;</span>
</span></code></pre></td></tr></table></div></figure>
<p>显示结果都是 <code>READ-COMMITTED</code></p>
<br>
<h1>查看事务源码:</h1>
<hr />
<p>重新回到回滚代码,既然都被包裹起来,但是又回滚不成功,我开始查阅 Laravel API,因为我怕会是框架有问题,</p>
<p>找到相应的 laravel 版本 5.1: <a href="https://laravel.com/api/5.1/">https://laravel.com/api/5.1/</a></p>
<p>寻找 <code>transaction</code>,显示有</p>
<blockquote><p>Illuminate\Database\ConnectionInterface::transaction</p>
<p>Illuminate\Database\Connection::transaction</p></blockquote>
<p>接口定义的是规则,所以我看数据库的连接类 <code>Illuminate\Database\Connection::transaction</code></p>
<p>这里有几个我们要看的方法:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="nx">mixed</span> <span class="nx">transaction</span><span class="p">(</span><span class="nx">Closure</span> <span class="nv">$callback</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nx">void</span> <span class="nx">beginTransaction</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="nx">void</span> <span class="nx">commit</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="nx">void</span> <span class="nx">rollBack</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>
<p>点击右边行号,可以跳转到 5.1 源代码文件:</p>
<p>先查看 <code>transaction</code> 方法,此处参数是一个闭包,我们项目中使用的不是这种写法:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="k">public</span> <span class="k">function</span> <span class="nf">transaction</span><span class="p">(</span><span class="nx">Closure</span> <span class="nv">$callback</span><span class="p">)</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">beginTransaction</span><span class="p">();</span>
</span><span class='line'> <span class="c1">// We'll simply execute the given callback within a try / catch block</span>
</span><span class='line'> <span class="c1">// and if we catch any exception we can rollback the transaction</span>
</span><span class='line'> <span class="c1">// so that none of the changes are persisted to the database.</span>
</span><span class='line'> <span class="k">try</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$result</span> <span class="o">=</span> <span class="nv">$callback</span><span class="p">(</span><span class="nv">$this</span><span class="p">);</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">commit</span><span class="p">();</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="c1">// If we catch an exception, we will roll back so nothing gets messed</span>
</span><span class='line'> <span class="c1">// up in the database. Then we'll re-throw the exception so it can</span>
</span><span class='line'> <span class="c1">// be handled how the developer sees fit for their applications.</span>
</span><span class='line'> <span class="k">catch</span> <span class="p">(</span><span class="nx">Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">rollBack</span><span class="p">();</span>
</span><span class='line'> <span class="k">throw</span> <span class="nv">$e</span><span class="p">;</span>
</span><span class='line'> <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">Throwable</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">rollBack</span><span class="p">();</span>
</span><span class='line'> <span class="k">throw</span> <span class="nv">$e</span><span class="p">;</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="k">return</span> <span class="nv">$result</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>整个事务用 <code>try catch</code> 包裹住,如果失败,直接抛出异常,并且回滚,<code>transactions</code> 自然减为 0;</p>
<p>然后我看看我们项目中所使用的方法,通过<code>DB::beginTransaction();</code>
开始,然后中途出错,直接写 <code>DB::rollback()</code> ,
然后 <code>return</code> 返回,最后成功提交 <code>DB::commit()</code>。</p>
<p>先看看 <code>beginTransaction()</code> 的源码:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="k">public</span> <span class="k">function</span> <span class="nf">beginTransaction</span><span class="p">()</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="k">try</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">beginTransaction</span><span class="p">();</span>
</span><span class='line'> <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">causedByLostConnection</span><span class="p">(</span><span class="nv">$e</span><span class="p">))</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">reconnect</span><span class="p">();</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">beginTransaction</span><span class="p">();</span>
</span><span class='line'> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'> <span class="k">throw</span> <span class="nv">$e</span><span class="p">;</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="p">}</span> <span class="k">elseif</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">>=</span> <span class="mi">1</span> <span class="o">&&</span> <span class="nv">$this</span><span class="o">-></span><span class="na">queryGrammar</span><span class="o">-></span><span class="na">supportsSavepoints</span><span class="p">())</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">exec</span><span class="p">(</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">queryGrammar</span><span class="o">-></span><span class="na">compileSavepoint</span><span class="p">(</span><span class="s1">'trans'</span><span class="o">.</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span>
</span><span class='line'> <span class="p">);</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span><span class="o">++</span><span class="p">;</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">fireConnectionEvent</span><span class="p">(</span><span class="s1">'beganTransaction'</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>可以看到,这里的 <code>transactions</code> 属性记录多少层事务,通过 <code>try catch</code> 包裹一个事务开始,如果失败,重新尝试连接,并将<code>transaction + 1</code>;</p>
<br>
<p>再看 <code>rollback()</code></p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="k">public</span> <span class="k">function</span> <span class="nf">rollBack</span><span class="p">()</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">rollBack</span><span class="p">();</span>
</span><span class='line'> <span class="p">}</span> <span class="k">elseif</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">></span> <span class="mi">1</span> <span class="o">&&</span> <span class="nv">$this</span><span class="o">-></span><span class="na">queryGrammar</span><span class="o">-></span><span class="na">supportsSavepoints</span><span class="p">())</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">exec</span><span class="p">(</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">queryGrammar</span><span class="o">-></span><span class="na">compileSavepointRollBack</span><span class="p">(</span><span class="s1">'trans'</span><span class="o">.</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span><span class="p">)</span>
</span><span class='line'> <span class="p">);</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">=</span> <span class="nx">max</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">fireConnectionEvent</span><span class="p">(</span><span class="s1">'rollingBack'</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>这里 <strong>只有在 transactions 为 1 的时候,才去 rollback 整个事务</strong>;</p>
<br>
<p>再来看看 <code>commit()</code></p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="k">public</span> <span class="k">function</span> <span class="nf">commit</span><span class="p">()</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">pdo</span><span class="o">-></span><span class="na">commit</span><span class="p">();</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">transactions</span><span class="o">--</span><span class="p">;</span>
</span><span class='line'> <span class="nv">$this</span><span class="o">-></span><span class="na">fireConnectionEvent</span><span class="p">(</span><span class="s1">'committed'</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>也是只有当 <code>transactions == 1</code> 的时候才会 <code>commit</code> 整个事务;</p>
<p>不像 <code>transaction</code> 闭包那样,有整个 <code>try catch</code> 包裹,这里每一个步骤都要自己控制,</p>
<p>项目中事务的写法经常是:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="nx">DB</span><span class="o">::</span><span class="na">beginTransaction</span><span class="p">();</span>
</span><span class='line'>
</span><span class='line'><span class="nv">$res</span> <span class="o">=</span> <span class="nx">OrderDelivery</span><span class="o">::</span><span class="na">insert</span><span class="p">([</span><span class="o">...</span><span class="p">]);</span>
</span><span class='line'><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$res</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="nx">DB</span><span class="o">::</span><span class="na">rollback</span><span class="p">();</span>
</span><span class='line'> <span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">fail</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'创建失败'</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nv">$delivery_skus</span> <span class="o">=</span> <span class="nb">json_decode</span><span class="p">(</span><span class="nv">$color_size_matrix</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span>
</span><span class='line'><span class="nx">OrderDeliveryGoodsSku</span><span class="o">::</span><span class="na">insert</span><span class="p">(</span><span class="nv">$delivery_skus</span><span class="p">);</span>
</span><span class='line'><span class="o">...</span>
</span><span class='line'><span class="o">...</span>
</span><span class='line'><span class="nx">DB</span><span class="o">::</span><span class="na">commit</span><span class="p">();</span>
</span></code></pre></td></tr></table></div></figure>
<p>那么就有问题了,如果某个步骤出错,但是因为没有抛出异常,因为查看 API 像 <code>insert</code> 失败会返回 <code>false</code>;</p>
<p>需要人工判断返回值去 <code>rollback</code>,如果没有判断,也没有 <code>rollback</code>,这样 <code>transactions</code> 就没有减到 1;
这就有可能跑到 <code>commit</code> 那里去给 <code>transactions - 1</code> 了;</p>
<h1>结论:</h1>
<hr />
<p>所以较好的方法还是,</p>
<ol>
<li><code>transaction</code> 带闭包参数;</li>
<li><code>beginTransaction + try catch</code></li>
</ol>
<p>用 <code>beginTransaction + try catch</code>,只要有问题,直接到 <code>catch</code> 那做一次回滚即可,不用担心哪里忘了 <code>rollback</code>;</p>
<p>示例:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="nx">DB</span><span class="o">::</span><span class="na">beginTrasaction</span><span class="p">();</span>
</span><span class='line'><span class="k">try</span> <span class="p">{</span>
</span><span class='line'> <span class="nx">OrderModificationGoodsSku</span><span class="o">::</span><span class="na">insertGetId</span><span class="p">();</span>
</span><span class='line'> <span class="nx">OrderOweGoodsSku</span><span class="o">::</span><span class="na">where</span><span class="p">()</span><span class="o">-></span><span class="na">update</span><span class="p">();</span>
</span><span class='line'> <span class="nx">DB</span><span class="o">::</span><span class="na">commit</span><span class="p">();</span>
</span><span class='line'><span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">\Throwable</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'><span class="err"> </span> <span class="c1">// For php 7</span>
</span><span class='line'> <span class="nx">DB</span><span class="o">::</span><span class="na">rollback</span><span class="p">();</span>
</span><span class='line'> <span class="c1">//throw $e;</span>
</span><span class='line'> <span class="nx">Log</span><span class="o">::</span><span class="na">error</span><span class="p">(</span><span class="nv">$e</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
</span><span class='line'> <span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">fail</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="s2">"创建失败"</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">\Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="c1">// For php5</span>
</span><span class='line'> <span class="nx">DB</span><span class="o">::</span><span class="na">rollback</span><span class="p">();</span>
</span><span class='line'> <span class="c1">//throw $e;</span>
</span><span class='line'> <span class="nx">Log</span><span class="o">::</span><span class="na">error</span><span class="p">(</span><span class="nv">$e</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
</span><span class='line'> <span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">fail</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="s2">"创建失败"</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[宕机后的架构思考]]></title>
<link href="https://TsaiKoga.github.com/blog/2019/04/18/dang-ji-hou-de-jia-gou-si-kao/"/>
<updated>2019-04-18T10:24:00+08:00</updated>
<id>https://TsaiKoga.github.com/blog/2019/04/18/dang-ji-hou-de-jia-gou-si-kao</id>
<content type="html"><![CDATA[<h1>背景</h1>
<p>上个月公司服务器频繁宕机,第一次是因为日志撑爆了服务器,已经在[服务器日志空间解决方案-外挂磁盘]中解决了;</p>
<p>后面又有两次宕机,都是数据库 RDS 使用率太高导致的;</p>
<p>由于阿里云监控 RDS 报警不及时,所以比较难及时进行处理;</p>
<p>后来我使用了 processlist 查看:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='sql'><span class='line'><span class="k">select</span> <span class="n">id</span><span class="p">,</span> <span class="n">db</span><span class="p">,</span> <span class="k">user</span><span class="p">,</span> <span class="k">host</span><span class="p">,</span> <span class="n">command</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="k">state</span><span class="p">,</span> <span class="n">info</span>
</span><span class='line'><span class="k">from</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">processlist</span>
</span><span class='line'><span class="k">where</span> <span class="n">command</span> <span class="o">!=</span> <span class="s1">'Sleep'</span>
</span><span class='line'><span class="k">order</span> <span class="k">by</span> <span class="n">time</span> <span class="k">desc</span>
</span></code></pre></td></tr></table></div></figure>
<p>可以实时的观测到是哪些 sql 现在执行慢,当然这些 sql 可能本身不慢,而是因为被慢 sql 阻塞了,导致执行时间太久;</p>
<p>我们可以通过 kill 命令将进程号杀掉(需要用有权限的账号登录);</p>
<p>这些慢 sql 是 java 的新项目部署上去没有增加有效索引导致的,而新项目 和 已经稳定并且拥有大部分用户的 PHP 项目放在同一个 ECS,数据库也是同一台 RDS;
一个库占用了大部分 CPU 牵连另一个库执行 sql,造成部分 PHP 项目的用户开始投诉;</p>
<p>不过该问题已经得到解决,目前已经为不同项目拆分不同 ECS 和 不同的 RDS;</p>
<br>
<p>当前的 PHP 项目一些表的数据也都到达 2,3k万,现在 java 的项目有数据仓,有中台;
数据增长速度会比 PHP 项目快;</p>
<p>虽然已经稳定运行,但是这件事让我思考一个问题,如果以后用户级上去了,该怎么办?</p>
<br>
<h1>每秒请求数 qps</h1>
<p>前阵子看过一篇关于架构设计的文章,这里借用他的方法来设计为当前项目的架构做分析;</p>
<p>架构设计一般通过估量 qps,然后来设计架构;</p>
<p>首先我去查看项目的 qps,因为目前项目已经在线上,可以通过动态截取每秒日志请求数来估算 QPS:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>tail -f laravel-2019-04-01.log | cut -d <span class="s1">' '</span> f2 | uniq -c
</span></code></pre></td></tr></table></div></figure>
<p>得出结论,大概 qps 为 100 左右;</p>
<p>目前 100 个请求每秒对于 8 核的 ECS 已经足够;</p>
<p>如果按照每个请求一秒要处理 3 个 sql 请求,那么对于 RDS 就是 300 请求每秒,也是足够的;</p>
<p>所以,就目前位置,服务器 ECS 和 数据库 RDS 都能够承载当前用户量访问。</p>
<br>
<br>
<h1>集群化部署</h1>
<h2>系统集群化部署</h2>
<p>那么以后销售人员推广力度变大,qps 将逐渐变大,那么一台 ECS 可能承载不了;</p>
<p>这时候我们可以加多一台 ECS 来拆分 qps;</p>
<p>假设 qps 为 500,我们使用“负载均衡”策略,将 qps 平分到两台 ECS 服务器上:</p>
<br>
<h3>架构</h3>
<pre><code> |
500/s
|
↓
{ nginx 负载均衡 }
| |
250/s 250/s
| |
↓ ↓
{ ECS1 } { ECS2 }
| |
750/s 750/s
| |
+————————+
| |
↓ ↓
{ 数据库 RDS }
</code></pre>
<h3>简单实现:</h3>
<p>Nginx 通过 upstream 来给两台 ECS 分配权重,
以下是 tomcat + nginx 配置,PHP + nginx 也类似,只是换一下 tomcat_server 和 端口 即可</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'><span class="c"># 负载均衡到 2 台 ECS 上,一个 130 的权重为 1/3,129 的权重为 2/3;</span>
</span><span class='line'>upstream tomcat_server <span class="o">{</span>
</span><span class='line'> server 192.168.200.130:8080 <span class="nv">weight</span><span class="o">=</span>10
</span><span class='line'> server 192.168.200.129:8080 <span class="nv">weight</span><span class="o">=</span>20
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># 监听 80 端口,并转发到 tomcat_server 这个 upstream</span>
</span><span class='line'>server <span class="o">{</span>
</span><span class='line'> listen 80
</span><span class='line'> location / <span class="o">{</span>
</span><span class='line'> proxy_pass http://tomcat_server;
</span><span class='line'> root html;
</span><span class='line'> index index.html, index.htm;
</span><span class='line'> <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>
<br>
<br>
<h2>数据库分库分表 + 读写分离</h2>
<p>但是现在有一个问题,就是数据量庞大,例如开单表已经到达了 26,934,626 条,虽然都有效的利用了索引,但是明显开单已经变慢;</p>
<p>再加上每天店铺小妹都在同一个高峰时段不停开单,若有卡顿,则可能会拖累其他 sql 执行,并且造成 CPU 使用率上升,从而宕机;</p>
<p>所以面临的问题主要是两个:</p>
<p>1:数据量太大,导致查询缓慢:</p>
<p>2:面对较大的 qps,例如前面 500 qps 已经分成两个请求 750/s 到 RDS 上;</p>
<p>当 qps 不断增加,ECS 都可以用负载均衡增加机器解决,但是系统层面上解决了,数据库层面上并发量到达 3000/s 就有问题了;</p>
<br>
<h3>解决方案:</h3>
<ol>
<li><p>通过将经常查询并且数据量大的表进行 “水平分割”;</p></li>
<li><p>分库分表 + 读写分离;
也就是把一个库拆分为多个库,部署在多个数据库服务上,这是作为主库承载写入请求的;然后每个主库都挂载至少一个从库,由从库来承载读请求。
此时假设对数据库层面的读写并发是 3000/s,其中写并发占到了 1000/s,读并发占到了 2000/s。
那么一旦分库分表之后,采用两台数据库服务器上部署主库来支撑写请求,每台服务器承载的写并发就是 500/s。每台主库挂载一个服务器部署从库,那么 2 个从库每个从库支撑的读并发就是 1000/s。</p></li>
</ol>
<br>
<h3>架构</h3>
<pre><code> |
高峰 1000/s
↓
{ nginx 负载均衡 }
| | |
300/s 300/s 300/s
↓ ↓ ↓
+———→ { ECS1 } { ECS2 } { ECS3 } ←————+
| | | |
| 500/s 500/s |
1000/s ↓ ↓ 1000/s
| { 主库1 } { 主库2 } |
| ↓ ↓ |
+———— { 从库1 } { 从库2 } ——————+
</code></pre>
<h3>简单实现</h3>
<h4>主从数据库:</h4>
<p>如果数据库是 ECS 自己创建的,需要通过开启 my.cnf 的 Binary log 功能实现 ;</p>
<p>如果是直接购买 RDS,可以购买两台,一台做主库,一台做从库,通过阿里云自带服务 DTS 同步数据;</p>
<h4>分库分表:</h4>
<p>最好的做法,是服务在 <strong>搭建之初</strong> 就设计为分库分表的存储模式,从根本上杜绝中后期的风险。
不过,会牺牲一些便利性,例如列表式的查询,同时,也增加了维护的复杂度。</p>
<p><strong>如何分表:</strong>
像 Laravel,我们可以事先写一个 model,对应这个 model 的所有分表,然后为这个 model 添加一个方法;</p>
<p>这个方法 通过传入查询条件,对每个表进行查询,然后 union 起来返回;</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="cm">/* 传入查询的开始日期和结束日期用于计算跨越的表和达到约束表数据的目的。</span>
</span><span class='line'><span class="cm"> 外部可以调整查询的列,还可以添加where条件</span>
</span><span class='line'><span class="cm">*/</span>
</span><span class='line'><span class="k">public</span> <span class="k">function</span> <span class="nf">setUnionAllTable</span><span class="p">(</span><span class="nv">$startTime</span> <span class="o">=</span> <span class="nx">LARAVEL_START</span><span class="p">,</span> <span class="nv">$endTime</span> <span class="o">=</span> <span class="nx">LARAVEL_START</span><span class="p">,</span> <span class="nv">$attributes</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'*'</span><span class="p">],</span> <span class="nv">$wheres</span> <span class="o">=</span> <span class="p">[])</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="c1">//约束条件</span>
</span><span class='line'> <span class="nv">$whereConditions</span> <span class="o">=</span> <span class="p">[];</span>
</span><span class='line'> <span class="nv">$wheres</span> <span class="o">=</span> <span class="nb">array_merge</span><span class="p">([[</span><span class="s1">'time'</span><span class="p">,</span> <span class="s1">'>='</span><span class="p">,</span> <span class="nv">$startTime</span><span class="p">],</span> <span class="p">[</span><span class="s1">'time'</span><span class="p">,</span> <span class="s1">'<'</span><span class="p">,</span> <span class="nv">$endTime</span><span class="p">]],</span> <span class="nv">$wheres</span><span class="p">);</span>
</span><span class='line'> <span class="c1">//时间戳转日期</span>
</span><span class='line'> <span class="nv">$startDate</span> <span class="o">=</span> <span class="nb">date</span><span class="p">(</span><span class="s1">'Y-m'</span><span class="p">,</span> <span class="nv">$startTime</span><span class="p">);</span>
</span><span class='line'> <span class="nv">$endDate</span> <span class="o">=</span> <span class="nb">date</span><span class="p">(</span><span class="s1">'Y-m'</span><span class="p">,</span> <span class="nv">$endTime</span><span class="p">);</span>
</span><span class='line'> <span class="c1">//涉及的表数组</span>
</span><span class='line'> <span class="nv">$tables</span> <span class="o">=</span> <span class="p">[];</span>