• 검색 결과가 없습니다.

Result and Analysis of Performance Comparison on Residential

문서에서 저작자표시 (페이지 82-88)

Chapter 6. Unified Methods for Group and Individualized STLF 67

6.2 Experiment

6.2.2 Result and Analysis of Performance Comparison on Residential

We evaluated the performances of seven models including one-for-all learning using XGB, one-for-all learning using MLP with/without HPO, statistical method with Exp- MovAvg, Low4of5, and Avg5, transfer learning using MLP with/without HPO, meta learning using MLP with/without HPO on residential dataset in terms of ME in Table 6.2, RMSE in Table 6.3, and SMAPE in Table 6.4. Overall, unified methods generally improved performances.

The ME performances evaluated from August to September 2017 over twenty unified models are shown in Table 6.2. First of all, fourteen unified models without HPO are improved with both types of virtual control group adjustment. Additive type of virtual control group adjustment for the seven models improved ME performance by average 61.54%. ME performances of seven models with ratio type adjustment was improved by average 59.74%. Three unified models with HPO showed significant

ME improvements. Additive type and ratio type adjustments for three models showed average 52.76% improvement and 51.69% improvement, respectively. Overall, unified models with additive type adjustment were marginally more improved. The ME perfor- mance of the unified model that showed best ME performance is improved by 66.04%

compared to Low4of5 that is widely used in the industry.

If we assume that the virtual control group adjustment has the same effect on one-for-all learning, transfer learning, and meta learning, we can expect that the unified model of meta learning performed best because ME performance of the original meta learning model outperform other models. The unified model with ratio type of the adjustment, however, outperformed other six unified models in 6.2. We may well adjust one-for-all learning model of XGB using the virtual control group adjustment, or it may just be due to random chance. Although the result didn’t follow our expectation, overall rounded absolute ME performances of unified methods were less than 4 Wh that can be considered quite small amount.

The RMSE performances evaluated from August to September 2017 over twenty unified models are shown in Table 6.3. In all models, the unified approach was helpful.

Ratio type adjustment performed marginally better than additive type adjustment.

Unified models that used additive type adjustment and ratio type adjustment without HPO are improved by 2.88% and 2.74%, respectively. Unified models that used additive type adjustment and ratio type adjustment with HPO are also improved by 1.67% and 1.94%, respectively. The RMSE performance of the unified model that showed best RMSE performance is improved by 9.66% compared to Avg5.

The SMAPE performances evaluated from August to September 2017 over seven models are shown in Table 6.4. Unified methods only with ratio type adjustment was able to reduce SMAPE. The performances of additive type adjustment were consistently worse than not only the performances of ratio type but also the performance of original models. Overall, SMAPE performances confirmed that our assumption that the virtual control group adjustment has the same effect on one-for-all learning, transfer learning,

Table 6.2.ME performances of unified methods on residential dataset. Unified methods includes original and adjusted one-for-all learning, statistical, transfer learning, meta learning models with/without HPO. The unit of the error values is Wh.

One-for-all learning Individual learning

Statistical method Transfer learning Meta learning

XGB MLP ExpMovAvg Low4of5 Avg5 MLP MLP

original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted Without HPO Additive type

-9.4314 -2.4592

-7.1772 -3.0539

-8.0358 4.4815

50.7865 -5.7129

-11.6066 -4.9961

-8.7074 -4.1613

-6.3855 -2.7290

Ratio type -2.4752 -3.1145 5.2018 -6.7537 -4.9275 -4.2046 -2.7714

With HPO Additive type

-10.8456 -2.6902

-9.3044 -3.8262

-3.9928 -3.0262

Ratio type -2.7093 -3.8621 -3.1314

70

Table 6.3.RMSE performances of unified methods on residential dataset. Unified methods includes original and adjusted one-for-all learning, statistical, transfer learning, meta learning models with/without HPO. The unit of the error values is kWh.

One-for-all learning Individual learning

Statistical method Transfer learning Meta learning

XGB MLP ExpMovAvg Low4of5 Avg5 MLP MLP

original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted Without HPO Additive type

0.2612 0.2559

0.2614 0.2564

0.2862 0.2693

0.2818 0.2706

0.2806 0.2730

0.2594 0.2544

0.2605 0.2560

Ratio type 0.2551 0.2556 0.2708 0.2767 0.2713 0.2536 0.2555

With HPO Additive type

0.2610 0.2563

0.2591 0.2544

0.2586 0.2551

Ratio type 0.2555 0.2535 0.2545

71

Table 6.4.SMAPE performances of unified methods on residential dataset. Unified methods includes original and adjusted one-for-all learning, statistical, transfer learning, meta learning models with/without HPO. The unit of the error values is %.

One-for-all learning Individual learning

Statistical method Transfer learning Meta learning

XGB MLP ExpMovAvg Low4of5 Avg5 MLP MLP

original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted original adjusted Without HPO Additive type

32.04 31.62

32.23 31.88

35.85 34.31

31.89 32.99

33.77 33.96

31.59 31.42

32.07 31.73

Ratio type 31.43 31.71 33.82 32.72 33.22 31.16 31.60

With HPO Additive type

32.33 31.86

31.63 31.50

31.31 31.29

Ratio type 31.67 31.21 31.10

72

Figure 6.1. RMSE and SMAPE of the most representative algorithms with ratio type of virtual control group adjustment on residential dataset.

and meta learning is correct in Table 6.4. Unified models using additive type and ratio type adjustment without HPO are improved by 0.61% and 1.57%, respectively. Unified models using additive type and ratio type adjustment with HPO are improved by 0.65%

and 1.35%, respectively. The SMAPE performance of the unified model that showed best SMAPE performance is improved by 2.48% compared to Low4of5.

When we consider RMSE and SMAPE only, a visual summary is shown for the most representative algorithms in Fig. 6.1. As we mentioned above, transfer learning and meta learning performed best in RMSE and SMAPE, respectively. Before applying the virtual control group adjustment, meta learning performed best in both metrics. Through the virtual control group adjustment, however, meta learning model was relatively less improved than other models.

6.2.3. Result and Analysis of Performance Comparison on Non-residential

문서에서 저작자표시 (페이지 82-88)

관련 문서