{"id":2814,"date":"2021-05-11T00:38:38","date_gmt":"2021-05-11T00:38:38","guid":{"rendered":"https:\/\/www.tejwin.com\/en\/insights\/predict-the-market\/"},"modified":"2021-05-11T00:38:38","modified_gmt":"2021-05-11T00:38:38","slug":"predict-the-market","status":"publish","type":"insight","link":"https:\/\/www.tejwin.com\/en\/insights\/predict-the-market\/","title":{"rendered":"Predict The Market?!"},"content":{"rendered":"<figure class=\"wp-block-image size-large is-resized caption-align-center\"><img loading=\"lazy\" decoding=\"async\" alt=\"\" class=\"wp-image-15628\" height=\"558\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/image-228.png\" width=\"838\"\/><figcaption class=\"wp-element-caption\">photo by Unsplash<\/figcaption><\/figure>\n<h2 class=\"wp-block-heading\" id=\"1a4e\">Preface<\/h2>\n<p class=\"wp-block-paragraph\" id=\"7868\">Machine learning(ML), deep learning(DL), and artificial intelligence have been widely used by industries recently. Applications such as\u00a0<strong>image recognition and voice recognition<\/strong>\u00a0are common in our life.<\/p>\n<p class=\"wp-block-paragraph\" id=\"6b19\">However, wall street, the world center of finance has already recruited many statistics, mathematics even physics experts as algorithmic engineers and quantitative traders to exploit trading strategies. There had many cross-domain integrations in the finance field, and one of the most famous is the\u00a0<strong>option pricing model, Black-Scholes Model(BS model)<\/strong>. It uses the Brownian motion, which is the physic term to describe the random fluctuations in a particle\u2019s position inside a fluid domain to quantify the daily price movement of options. Although there had some flaw in the model\u2019s assumption, it still got the honor of\u00a0<strong>the Nobel prize in economics<\/strong>\u00a0at that time. In this case, we could know the contribution of the BS model to both academies and practices.<\/p>\n<p class=\"wp-block-paragraph\" id=\"4db7\">In this article,\u00a0<strong>we would guide you to the world of machine learning, and introduce how to use it to predict market movement~<\/strong><\/p>\n<h2 class=\"wp-block-heading\" id=\"f8c5\"><strong>Highlights of this article <\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li>\ud83c\udf1f Linear regression Intro\/application<\/li>\n<li>\ud83c\udf1f Logistic regression Intro\/application<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\" id=\"ba01\"><strong>Links related to this article<\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>1\ufe0f\u20e3 API Official Website:\u00a0<\/strong><a href=\"https:\/\/api.tej.com.tw\/\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ API Official Website<\/strong><\/a><\/li>\n<li><strong>2\ufe0f\u20e3 The Product Package:\u00a0<\/strong><a href=\"https:\/\/eshop.tej.com.tw\/E-Shop\/Edata_caseIntro\/1\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ E-SHOP<\/strong><\/a><\/li>\n<li><strong>3\ufe0f\u20e3 Source Code:\u00a0<\/strong><a href=\"https:\/\/github.com\/tejtw\/TEJAPI_Python_Medium_Quant\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ GITHUB<\/strong><\/a><\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\" id=\"f8d8\"><strong>Linear Regression<\/strong><\/h2>\n<p class=\"wp-block-paragraph\" id=\"3517\">Linear regression is the statistical method to find the pattern of the data and further draw a line to fit the data. While using linear regression, we only need to determine the<strong>\u00a0independent variables (X value)\u00a0<\/strong>and\u00a0<strong>dependent variables<\/strong>\u00a0<strong>(Y value).<\/strong>\u00a0More intuitively, for example,\u00a0<strong>if we want to use today\u2019s return to predict the future return<\/strong>, to meet this purpose,\u00a0<strong>we take the yesterday return as the independent variable(X), the tomorrow return as dependent variables (Y).<\/strong><\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1Q1kif-WmdnDNlgu6fi0JcA.png\"\/><figcaption class=\"wp-element-caption\">Linear regression formula<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\" id=\"0cb2\">We can find the best line to fit the data according to the formula above. The score measures by the average distance among points to the straight, called mean square error.\u00a0<strong>The linear regression aims to find the best combination of the coefficients (\u03b20, \u03b21) that minimize the mean square error.<\/strong><\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1_1O-GYtEopVvaYCsWefGo1ww.png\"\/><figcaption class=\"wp-element-caption\">Linear Regression \u2014 example<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"ab64\"><strong>Import packages <\/strong><\/h3>\n<pre class=\"wp-block-code\"><code>import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nplt.style.use('seaborn')\nplt.rcParams['font.sans-serif'] = ['Microsoft JhengHei']\nimport tejapi\ntejapi.ApiConfig.api_key = 'your key'\ntejapi.ApiConfig.ignoretz = True<\/code><\/pre>\n<h3 class=\"wp-block-heading\" id=\"2139\"><strong>Data extraction <\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"6a93\">Grabbing the daily stock price and return of MEDIATEK (2454), the data sourced from TEJ API. In this episode, we use the securities trading database and securities return database to get the close-adjusted price and daily return.<\/p>\n<pre class=\"wp-block-code\"><code>data = tejapi.get('TWN\/EWPRCD',\n                  coid='2454',\n                  paginate = True,\n                  chinese_column_name = True\n                 )\nreturn_ = tejapi.get('TWN\/EWPRCD2',\n                  coid='2454',\n                  paginate = True,\n                  chinese_column_name = True\n                 )\ndata['\u5831\u916c\u7387\uff05'] = return_['\u65e5\u5831\u916c\u7387(%)']<\/code><\/pre>\n<h3 class=\"wp-block-heading\" id=\"718c\"><strong>Test <\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"22cb\">Grabbing the daily stock price and return of\u00a0<strong>MEDIATEK (2454)<\/strong>,\u00a0<strong>the data sourced from TEJ API.<\/strong>\u00a0<strong>To get the close-adjusted price and daily return, we extract the data from the securities trading table and the securities return table. (<\/strong><a href=\"https:\/\/eshop.tej.com.tw\/E-Shop\/Edata_caseIntro\/2\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>Data source<\/strong><\/a><strong>)<\/strong><\/p>\n<p class=\"wp-block-paragraph\" id=\"2be8\">The figure below shows that stock prices grow along with time. In other words, the overall trend of the stock price is upward, and we can also say\u00a0<strong>stock price is non-stationary data.<\/strong>\u00a0So, if we use yesterday\u2019s stock price as X, today\u2019s stock price as Y.<\/p>\n<p class=\"wp-block-paragraph\" id=\"ccc3\">Even though the statistic result is significant, but just a coincidence. There doesn\u2019t have a strong relation between X and Y. This kind of circumstance so-called\u00a0<strong>spurious regression<\/strong>. (<a href=\"https:\/\/en.wikipedia.org\/wiki\/Spurious_relationship\" rel=\"noreferrer noopener\" target=\"_blank\">reference<\/a>)<\/p>\n<p class=\"wp-block-paragraph\" id=\"dbcf\">\u23f0 The solution to fix non-stationary data:<\/p>\n<ul class=\"wp-block-list\">\n<li>Make the first-order difference on the stock prices.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\" id=\"633c\">The first-order difference means to calculate the stock return.\u00a0<strong>After calculation, the trend of the data has been removed. Now, we can put the data into the linear regression model.<\/strong><\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1Kt0MG8Wqy9OZktdtv2JfIw.png\"\/><figcaption class=\"wp-element-caption\">stock price \u2014 non-stationary data<\/figcaption><\/figure>\n<figure class=\"wp-block-image is-resized caption-align-center\"><img loading=\"lazy\" decoding=\"async\" alt=\"\" height=\"421\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/10oYbqtSzSLJZHoZ1QK-uew.png\" width=\"841\"\/><figcaption class=\"wp-element-caption\">daily return\u00a0<strong>\u2014<\/strong>\u00a0stationary data<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"8065\"><strong>Linear regression analysis <\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"6fb9\">According to the graph below, we realize that it\u2019s hard to predict return by linear regression model.<\/p>\n<pre class=\"wp-block-code\"><code># \u7528\u6628\u5929(\u7b2ct-1\u65e5)\u7684\u5831\u916c\u7387\u53bb\u9810\u6e2c\u4eca\u5929(\u7b2ct\u65e5)\u7684\u5831\u916c\u7387\n# \u56e0\u524d\u5169\u500b\u4f4d\u7f6e\u7684\u503c\u70baNA\uff0c\u6545\u53d6\u7b2c3\u500b\u4f4d\u7f6e\u4ee5\u5f8c\u7684\u6578\u503c\nx = data['\u5831\u916c\u7387\uff05'].shift(1)[2:].values\ny = data['\u5831\u916c\u7387\uff05'][2:].values\nreg = np.polyfit(x,y,deg=1)\n# \u7e6a\u5716\nplt.figure(figsize=(10,6))\nplt.plot(x, y, 'bo', label='data')\nplt.plot(x, np.polyval(reg,x),'r',label='linear regression')\nplt.legend(loc=0)<\/code><\/pre>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1E0EwHUwdiu2avRtTB0ZARw.png\"\/><figcaption class=\"wp-element-caption\">yesterday\u2019s return vs today\u2019s return<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\" id=\"8fda\">The summary report shows that the T statistics is greater than 2, so yesterday\u2019s return(x1) has a significant effect on today\u2019s return, so we should include x1 into our model.<\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/19VgdkVzp9rHp0xfWEALmwg.png\"\/><figcaption class=\"wp-element-caption\">summary report<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"77bd\"><strong>Consider more lags<\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"1eeb\">We think<strong>\u00a0the information contains in x1 is not enough for us to well predicted the future return<\/strong>. Therefore, we adjust our model,\u00a0<strong>consider more lag into the model. Finally, we take lag 2, 3-period return into account.<\/strong><\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1KrU_0QsPabgbJp7apQxNYQ.png\"\/><figcaption class=\"wp-element-caption\">modified formula<\/figcaption><\/figure>\n<pre class=\"wp-block-code\"><code># \u843d\u5f8c\u4e00\u671f\u81f3\u4e09\u671f\u7684\u5831\u916c\u7387\ndata['R_t-1'] = data['\u5831\u916c\u7387\uff05'].shift(1)\ndata['R_t-2'] = data['\u5831\u916c\u7387\uff05'].shift(2)\ndata['R_t-3'] = data['\u5831\u916c\u7387\uff05'].shift(3)\nx = data[['R_t-1','R_t-2','R_t-3']][4:].values\ny = data['\u5831\u916c\u7387\uff05'][4:].values\nreg_t3 = np.linalg.lstsq(x,y, rcond=None)[0]\n# \u7e6a\u5716\nplt.figure(figsize=(10,6))\nplt.plot(y[1000:1300],label = 'true_value')\nplt.plot(np.dot(x, reg_t3)[1000:1300],color = 'green',label = 'predict')\nplt.legend()<\/code><\/pre>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/16hj5Xo2JV-JfSdFo9FlHEQ.png\"\/><figcaption class=\"wp-element-caption\">predict the result<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\" id=\"b369\"><strong>According to the summary report, the T statistics of x1 and x3 are higher than 2, so those two variables should include in our model.<\/strong><\/p>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1FE8AUAItMCJXgfjvzdGcbQ.png\"\/><figcaption class=\"wp-element-caption\">summary report<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"5475\"><strong>Summary <\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"89b4\">The part of linear regression has come to an end. Forecasting the market has always been a popular topic in finance and statistics. Interested friends can refer to relevant literature or explore other effective factors yourselves!!<\/p>\n<h2 class=\"wp-block-heading\" id=\"92f5\"><strong>Logistic regression<\/strong><\/h2>\n<p class=\"wp-block-paragraph\" id=\"1313\">Although logistic regression contains a regression in its name, it\u2019s a kind of<strong>\u00a0classification.<\/strong><\/p>\n<p class=\"wp-block-paragraph\" id=\"d8e0\">The difference between classification and regression their outputs. The classification outputs\u00a0<strong>a discrete variable<\/strong>(such as 0,1,2, etc.)\uff1bthe regression outputs\u00a0<strong>a continuous variable.<\/strong>\u00a0(such as 0,0.1,0.5,1.2, etc.)<\/p>\n<p class=\"wp-block-paragraph\" id=\"e560\">The concept of linear regression is to find a line whose average distance of points is minimization.<strong>\u00a0The logistic regression aims to find a line that can classify data accurately the most.<\/strong>\u00a0The illustrations below show the ideal condition, the model miss 2 in classify group 1, and miss 1 in classify group 0.<\/p>\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1fb65ooypLkeUXIHCnOqF8Q.png\"\/><\/figure>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1r2lpDazOuho1HOCJY0G03g.png\"\/><figcaption class=\"wp-element-caption\">logistic regression \u2014 ideal condition<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"de0b\"><strong>How to use logistic regression to predict return?<\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"d593\">The predicting ability of the linear regression is not good, and hard to predict the actual value.\u00a0<strong>Therefore, we try to simplify the problem. Change the target from its value to market movement(up and down).<\/strong>\u00a0After all, if we could know the market movement tomorrow, we can construct a trading strategy.<\/p>\n<h3 class=\"wp-block-heading\" id=\"f3a1\"><strong>Logistic regression analysis <\/strong><\/h3>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\" id=\"5369\">np.where(condition, Ture(ouput), False(output))\uff1a<br \/>While return is higher than 0, ouput 1, otherwise, -1.<\/p>\n<\/blockquote>\n<pre class=\"wp-block-code\"><code>data['signal'] = np.where(data['\u5831\u916c\u7387\uff05']&gt;0,1,-1)\n# \u7e6a\u5716\ndata[:100].plot(x = '\u65e5\u671f',\n                y = 'signal',\n                style = 'ro',\n                figsize = (10,6)\n               )<\/code><\/pre>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1mJzuGA4GOM_Npi0Q-SLoTA.png\"\/><figcaption class=\"wp-element-caption\">scatter<\/figcaption><\/figure>\n<pre class=\"wp-block-code\"><code>from sklearn import linear_model\nlm = linear_model.LogisticRegression(solver = 'lbfgs')\nR_t1 = x.reshape(1,-1).T\nsignal = y\nlm.fit(R_t1,signal)\npreditions = lm.predict(R_t1)\nplt.figure(figsize = (10, 6))\nplt.plot(x, y, 'ro', label = 'data')\nplt.plot(x, preditions, 'b', label = 'preditions')\nplt.legend(loc = 0)<\/code><\/pre>\n<figure class=\"wp-block-image aligncenter caption-align-center\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.tejwin.com\/en\/wp-content\/uploads\/2026\/08\/1KQm6-E9Ck8WFebQicKq_tQ.png\"\/><figcaption class=\"wp-element-caption\">predict result<\/figcaption><\/figure>\n<h3 class=\"wp-block-heading\" id=\"7942\"><strong>Accuracy <\/strong><\/h3>\n<p class=\"wp-block-paragraph\" id=\"9508\">The accuracy of prediction is 0.5471.<\/p>\n<pre class=\"wp-block-code\"><code>from sklearn.metrics import accuracy_score\npreditions = lm.predict(R_t1)\naccuracy_score(preditions, y)<\/code><\/pre>\n<h2 class=\"wp-block-heading\" id=\"3167\"><strong>Conclusion<\/strong><\/h2>\n<p class=\"wp-block-paragraph\" id=\"dd3e\">After implement machine learning projection by yourself,\u00a0<strong>did you find that it is difficult?<\/strong>\u00a0\ud83d\ude0e\ud83d\ude0e Although the accuracy of the model is not high, don\u2019t be discouraged about it.\u00a0<strong>Predicting the market is not the only way to make money in the market<\/strong>. In the future, we will share the reasons why you can make money without prediction, so please pay attention to our Medium continuously.<\/p>\n<p class=\"wp-block-paragraph\" id=\"ba5f\">If you like this topic, please click \ud83d\udc4f below, giving us more support and encouragement. Additionally, if you have any questions or suggestions, please leave a message or email us. We will try our best to reply to you.\ud83d\udc4d\ud83d\udc4d<\/p>\n<h2 class=\"wp-block-heading\" id=\"9fa2\"><strong>Links related to this article again!<\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>1\ufe0f\u20e3 API Official Website:\u00a0<\/strong><a href=\"https:\/\/api.tej.com.tw\/\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ API Official Website<\/strong><\/a><\/li>\n<li><strong>2\ufe0f\u20e3 The Product Package:\u00a0<\/strong><a href=\"https:\/\/eshop.tej.com.tw\/E-Shop\/Edata_caseIntro\/1\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ E-SHOP<\/strong><\/a><\/li>\n<li><strong>3\ufe0f\u20e3 Source Code:\u00a0<\/strong><a href=\"https:\/\/github.com\/tejtw\/TEJAPI_Python_Medium_Quant\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>TEJ GITHUB<\/strong><\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Preface Machine learning(ML), deep learning(DL), and artificial intelligence have been widely used by industries recently. Applications such as\u00a0image recognition and voice recognition\u00a0are common in our life. However, wall street, the world center of finance has already recruited many statistics, mathematics even physics experts as algorithmic engineers and quantitative traders to exploit trading strategies. There had [\u2026]<\/p>\n","protected":false},"featured_media":2813,"template":"","tags":[50,65,70],"insight_category":[16],"class_list":["post-2814","insight","type-insight","status-publish","has-post-thumbnail","hentry","tag-market-data","tag-python","tag-tej-api","insight_category-quant-data-science"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/insight\/2814","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/insight"}],"about":[{"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/types\/insight"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/media\/2813"}],"wp:attachment":[{"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/media?parent=2814"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/tags?post=2814"},{"taxonomy":"insight_category","embeddable":true,"href":"https:\/\/www.tejwin.com\/en\/wp-json\/wp\/v2\/insight_category?post=2814"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}