Machine LearningへのMathematicaの活用
～Wolfram Communityの活用実践例のご紹介～

Fujitsu Limited Principal Expert
Wolfram Community Featured Contributor
 岡崎 耕太郎

Join the Conversation @WolframJapan

Contents

Wolfram Community
	
投稿.  Deep Learning frameworkの動向
		  ～ クローリング･スクレイピングによるデータ収集 ～
	
投稿.  学習データの生成
		  ～ 紙面上のグラフをデータ化 ～
	
投稿.  学習データの収集から学習まで
		  ～ 音楽の生成 ～
	
最後に

Wolfram Community       http://community.wolfram.com/

･世界中のMathematicaユーザの多彩で思いもつかないアイデアに出会える
･125カ国以上、2万人を越えるMathematicaユーザが 100以上のグループで議論

Deep Learning frameworkの動向 ～ クローリング･スクレイピングによるデータ収集 ～
Open Source Deep Learning framework GitHub Stars  (MouseoverTemplate["http://community.wolfram.com/groups/-/m/t/1345642", "http://community.wolfram.com/groups/-/m/t/1345642", ])

TensorFlow, PyTorch, MXNetなどのGitHubをクローリング&スクレイピング

(MouseoverTemplate["https://github.com/tensorflow/tensorflow", "https://github.com/tensorflow/tensorflow", ])

1. トータルのスター数

・GitHub DeveloperのREST API v3を元にベースに情報を取得

(MouseoverTemplate["https://developer.github.com/v3/repos/", "https://developer.github.com/v3/repos/", ])

(* リポジトリのURL生成 *)
totalstarsURL[r_String]:=StringTemplate["https://api.github.com/repos/`r`"][<|"r"->r|>];

(* TensorFlowのURL *)
totalstarsURL["tensorflow/tensorflow"]

(* トークン *)
token="token abcdefghijklmnopqrstuvwxyz0123456789abcd"; (*sample*)

(* TensorFlowのトップページをJSON形式で獲得 *)
req=HTTPRequest[totalstarsURL["tensorflow/tensorflow"],<|"Headers"->{"Authorization"->token}|>]

res=Association@Import[req,"JSON"]

\[LeftAssociation]"id"->45717250,"node_id"->"MDEwOlJlcG9zaXRvcnk0NTcxNzI1MA==","name"->"tensorflow","full_name"->"tensorflow/tensorflow","private"->False,"owner"->{"login"->"tensorflow","id"->15658638,"node_id"->"MDEyOk9yZ2FuaXphdGlvbjE1NjU4NjM4","avatar_url"->"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id"->"","url"->"https://api.github.com/users/tensorflow","html_url"->"https://github.com/tensorflow","followers_url"->"https://api.github.com/users/tensorflow/followers","following_url"->"https://api.github.com/users/tensorflow/following{/other_user}","gists_url"->"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url"->"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url"->"https://api.github.com/users/tensorflow/subscriptions","organizations_url"->"https://api.github.com/users/tensorflow/orgs","repos_url"->"https://api.github.com/users/tensorflow/repos","events_url"->"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url"->"https://api.github.com/users/tensorflow/received_events","type"->"Organization","site_admin"->False},"html_url"->"https://github.com/tensorflow/tensorflow","description"->"An Open Source Machine Learning Framework for Everyone","fork"->False,"url"->"https://api.github.com/repos/tensorflow/tensorflow","forks_url"->"https://api.github.com/repos/tensorflow/tensorflow/forks","keys_url"->"https://api.github.com/repos/tensorflow/tensorflow/keys{/key_id}","collaborators_url"->"https://api.github.com/repos/tensorflow/tensorflow/collaborators{/collaborator}","teams_url"->"https://api.github.com/repos/tensorflow/tensorflow/teams","hooks_url"->"https://api.github.com/repos/tensorflow/tensorflow/hooks","issue_events_url"->"https://api.github.com/repos/tensorflow/tensorflow/issues/events{/number}","events_url"->"https://api.github.com/repos/tensorflow/tensorflow/events","assignees_url"->"https://api.github.com/repos/tensorflow/tensorflow/assignees{/user}","branches_url"->"https://api.github.com/repos/tensorflow/tensorflow/branches{/branch}","tags_url"->"https://api.github.com/repos/tensorflow/tensorflow/tags","blobs_url"->"https://api.github.com/repos/tensorflow/tensorflow/git/blobs{/sha}","git_tags_url"->"https://api.github.com/repos/tensorflow/tensorflow/git/tags{/sha}","git_refs_url"->"https://api.github.com/repos/tensorflow/tensorflow/git/refs{/sha}","trees_url"->"https://api.github.com/repos/tensorflow/tensorflow/git/trees{/sha}","statuses_url"->"https://api.github.com/repos/tensorflow/tensorflow/statuses/{sha}","languages_url"->"https://api.github.com/repos/tensorflow/tensorflow/languages","stargazers_url"->"https://api.github.com/repos/tensorflow/tensorflow/stargazers","contributors_url"->"https://api.github.com/repos/tensorflow/tensorflow/contributors","subscribers_url"->"https://api.github.com/repos/tensorflow/tensorflow/subscribers","subscription_url"->"https://api.github.com/repos/tensorflow/tensorflow/subscription","commits_url"->"https://api.github.com/repos/tensorflow/tensorflow/commits{/sha}","git_commits_url"->"https://api.github.com/repos/tensorflow/tensorflow/git/commits{/sha}","comments_url"->"https://api.github.com/repos/tensorflow/tensorflow/comments{/number}","issue_comment_url"->"https://api.github.com/repos/tensorflow/tensorflow/issues/comments{/number}","contents_url"->"https://api.github.com/repos/tensorflow/tensorflow/contents/{+path}","compare_url"->"https://api.github.com/repos/tensorflow/tensorflow/compare/{base}...{head}","merges_url"->"https://api.github.com/repos/tensorflow/tensorflow/merges","archive_url"->"https://api.github.com/repos/tensorflow/tensorflow/{archive_format}{/ref}","downloads_url"->"https://api.github.com/repos/tensorflow/tensorflow/downloads","issues_url"->"https://api.github.com/repos/tensorflow/tensorflow/issues{/number}","pulls_url"->"https://api.github.com/repos/tensorflow/tensorflow/pulls{/number}","milestones_url"->"https://api.github.com/repos/tensorflow/tensorflow/milestones{/number}","notifications_url"->"https://api.github.com/repos/tensorflow/tensorflow/notifications{?since,all,participating}","labels_url"->"https://api.github.com/repos/tensorflow/tensorflow/labels{/name}","releases_url"->"https://api.github.com/repos/tensorflow/tensorflow/releases{/id}","deployments_url"->"https://api.github.com/repos/tensorflow/tensorflow/deployments","created_at"->"2015-11-07T01:19:20Z","updated_at"->"2018-12-16T00:48:39Z","pushed_at"->"2018-12-15T22:18:53Z","git_url"->"git://github.com/tensorflow/tensorflow.git","ssh_url"->"git@github.com:tensorflow/tensorflow.git","clone_url"->"https://github.com/tensorflow/tensorflow.git","svn_url"->"https://github.com/tensorflow/tensorflow","homepage"->"https://tensorflow.org","size"->293361,"stargazers_count"->116716,"watchers_count"->116716,"language"->"C++","has_issues"->True,"has_projects"->True,"has_downloads"->True,"has_wiki"->False,"has_pages"->False,"forks_count"->70577,"mirror_url"->Null,"archived"->False,"open_issues_count"->1800,"license"->{"key"->"apache-2.0","name"->"Apache License 2.0","spdx_id"->"Apache-2.0","url"->"https://api.github.com/licenses/apache-2.0","node_id"->"MDc6TGljZW5zZTI="},"forks"->70577,"open_issues"->1800,"watchers"->116716,"default_branch"->"master","permissions"->{"admin"->False,"push"->False,"pull"->True},"organization"->{"login"->"tensorflow","id"->15658638,"node_id"->"MDEyOk9yZ2FuaXphdGlvbjE1NjU4NjM4","avatar_url"->"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id"->"","url"->"https://api.github.com/users/tensorflow","html_url"->"https://github.com/tensorflow","followers_url"->"https://api.github.com/users/tensorflow/followers","following_url"->"https://api.github.com/users/tensorflow/following{/other_user}","gists_url"->"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url"->"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url"->"https://api.github.com/users/tensorflow/subscriptions","organizations_url"->"https://api.github.com/users/tensorflow/orgs","repos_url"->"https://api.github.com/users/tensorflow/repos","events_url"->"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url"->"https://api.github.com/users/tensorflow/received_events","type"->"Organization","site_admin"->False},"network_count"->70577,"subscribers_count"->8570\[RightAssociation]

・スターの最新更新日付とトータル数

(* トップページからスターの最新更新日付とトータル数を獲得 *)
gettotalstars[r_String]:=Module[{req,res},
req=HTTPRequest[totalstarsURL[r],<|"Headers"->{"Authorization"->token}|>];
res=Association@Import[req,"JSON"];
{DateObject@res["updated_at"],res["stargazers_count"]}]  (* 評価時は黄色の背景色を削除 *)

gettotalstars["tensorflow/tensorflow"]

2. スターの履歴

(MouseoverTemplate["https://developer.github.com/v3/activity/starring/", "https://developer.github.com/v3/activity/starring/", ])

・スターを付けた人を1page当たり100人単位で獲得

(* nページ目のURL *)
stargazersURL[r_String,n_Integer]:=totalstarsURL[r]<>"/stargazers?page="<>ToString[n]<>"&per_page=100";

(* 1ページ目のURL *)
stargazersURL["tensorflow/tensorflow",1]

・獲得できる最大page数の獲得

req=HTTPRequest[stargazersURL["tensorflow/tensorflow",1],<|"Headers"->{"Authorization"->token,"Accept"->"application/vnd.github.v3.star+json"}|>];
res=Association@URLRead[req,"Headers"]

\[LeftAssociation]"date"->"Mon, 03 Dec 2018 06:30:32 GMT","content-type"->"application/json; charset=utf-8","transfer-encoding"->"chunked","server"->"GitHub.com","status"->"200 OK","x-ratelimit-limit"->"5000","x-ratelimit-remaining"->"4992","x-ratelimit-reset"->"1543821120","cache-control"->"private, max-age=60, s-maxage=60","vary"->"Accept-Encoding","etag"->"W/\"e755922d9856177ffc831a6d397a2f99\"","x-oauth-scopes"->"repo","x-accepted-oauth-scopes"->"","x-github-media-type"->"github.v3; param=star; format=json","link"->"<https://api.github.com/repositories/45717250/stargazers?page=2&per_page=100>; rel=\\\, <https://api.github.com/repositories/45717250/stargazers?page=400&per_page=100>; rel=\\\","access-control-expose-headers"->"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type","access-control-allow-origin"->"*","strict-transport-security"->"max-age=31536000; includeSubdomains; preload","x-frame-options"->"deny","x-content-type-options"->"nosniff","x-xss-protection"->"1; mode=block","referrer-policy"->"origin-when-cross-origin, strict-origin-when-cross-origin","content-security-policy"->"default-src 'none'","content-encoding"->"gzip","x-github-request-id"->"8BD0:166B:3E0DB41:458D067:5C04CD88"\[RightAssociation]

(* 獲得できる最大page数 *)
getpageMax[r_String]:=Module[{req,res,max},
req=HTTPRequest[stargazersURL[r,1],<|"Headers"->{"Authorization"->token,"Accept"->"application/vnd.github.v3.star+json"}|>];
res=Association@URLRead[req,"Headers"];
max=StringCases[res[["link"]],RegularExpression["next.*?\!\(\*StyleBox[RowBox[{"page", "=", RowBox[{RowBox[{RowBox[{"(", RowBox[{RowBox[{"\\\\", "d"}], "+"}], ")"}], "."}], "*", RowBox[{"?", "last"}]}]}],Background->RGBColor[1, 1, 0]]\)"]->"$1"][[1]]; (* 評価時は黄色の背景色を削除 *)
ToExpression[max]]

(* TensorFlowの最大page数は400 *)
getpageMax["tensorflow/tensorflow"]

・page番号指定でpageの獲得

getpage[r_String,n_Integer]:=Module[{req,res},
req=HTTPRequest[stargazersURL[r,n],<|"Headers"->{"Authorization"->token,"Accept"->"application/vnd.github.v3.star+json"}|>];
Association/@Import[req,"JSON"]]

・全pageの獲得

getpages[r_String]:=Module[{pageMax,pages},
pageMax=getpageMax[r];
pages=getpage[r,#]&/@Range[pageMax]]

pages=getpages["tensorflow/tensorflow"];

Length@pages

・各スターを付けた日付の獲得

pages[[1,1]]

\[LeftAssociation]"starred_at"->"2015-11-09T13:17:30Z","user"->{"login"->"mrry","id"->192142,"node_id"->"MDQ6VXNlcjE5MjE0Mg==","avatar_url"->"https://avatars1.githubusercontent.com/u/192142?v=4","gravatar_id"->"","url"->"https://api.github.com/users/mrry","html_url"->"https://github.com/mrry","followers_url"->"https://api.github.com/users/mrry/followers","following_url"->"https://api.github.com/users/mrry/following{/other_user}","gists_url"->"https://api.github.com/users/mrry/gists{/gist_id}","starred_url"->"https://api.github.com/users/mrry/starred{/owner}{/repo}","subscriptions_url"->"https://api.github.com/users/mrry/subscriptions","organizations_url"->"https://api.github.com/users/mrry/orgs","repos_url"->"https://api.github.com/users/mrry/repos","events_url"->"https://api.github.com/users/mrry/events{/privacy}","received_events_url"->"https://api.github.com/users/mrry/received_events","type"->"User","site_admin"->False}\[RightAssociation]

(* page内のスターを付けた日付の獲得 *)
getstardates[pages_List]:=Module[{res},
res=#["starred_at"]&/@Flatten[pages]; (* 評価時は黄色の背景色を削除 *)
Select[DateObject/@res,DateObjectQ]]

(* 全pageのスターを付けた日付の獲得 *)
dates=getstardates[pages];

dates

・スターを付けた日付のグラフ化

plotlist[r_String,dates_List]:=Module[{pageMax,total,lastdate,pl},
pageMax=getpageMax[r]; (* 最大page数 *)
{lastdate,total}=gettotalstars[r]; (* スターの最新更新日付とトータル数 *)
pl=Transpose[{dates,Range[Length[dates]]}]; (* 更新日付の累積リスト *)
(* トータル数が制限の400ページに納まらない場合を考慮 *)
If[pageMax==400&&dates[[-1]]≠lastdate,AppendTo[pl,{lastdate,total}],pl]]

tfd=plotlist["tensorflow/tensorflow",dates];
DateListPlot[tfd,"Day",PlotRange->All,FrameLabel->{None,"GitHub Stars"},PlotTheme->"Detailed"]

・主なOpen Source Deep Learning framework

3. スターを付けた人の国の分布

・スターを付けた人のURLの獲得

pages[[1,1]]

\[LeftAssociation]"starred_at"->"2015-11-09T13:17:30Z","user"->{"login"->"mrry","id"->192142,"node_id"->"MDQ6VXNlcjE5MjE0Mg==","avatar_url"->"https://avatars1.githubusercontent.com/u/192142?v=4","gravatar_id"->"","url"->"https://api.github.com/users/mrry","html_url"->"https://github.com/mrry","followers_url"->"https://api.github.com/users/mrry/followers","following_url"->"https://api.github.com/users/mrry/following{/other_user}","gists_url"->"https://api.github.com/users/mrry/gists{/gist_id}","starred_url"->"https://api.github.com/users/mrry/starred{/owner}{/repo}","subscriptions_url"->"https://api.github.com/users/mrry/subscriptions","organizations_url"->"https://api.github.com/users/mrry/orgs","repos_url"->"https://api.github.com/users/mrry/repos","events_url"->"https://api.github.com/users/mrry/events{/privacy}","received_events_url"->"https://api.github.com/users/mrry/received_events","type"->"User","site_admin"->False}\[RightAssociation]

(* ｽﾀｰを付けた人のＵＲＬの獲得*)
getpersonURLs[pages_List]:=Union[("url"/.#[["user"]])&/@Flatten[pages]] (* 評価時は黄色の背景色を削除 *)

(*　４万人分のＵＲＬ *)
purls=getpersonURLs[pages]

・スターを付けた人のlocationの獲得

req=HTTPRequest["https://api.github.com/users/000",<|"Headers"->{"Authorization"->token}|>];
res=Association@Import[req,"JSON"]

\[LeftAssociation]"login"->"000","id"->575755,"node_id"->"MDQ6VXNlcjU3NTc1NQ==","avatar_url"->"https://avatars3.githubusercontent.com/u/575755?v=4","gravatar_id"->"","url"->"https://api.github.com/users/000","html_url"->"https://github.com/000","followers_url"->"https://api.github.com/users/000/followers","following_url"->"https://api.github.com/users/000/following{/other_user}","gists_url"->"https://api.github.com/users/000/gists{/gist_id}","starred_url"->"https://api.github.com/users/000/starred{/owner}{/repo}","subscriptions_url"->"https://api.github.com/users/000/subscriptions","organizations_url"->"https://api.github.com/users/000/orgs","repos_url"->"https://api.github.com/users/000/repos","events_url"->"https://api.github.com/users/000/events{/privacy}","received_events_url"->"https://api.github.com/users/000/received_events","type"->"User","site_admin"->False,"name"->"Tanat Tonguthaisri","company"->"@data-society","blog"->"https://gitshowcase.com/000","location"->"Bangkok","email"->Null,"hireable"->True,"bio"->"Epicure","public_repos"->12,"public_gists"->0,"followers"->9,"following"->158,"created_at"->"2011-01-21T04:29:48Z","updated_at"->"2018-12-01T05:05:40Z"\[RightAssociation]

(* ｽﾀｰを付けた人のlocationの獲得 *)
getlocations[url_String]:=Module[{req,res,loc},
req=HTTPRequest[url,<|"Headers"->{"Authorization"->token}|>];
res=Association@Import[req,"JSON"];
loc=res[["location"]]] (* 評価時は黄色の背景色を削除 *)

(*　４万人分のlocation *)
locs=getlocations/@purls;

・locationを国に変換

- locationに国名を記載している場合

locs[[10]]
SemanticInterpretation@locs[[10]]

- locationに都市を記載している場合

locs[[5]]
SemanticInterpretation@locs[[5]]

EntityValue[(Entity["Bengaluru", Entity["City",{"Bengaluru","Karnataka","India"}], "Entity[\"City\", {\"Bengaluru\", \"Karnataka\", \"India\"}]", "city"]),"Country"]

(* ｽﾀｰを付けた人の国の獲得 *)
getCountry[loc_String]:=Module[{s,tmp},
s=SemanticInterpretation[loc];
tmp=If[s[[1]]=="Country",s,EntityValue[s,"Country"]]]

(*　４万人分の国 *)
countries=getCountry/@locs;
s=Select[countries,SameQ[#[[1]],"Country"]&];
Length@s

・スターを付けた人の国 Best8

country8tf=Sort[Tally[s],#1[[2]]>#2[[2]]&][[1;;8]]

・スターを付けた人の国 Best8の可視化

Grid[{Prepend[Range[8],""],Prepend[ImageResize[#,50]&/@(#["Flag"]&/@Transpose[country8tf][[1]]),"TensorFlow"]},Frame->All]

・主なOpen Source Deep Learning framework

学習データの生成 ～ 紙面上のグラフをデータ化 ～
GraphImage2List: more automatically   (MouseoverTemplate["http://community.wolfram.com/groups/-/m/t/1333219", "http://community.wolfram.com/groups/-/m/t/1333219", ])

紙面上のグラフをデータ化し、機械学習の学習データなどに利用

1. グラフのｘ軸、ｙ軸、原点を検出

・紙面上のグラフをスキャン

img=;

・画像中の線分を検出

lines=ImageLines[EdgeDetect[img,1],0.2];
lines[[1]]
HighlightImage[img,lines]

・水平な線(x軸）と垂直な線(y軸）を検出

axisX=Select[lines,Abs[#[[1,1,2]]-#[[1,2,2]]]<1&]
axisY=Select[lines,Abs[#[[1,1,1]]-#[[1,2,1]]]<1&]

・x軸とy軸の交点である原点を検出

{OX,OY}={axisY[[1,1,1,1]],axisX[[1,1,1,2]]};
HighlightImage[img,Point[{OX,OY}]]

2. グラフのｘ軸とｙ軸の目盛りを検出

・画像中の文字(目盛り)を認識

ToExpression/@TextRecognize[img,"Block",Language->"English"]

・ｘ軸とｙ軸の目盛りの値が記載されている位置を検出

{scaleY,scaleX}=ToExpression/@TextRecognize[img,"Block",Language->"English"][[{1,-1}]]
rect=TextRecognize[img,"Block","BoundingBox",Language->"English"][[{1,-1}]]
HighlightImage[img,{"Boundary",rect}]

・ｘ軸とｙ軸の目盛りの位置を検出

scaleposX={RegionCentroid[rect[[2]]][[1]],OY};
scaleposY={OX,RegionCentroid[rect[[1]]][[2]]};
HighlightImage[img,{"Boundary",{scaleposX,scaleposY,{OX,OY}}}]

3. グラフの曲線を検出

・画像中の主調色を認識

DominantColors[img,ColorCoverage->0,MinColorDistance->.25]

・曲線の色を選択

curvecolor=DominantColors[img,ColorCoverage->0,MinColorDistance->.25][[2]]

・曲線の画素位置を検出

points=PixelValuePositions[img,curvecolor,.075];
HighlightImage[img,{"Boundary",points}]

4. グラフのデータ化

・求めた原点、目盛り、曲線の位置を元にデータ化

convert[{x_,y_}]:={(x-OX)*(scaleX/(scaleposX[[1]]-OX)),(y-OY)*(scaleY/(scaleposY[[2]]-OY))};
list=Sort[convert/@points];
list//Shallow

・グラフ表示

Grid[{{"紙面上のグラフ","ListPlot"},{img,ListPlot[list,Joined->True,
PlotStyle->PointSize[0.0025],LabelStyle->Directive[Black,12]]}},Frame->All]

学習データの収集から学習まで ～ 音楽の生成 ～
Music Generation with GAN MidiNet  (MouseoverTemplate["http://community.wolfram.com/groups/-/m/t/1435251", "http://community.wolfram.com/groups/-/m/t/1435251", ])

GAN（MidiNet)を使い音楽を生成  (MouseoverTemplate["https://arxiv.org/abs/1703.10847", "https://arxiv.org/abs/1703.10847", ])

1. MIDIデータの収集

・最初に学習データの元となる音源のMIDIデータをウェブサイトから収集

url="http://www.midiworld.com/download/1366";
notes=Select[Import[url,{"SoundNotes"}],Length[#]>0&];
notes[[All,1,3]]

Sound[notes[[1]]]

・べースのSoundNote詳細

notes[[1]]

SoundNote[pitch, {開始時刻、終了時刻}, スタイル, ボリューム]

2. MIDIデータを画像データに変換

・MIDIデータのpitch(C-1～G9）を1～128の数値に変換するルールを用意

codebase={"C","C#","D","D#","E","F","F#","G","G#","A","A#","B"};
num=ToString/@Range[-1,9];
pitch2numberrule=Take[Thread[StringJoin/@Reverse/@Tuples[{num,codebase}]->Range[0,131]+1],128]

・16分音符を最小単位としMIDIデータ1小節を128*16画像データに変換

(* 16部音符の長さ *)
tempo=108;
note16=60/(4*tempo);

(* 時刻tのSoundNoteを1つ選択 *)
select16[snlist_,t_]:=Select[snlist,(t≤#[[2,1]]≤t+note16)||(t≤#[[2,2]]≤t+note16)||(#[[2,1]]<t&&#[[2,2]]>t+note16)&,1]

(* 時刻tからの1小節分の選択 *)
selectbar[snlist_,t_]:=select16[snlist,#]&/@Most@Range[t,t+note16*16,note16]

(* SoundNoteのpitchを1～128の数値に変換 *)
selectpitch[x_]:=If[x==={},0,x[[1,1]]]/.pitch2numberrule

(* 時刻tからの1小節分のpitchのリスト化 *)
pixelbar[snlist_,t_]:=Module[{bar,x,y},
bar=selectbar[snlist,t];
x=selectpitch/@bar;
y=Range[16];
Transpose[{x,y}]]

(* 時刻tからの1小節分を128*16の画像データ化 *)
imagebar[snlist_,t_]:=Module[{image},
image=ConstantArray[0,{128,16}];
Quiet[(image[[129-#[[1]],#[[2]]]]=1)&/@pixelbar[snlist,t]];
Image[image]]

(* SoundNoteを1小節単位で128*16の画像データ化 *)
soundnote2image[soundnotelist_]:=Module[{min,max,data2},
{min,max}=MinMax[#[[2]]&/@soundnotelist//Flatten];
data2={#[[1]],#[[2]]-min}&/@soundnotelist;
Table[imagebar[data2,t],{t,0,max-min,note16*16}]]

(images=soundnote2image[notes[[1]]])[[;;16]]

3. ネットワークモデルの生成

・MidiNetの論文を参考にネットワークモデルを作成

100次元の乱数(図:Noize z)と 1小節前のデータ(図:2D conditions)を

元に次の1小節を生成　※今回はコード(図:1D conditions)は使用しない

(MouseoverTemplate["https://arxiv.org/abs/1703.10847", "https://arxiv.org/abs/1703.10847", ])

・Generator

generator=NetGraph[{1024,BatchNormalizationLayer[],Ramp,256,BatchNormalizationLayer[],Ramp,ReshapeLayer[{128,1,2}],DeconvolutionLayer[64,{1,2},"Stride"->{2,2}],BatchNormalizationLayer[],Ramp,DeconvolutionLayer[64,{1,2},"Stride"->{2,2}],BatchNormalizationLayer[],Ramp,DeconvolutionLayer[64,{1,2},"Stride"->{2,2}],BatchNormalizationLayer[],Ramp,DeconvolutionLayer[1,{128,1},"Stride"->{2,1}],LogisticSigmoid,ConvolutionLayer[16,{128,1},"Stride"->{2,1}],BatchNormalizationLayer[],Ramp,ConvolutionLayer[16,{1,2},"Stride"->{1,2}],BatchNormalizationLayer[],Ramp,ConvolutionLayer[16,{1,2},"Stride"->{1,2}],BatchNormalizationLayer[],Ramp,ConvolutionLayer[16,{1,2},"Stride"->{1,2}],BatchNormalizationLayer[],Ramp,CatenateLayer[],CatenateLayer[],CatenateLayer[],CatenateLayer[]},{NetPort["noise"]->1,NetPort["prev"]->19,19->20->21->22->23->24->25->26->27->28->29->30,1->2->3->4->5->6->7,{7,30}->31,31->8->9->10,{10,27}->32,32->11->12->13,{13,24}->33,33->14->15->16,{16,21}->34,34->17->18},"noise"->{100},"prev"->{1,128,16}]

・Discriminator

discriminator=NetGraph[{ConvolutionLayer[64,{89,4},"Stride"->{1,1}],Ramp,ConvolutionLayer[64,{1,4},"Stride"->{1,1}],Ramp,ConvolutionLayer[16,{1,4},"Stride"->{1,1}],Ramp,1},{1->2->3->4->5->6->7},"Input"->{1,128,16}]

・GeneratorとDiscriminatorを使い Wasserstein GAN

(MouseoverTemplate["https://arxiv.org/abs/1701.07875", "https://arxiv.org/abs/1701.07875", ])

ganNet=NetInitialize[NetGraph[<|"gen"->generator,"discrimop"->NetMapOperator[discriminator],"cat"->CatenateLayer[],"reshape"->ReshapeLayer[{2,1,128,16}],"flat"->ReshapeLayer[{2}],"scale"->ConstantTimesLayer["Scaling"->{-1,1}],"total"->SummationLayer[]|>,{{NetPort["noise"],NetPort["prev"]}->"gen"->"cat",NetPort["Input"]->"cat","cat"->"reshape"->"discrimop"->"flat"->"scale"->"total"},"Input"->{1,128,16}]]

4. 学習データの作成

・Midiデータから変換した画像データをバッチサイズ16の学習データに変換
  - 学習データは、“noise”, “prev”, “Input”の3つで構成
  - “noise”は100次元の乱数
  - “prev”は各バッチの先頭は all 0、以降は1小節前の真のデータ
  - “Input”は真のデータ

(* バッチサイズ *)
batchsize=16;

(* 画像群からバッチサイズの整数倍を抽出 *)
getbatchsizeimages[i_]:=i[[;;batchsize*Floor[Length[i]/batchsize]]]

(* Midiデータから変換した画像群からバッチサイズの整数倍を抽出 *)
imagesall=Flatten[Join[getbatchsizeimages/@{images}]];
{Length[imagesall],Length[imagesall]*note16*16//N}

(* noiseの次元 *)
randomDim=100;

(* バッチ数 *)
n=Floor[Length@imagesall/batchsize]; 

(* noiseの生成 *)
noise=Table[RandomReal[NormalDistribution[0,1],{randomDim}],batchsize*n];

(* inputの生成 *)
input=ArrayReshape[ImageData[#],{1,128,16}]&/@imagesall[[;;batchsize*n]];

(* prevの生成 *)
prev=Flatten[Join[Table[{{ConstantArray[0,{1,128,16}]},input[[batchsize*(i-1)+1;;batchsize*i-1]]},{i,1,n}]],2];

(* 学習データの生成 *)
trainingData=AssociationThread[{"noise","prev","Input"}->{#[[1]],#[[2]],#[[3]]}]&/@Transpose[{noise,prev,input}];
trainingData[[1]]

5. 学習

・GPUで学習

net=NetTrain[ganNet,trainingData,All,LossFunction->"Output",Method->{"RMSProp","LearningRate"->0.00005,"WeightClipping"->{"discrimop"->0.01}},LearningRateMultipliers->{"scale"->0,"gen"->-0.2},TargetDevice->"GPU",BatchSize->batchsize,MaxTrainingRounds->50000]

6. MIDIデータの生成

・学習したネットワークモデルを使って16小節を生成

bars={};
newbar=Image[ConstantArray[0,{1,128,16}]];
For[i=1,i<17,i++,noise1=RandomReal[NormalDistribution[0,1],{randomDim}];
prev1={ImageData[newbar]};
newbar=NetDecoder[{"Image","Grayscale"}][NetExtract[net["TrainedNet"],"gen"][<|"noise"->noise1,"prev"->prev1|>]];
AppendTo[bars,newbar]]

(* 生成された16小節分の画像 *)
bars

・Wasserstein GANは生成画像がぼやける傾向があるため、生成画像にフィルタをかけてクリア化

clearbar[bar_,threshold_]:=Module[{i,barx,col,max},
barx=ConstantArray[0,{128,16}];
col=Transpose[bar//ImageData]; (* 画像データbarをデータ化 *)
For[i=1,i<17,i++,max=Max[col[[i]]]; (* 各列の最大値を検出 *)
(* 最大値がしきい値以上なら１を設定 *)
If[max≥threshold,barx[[First@Position[col[[i]],max,1],i]]=1]];
Image[barx]]

bars2=clearbar[#,0.1]&/@bars

・画像データをMIDIデータに逆変換

(* 1～128の数値をpitch(C-1～G-9）に変換するﾙｰﾙ *)
number2pitchrule=Reverse/@pitch2numberrule;

(* 画像データをSoundNote形式に変換　*)
images2soundnote[img_,start_]:=
SoundNote[(129-#[[2]])/.number2pitchrule,
{(#[[1]]-1)*note16,#[[1]]*note16}+start,
"ElectricBass",SoundVolume->1]&/@Sort@(Reverse/@Position[(img//ImageData)/.(1->1.),1.])

(* 連続する同音を連結 *)
snjoinrule={x___,SoundNote[s_,{t_,u_},v_,w_],SoundNote[s_,{u_,z_},v_,w_],y___}->{x,SoundNote[s,{t,z},v,w],y};

(* 生成した画像データをMidiデータに変換し再生 *)
Sound[Flatten@MapIndexed[
(images2soundnote[#1,note16*16*(First[#2]-1)]//.snjoinrule)&,bars2]]

最後に

Sign In & Connect

“make the world as computable as possible”

Thank you

okazaki.kotaro@fujitsu.com