Universal Music Scraper broken (fix)

Donator garretn, 38 days ago

Looks like MusicBrainz now flat out denies requests that don't include a UserAgent. Adding a useragent to the source seems to fix the issue:

--- scrapers/UniversalMusicScraper.cpp.orig    2017-02-13 09:03:11.801668907 -0600
+++ scrapers/UniversalMusicScraper.cpp    2017-02-13 09:03:47.657669802 -0600
@@ -71,6 +71,7 @@
 {
     QUrl url(QString("http://musicbrainz.org/ws/2/artist/?query=artist:\"%1\"").arg(QString(QUrl::toPercentEncoding(searchStr))));
     QNetworkRequest request(url);
+    request.setRawHeader("User-Agent" , "MediaElch");
     QNetworkReply *reply = qnam()->get(request);
     new NetworkReplyWatcher(this, reply);
     connect(reply, SIGNAL(finished()), this, SLOT(onSearchArtistFinished()));

Though this is better:

--- scrapers/UniversalMusicScraper.cpp.orig    2017-02-13 09:10:26.305679745 -0600
+++ scrapers/UniversalMusicScraper.cpp    2017-02-13 09:10:56.909680509 -0600
@@ -69,8 +69,9 @@
 
 void UniversalMusicScraper::searchArtist(QString searchStr)
 {
-    QUrl url(QString("http://www.musicbrainz.org/ws/2/artist/?query=artist:\"%1\"").arg(QString(QUrl::toPercentEncoding(searchStr))));
+    QUrl url(QString("http://musicbrainz.org/ws/2/artist/?query=artist:\"%1\"").arg(QString(QUrl::toPercentEncoding(searchStr))));
     QNetworkRequest request(url);
+    request.setRawHeader("User-Agent" , "MediaElch");
     QNetworkReply *reply = qnam()->get(request);
     new NetworkReplyWatcher(this, reply);
     connect(reply, SIGNAL(finished()), this, SLOT(onSearchArtistFinished()));
@@ -119,7 +120,7 @@
     artist->clear(infos);
     artist->setMbId(mbId);
     artist->setAllMusicId("");
-    QUrl url(QString("http://www.musicbrainz.org/ws/2/artist/%1?inc=url-rels").arg(mbId));
+    QUrl url(QString("http://musicbrainz.org/ws/2/artist/%1?inc=url-rels").arg(mbId));
     QNetworkReply *reply = qnam()->get(QNetworkRequest(url));
     reply->setProperty("storage", Storage::toVariant(reply, artist));
     reply->setProperty("infosToLoad", Storage::toVariant(reply, infos));
@@ -303,7 +304,7 @@
     QString searchQuery = "release:\"" + QString(QUrl::toPercentEncoding(cleanSearchStr)) + "\"";
     if (!artistName.isEmpty())
         searchQuery += "%20AND%20artist:\"" + QString(QUrl::toPercentEncoding(artistName)) + "\"";
-    QUrl url(QString("http://www.musicbrainz.org/ws/2/release/?query=%1").arg(searchQuery));
+    QUrl url(QString("http://musicbrainz.org/ws/2/release/?query=%1").arg(searchQuery));
     QNetworkRequest request(url);
     QNetworkReply *reply = qnam()->get(request);
     new NetworkReplyWatcher(this, reply);
@@ -386,7 +387,7 @@
     album->setMbAlbumId(mbAlbumId);
     album->setMbReleaseGroupId(mbReleaseGroupId);
     album->setAllMusicId("");
-    QUrl url(QString("http://www.musicbrainz.org/ws/2/release/%1?inc=url-rels+labels+artist-credits").arg(mbAlbumId));
+    QUrl url(QString("http://musicbrainz.org/ws/2/release/%1?inc=url-rels+labels+artist-credits").arg(mbAlbumId));
     QNetworkReply *reply = qnam()->get(QNetworkRequest(url));
     new NetworkReplyWatcher(this, reply);
     reply->setProperty("storage", Storage::toVariant(reply, album));
 

All of the URLs seem to redirect to the non-www version, so this shaves off needless requests.

  • Donator

    garretn, 38 days ago

    Updated patch

    --- scrapers/UniversalMusicScraper.cpp.orig    2017-02-13 09:10:26.305679745 -0600
    +++ scrapers/UniversalMusicScraper.cpp    2017-02-13 09:15:08.765686791 -0600
    @@ -69,8 +69,9 @@
     
     void UniversalMusicScraper::searchArtist(QString searchStr)
     {
    -    QUrl url(QString("http://www.musicbrainz.org/ws/2/artist/?query=artist:\"%1\"").arg(QString(QUrl::toPercentEncoding(searchStr))));
    +    QUrl url(QString("http://musicbrainz.org/ws/2/artist/?query=artist:\"%1\"").arg(QString(QUrl::toPercentEncoding(searchStr))));
         QNetworkRequest request(url);
    +    request.setRawHeader("User-Agent" , "MediaElch");
         QNetworkReply *reply = qnam()->get(request);
         new NetworkReplyWatcher(this, reply);
         connect(reply, SIGNAL(finished()), this, SLOT(onSearchArtistFinished()));
    @@ -119,8 +120,10 @@
         artist->clear(infos);
         artist->setMbId(mbId);
         artist->setAllMusicId("");
    -    QUrl url(QString("http://www.musicbrainz.org/ws/2/artist/%1?inc=url-rels").arg(mbId));
    -    QNetworkReply *reply = qnam()->get(QNetworkRequest(url));
    +    QUrl url(QString("http://musicbrainz.org/ws/2/artist/%1?inc=url-rels").arg(mbId));
    +    QNetworkRequest request(url);
    +    request.setRawHeader("User-Agent" , "MediaElch");
    +    QNetworkReply *reply = qnam()->get(request);
         reply->setProperty("storage", Storage::toVariant(reply, artist));
         reply->setProperty("infosToLoad", Storage::toVariant(reply, infos));
         connect(reply, SIGNAL(finished()), this, SLOT(onArtistRelsFinished()));
    @@ -303,8 +306,9 @@
         QString searchQuery = "release:\"" + QString(QUrl::toPercentEncoding(cleanSearchStr)) + "\"";
         if (!artistName.isEmpty())
             searchQuery += "%20AND%20artist:\"" + QString(QUrl::toPercentEncoding(artistName)) + "\"";
    -    QUrl url(QString("http://www.musicbrainz.org/ws/2/release/?query=%1").arg(searchQuery));
    +    QUrl url(QString("http://musicbrainz.org/ws/2/release/?query=%1").arg(searchQuery));
         QNetworkRequest request(url);
    +    request.setRawHeader("User-Agent" , "MediaElch");
         QNetworkReply *reply = qnam()->get(request);
         new NetworkReplyWatcher(this, reply);
         connect(reply, SIGNAL(finished()), this, SLOT(onSearchAlbumFinished()));
    @@ -386,8 +390,10 @@
         album->setMbAlbumId(mbAlbumId);
         album->setMbReleaseGroupId(mbReleaseGroupId);
         album->setAllMusicId("");
    -    QUrl url(QString("http://www.musicbrainz.org/ws/2/release/%1?inc=url-rels+labels+artist-credits").arg(mbAlbumId));
    -    QNetworkReply *reply = qnam()->get(QNetworkRequest(url));
    +    QUrl url(QString("http://musicbrainz.org/ws/2/release/%1?inc=url-rels+labels+artist-credits").arg(mbAlbumId));
    +    QNetworkRequest request(url);
    +    request.setRawHeader("User-Agent" , "MediaElch");
    +    QNetworkReply *reply = qnam()->get(request);
         new NetworkReplyWatcher(this, reply);
         reply->setProperty("storage", Storage::toVariant(reply, album));
         reply->setProperty("infosToLoad", Storage::toVariant(reply, infos));

  • fernandojconti, 16 days ago

    Great! So, how can I apply it?

  • bumblebee, 5 days ago

    wie kann man das einbauen?
     

     

    Das ist jetzt schon so lange bekannt und es noch kein Fix released worden?

    also entweder besteht kein interesse seine Software ordentlich an die Leute zu bringen oder ist das absicht das gespendet wird für nightlies?

     

    finde ich echt sehr traurig