Yahoo! has finally given up with Y! Pipes, as of August 30th 2015, users will no longer be able to create new Pipes. The Pipes team will keep the infrastructure running until end of September 30th 2015 in a read-only mode (Y! Pipes End-of-Life Announcement). As a consequence, I was looking for an alternative (snif). For my demo, I needed a hosted service (with a free plan for a first try) that can aggregate different RSS feeds and return JSON formatted data as streamdata.io only accepts JSON in input. After reading some blog posts listing the different alternatives, I finally picked out Kimono Labs (see here) that seemed to fit my needs. In two words, KimonoLabs is a hosted net scrapping service that helps you turn websites into APIs. It is not exactly the same approach than Yahoo Pipes! that was able to ingest RSS feed directly but still worth a try.
Now, lets see how we can use this service together with streamdata.io to receive realtime updates from an aggregated feed in four simple steps:
First, create an account on KimonoLabs and download their Bookmarklet to add an extension to your browser in order to help you map the UI elements to data properties.
Once settled, navigate to your favorite websites and start mapping UI elements to data properties. You can follow this interactive tutorial if you need more help, but don’t worry it’s pretty simple. Once done with the mapping, click Done button in the Bookmarklet, this will add an API in your KimonoLab Portal account. Repeat this step for different websites to create different APIs.
Once you have created several APIs, make sure they are present in your KimonoLabs Portal account.
You can list them by clicking « My Apis » under your account name.
To aggregate content of different APIs, click « Combine APIs » under your account name.
Then, choose a name for your new « combined » API and select which APIs to combine.
Finally, click Next and you are good to go.
Once your combined API is created, you can visualize a preview of your aggregated content and choose the type of endpoint you want to use (JSON, csv or RSS).
The JSON endpoint can now be used to receive realtime updates from streamdata.io.
Streamdata.io will poll KimonoLabs URL and send you patches of data as soon as new entry is available in one of the aggregated feeds.
Run either the Angular or JQuery application by launching the index.html in your favorite browser.
Use the JSON endpoint provided by KimonoLabs as URL and your streamdata.io App token and click Connect.
You should now receive a first snapshot of your aggregated content followed by JSON patch updates whenever a change is happening.