Data Miner System Update

Dear Data Miner System user,

We would like to inform you about a number of chages in our Data Miner System and API.

First, effective tonight, Twitter is enforcing new rules they announced in August 2012 (https://dev.twitter.com/blog/changes-coming-to-twitter-api) and introducing changes to their policies and data limits.

We would like to let you know that we have prepared and tested our systems to comply fully with new rules posited by Twitter. For you, as the user of our Data Miner System, nothing changes in the API interfaces
you use to communicate with our systems. Your keyword tracking on Twitter should not experience any problems or delays.

It is however possible that the volume of messages obtained though tracking will increase as a consequence of Twitter allowing more relaxed limits for the type of data use Data Miner System utilizes.
Please consider this fact when making reports based on frequencies of tweets which you have tracked through the system.

If you have any questions regarding this announcement, or if you experience any inconveniences, please contact us - we appreciate the feedback.

Second, the currently available documentation (http://ai-applied.nl/api-documentation/2013/1/17/data-miner-api-documentation) will soon be deprecated for the case of requesting data from our API. We have made a number of changes for the new way of interfacing though the Data miner API, consisting of a new, sparse, and more consistent format for the replies from the API. For example, until now, if you would request "sentiment_class/gender", the API would return time chunk results looking like:

{"timestamp_utimestamp": 1357603200, "summary": {"positive": {"unknown": 1038, "total": 24999, "male": 20839, "female": 3122}, "total": 45696, "negative": {"unknown": 1634, "total": 19026, "male": 15304, "female": 2088}, "unknown": {"unknown": 1671, "total": 1671}}}

where every node but the last would have a sub-dictionary as it's value, and a "total" parameter representing the count (e.g. for "positive":

"positive": {"unknown": 1038, "total": 24999, "male": 20839, "female": 3122}

), while the terminating node would only have a count parameter such as "male": 20839.

This is inconsistent, as there is a clear difference between the way intermediate counts are being returned, and the way the final counts are being returned. We have decided to change this and make it fully consistent, by always including a sub-dictionary as value for every parameter, and within that sub-dictionary always including a "total" parameter representing the number of results. In the new representation, which is also sparse (so, always returns the 0 values), the same result would look like this:

{"timestamp_utimestamp": 1357603200, "summary": {"positive": {"unknown": {"total": 1038}, "total": 24999, "male": {"total": 20839}, "female": {"total": 3122}}, "neutral": {"unknown": {"total": 0}, "total": 0, "male": {"total": 0}, "female": {"total": 0}}, "total": 45696, "negative": {"unknown": {"total": 1634}, "total": 19026, "male": {"total": 15304}, "female": {"total": 2088}}, "unknown": {"unknown": {"total": 1671}, "total": 1671, "male": {"total": 0}, "female": {"total": 0}}}}

with the representation of "positive" changed to:

"positive": {"unknown": {"total": 1038}, "total": 24999, "male": {"total": 20839}, "female": {"total": 3122}}

You can test the new interface by adding the parameter "interface": 1 to your API call, e.g. for the above example:

http://api.ai-applied.nl/data_miner/data_summary/?request={"data":{"api_key":"977a0b738d36942eb510b41c17cc7f4be1e019a6","user":"DEMO","call":{"interface":1,"from":1357662237,"to":1358267037,"keyword":"big data","statistics_path":"sentiment_class/gender","resolution":86400}}}

On March the 20th 2013, the new interface will become standard, so please make sure to make your API implementations compliant with it until that date.

If you have any questions, please don't hesitate to ask.

Soon, the "related key phrases" functionality will also be integrated into this System and API as well.

Best regards,

Ai Applied Team