1. Home
  2. Categories
  3. General Discussion
  4. Probable inconsistency in the streaming data

Probable inconsistency in the streaming data

General Discussion
Log in to reply
  • T

    In order to to utilise streaming data for any kind of index algorithm the data should stream in a fixed fashion

    I have noticed some inconsistency in the streaming data which i am posting here

    [{"tvalue":"16/12/2020 11:22:25","name":"tm"},{"e":"nse_cm","name":"sf","ltp":"1977.00","ltq":"11","tk":"2885","ltt":"NA"}]
[{"to":"7306167033.00","lo":"1970.05","e":"nse_cm","sp":"1977.10","c":"1974.35","ltp":"1977.00","ltq":"23","tk":"2885","bs":"7","ltt":"16/12/2020 11:22:25","tsq":"971806","v":"3687525","bp":"1977.00","cng":"02.65","bq":"787","ap":"1981.32","nc":"00.13","name":"sf","tbq":"495602"}]

[{"tvalue":"16/12/2020 11:22:26","name":"tm"},{"e":"nse_cm","name":"sf","ltp":"1977.00","ltq":"25","tk":"2885","ltt":"NA"}]
[{"to":"7306315632.00","lo":"1970.05","e":"nse_cm","sp":"1977.10","c":"1974.35","ltp":"1977.00","ltq":"32","tk":"2885","bs":"7","ltt":"16/12/2020 11:22:26","tsq":"972439","v":"3687600","bp":"1977.00","cng":"02.65","bq":"751","ap":"1981.32","nc":"00.13","name":"sf","tbq":"495582"}]

    I hope you have noticed the pattern of response first line of one stream always starts with tvalue and second line with to

    but after some time randomly here and there this is the response

    [{"tvalue":"16/12/2020 11:22:36","name":"tm"},{"e":"nse_cm","name":"sf","ltp":"1976.65","ltq":"1","tk":"2885","ltt":"NA"}]

[{"tvalue":"16/12/2020 11:22:37","name":"tm"},{"to":"7316797530.79","lo":"1970.05","e":"nse_cm","sp":"1977.00","c":"1974.35","ltp":"1976.65","ltq":"1","tk":"2885","bs":"1019","ltt":"16/12/2020 11:22:36","tsq":"976276","v":"3692909","bp":"1976.65","cng":"02.30","bq":"32","ap":"1981.31","nc":"00.12","name":"sf","tbq":"492910"}]

[{"e":"nse_cm","name":"sf","ltp":"1976.65","ltq":"1","tk":"2885","ltt":"NA"}]

[{"to":"7316811399.96","lo":"1970.05","e":"nse_cm","sp":"1977.00","c":"1974.35","ltp":"1976.65","ltq":"5","tk":"2885","bs":"1190","ltt":"16/12/2020 11:22:37","tsq":"976505","v":"3692916","bp":"1976.65","cng":"02.30","bq":"25","ap":"1981.31","nc":"00.12","name":"sf","tbq":"493128"}]


[{"to":"7316813381.27","lo":"1970.05","e":"nse_cm","sp":"1977.00","c":"1974.35","ltp":"1976.65","ltq":"1","tk":"2885","bs":"977","ltt":"16/12/2020 11:22:38","tsq":"976246","v":"3692917","bp":"1976.65","cng":"02.30","bq":"24","ap":"1981.31","nc":"00.12","name":"sf","tbq":"493058"}]

    Both the tValue and to lines get mixed up with each other

    P.S : This question seems confusing, will be happy to discuss this over a call

    0 R 1 Reply
  • R

    @taleevaalam1 You can try any commodity like copper, lead, zinc or any currency derivatives. They don't move that much.

    0
  • T

    @rajanprabu I don't think we should have any kind of limitations due to the liquidity of the stocks, whatever the volatility of the stock it should be treated in a same manner.

    @rjbanna we should see the tick response for any non-volatile stock and then determine how the response should be

    Can you guys share any token id for some nonvolatile stock

    0 R 1 Reply
  • R

    @rajanprabu Yes I know, no one would choose illiquid stocks for algos but I just suggested you how you can overcome this issue for this API. As I mentioned earlier, I totally agree that every tick should get the timestamp and I also request developers to do the same. You can wait for the fix if they are going to do this or else you can use workaround.

    0
  • R

    @rjbanna Thanks for your views. Algo trading is done on scrips which has lots of liquidity/volume. I would assume no algo trader would touch illiquid scrip that has no trades for 10 minutes. so Its mostly index and stocks listed under FnO. Algorithms are backtested using data with true timestamp from an authorised vendor. All brokers subscribe to 1 sec snapshot from NSE and relay it to the users as it is. I am using one international broker ( for US/Europe Markets ) and one large Indian broker for the same. I have always used timestamp of the ticks for making my candles.

    0 R T 2 Replies
  • R

    @dorje I have to check. I'm using the tickdata with 4 scrips and I'm receiving quotes for all of them. I'll add more to it and let you know the outcome.

    0
  • R

    @rajanprabu Yes it would be helpful if we receive timestamp in every tick, but I don't think so it is necessary. Because if there's no tickdata from the exchange, then you won't be able to confirm whether you should complete one candle formation or not. There are many scrips in which you won't get tickdata if there is no buyer/seller. It might happen that you won't get tickdata even after 10 minutes. In this case if you are using M5 timeframe, you won't be able to know when to close your candle and this will raise many issues. So I would suggest you not to use tickdata timestamps. Create your own timer function for it.

    0 R 1 Reply
  • A

    @dorje said in Probable inconsistency in the streaming data:

    hi bhavesh this is the issue i was speaking to you during conversation held with you, day before yesterday.

    Noted. I have asked my development team to work on the feedback shared on this post. We will keep you posted.

    0
  • T

    @dorje Will have to check that , you can run more than one program and divide the number of scrips among them

    i.e : Run 5 programs with 10 scrips each

    0
  • D

    @taleevaalam1 @bhaveshjain hi bhavesh this is the issue i was speaking to you during conversation held with you, day before yesterday.

    also @taleevaalam1 @rjbanna i need to ask you how many scripts data you can stream at a time cause i have subscribed for 50 scripts but i'm able to get data of only few scripts.

    0 T A 2 Replies
  • T

    @rjbanna agreed with you we can use ltp of tvalue response to build the candle

    @admin @bhaveshjain Will it not be possible to just give us the following line as tick response continuously

    [{"to":"7128494980.08","lo":"1970.05","e":"nse_cm","sp":"1977.05","c":"1974.35","ltp":"1977.00","ltq":"47","tk":"2885","bs":"6","ltt":"16/12/2020 11:10:44","tsq":"955550","v":"3597688","bp":"1977.00","cng":"02.65","bq":"53","ap":"1981.41","nc":"00.13","name":"sf","tbq":"468907"}]

    and include the timeframe also in this response and get rid of the following data

    [{"tvalue":"16/12/2020 11:10:44","name":"tm"},{"e":"nse_cm","name":"sf","ltp":"1977.05","ltq":"40","tk":"2885","ltt":"NA"}]

    It would make us developers life easier

    If we want we can make do with the existing format as well but i don't want to work too hard on rearranging the data that i got from the API , the data should come in clean and consistent way

    0 D 1 Reply
  • R

    @rjbanna Thank for the explanations. Very much appreciated.

    Main motivation for me to switch to AB API was to get good quality ticks and not to miss ticks as it happened often with other big broker in India. If I am expected to use ticks with timestamp every 3-4 seconds, I need to switch back. I also dont understand why cant time be included in every tick that's it transmitted. Isn't that a basic requirement for a API ?

    0 T 1 Reply
  • R

    @rajanprabu If I were to make candle, I would use tvalue as it gets updated at every 1 or 2 seconds. You can create logic that compares the tvalue with your desired timeframe. ltt is last traded time, so it will only get updated if there's any price change. If there's no price update from the exchange for lets say 10 seconds, ltt won't update but tvalue will be updated to the recent time. So it might happen that values don't match.

    Now your second question. Yes sometimes you do get tvalue at the end of the response. That's why it is important not to hard code index places in your code. You should always check for the values, not the index, because tvalue will be the unique key in particular response. You can always loop through list and check for the tvalue key in the dict. By doing this way, you will always get the right value irrespective of the order.

    0 R 1 Reply
  • R

    @rjbanna Thanks for your answer. To parse it and make a candle out of it which timestamp do you use ? According to API documentation ltt is last updated time.. In some ticks value and ltt and tvalue are not matching ( whenever tvalue is available).. so wondering which one to use ?

    Another question is that

    {"tvalue":"16\/12\/2020 22:48:51","name":"tm"},{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5761","v":"256441","bp":"65838.00","cng":"1014.00","bq":"4","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8733"}]
[{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5761","v":"256441","bp":"65838.00","cng":"1014.00","bq":"4","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8730"}]
[{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5761","v":"256441","bp":"65839.00","cng":"1014.00","bq":"4","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8734"}]
[{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5762","v":"256441","bp":"65839.00","cng":"1014.00","bq":"4","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8734"}]
[{"tvalue":"16\/12\/2020 22:48:52","name":"tm"},{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5761","v":"256441","bp":"65839.00","cng":"1014.00","bq":"4","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8784"}]
[{"to":"16900562031.89","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:50","tsq":"5761","v":"256441","bp":"65839.00","cng":"1014.00","bq":"8","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8768"}]
[{"to":"16900759744.76","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:52","tsq":"5760","v":"256444","bp":"65840.00","cng":"1014.00","bq":"2","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8785"}]
[{"tvalue":"16\/12\/2020 22:48:53","name":"tm"},{"to":"16900891553.34","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:52","tsq":"5765","v":"256446","bp":"65838.00","cng":"1014.00","bq":"6","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8764"}]
[{"to":"16900891553.34","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:52","tsq":"5855","v":"256446","bp":"65840.00","cng":"1014.00","bq":"1","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8760"}]
[{"to":"16900891553.34","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:52","tsq":"5855","v":"256446","bp":"65841.00","cng":"1014.00","bq":"5","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8751"}]
[{"to":"16900891553.34","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:52","tsq":"5855","v":"256446","bp":"65845.00","cng":"1014.00","bq":"1","ap":"65904.29","nc":"01.56","name":"sf","tbq":"8751"}]
`[{"to":"16901482127.40","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:54","tsq":"5777","v":"256455","bp":"65841.00","cng":"1014.00","bq":"7","ap":"65904.28","nc":"01.56","name":"sf","tbq":"8777"},{"tvalue":"16\/12\/2020 22:48:54","name":"tm"}]
[{"to":"16901482127.40","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:54","tsq":"5725","v":"256455","bp":"65845.00","cng":"1014.00","bq":"2","ap":"65904.28","nc":"01.56","name":"sf","tbq":"8793"}]
[{"to":"16901482127.40","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65840.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:54","tsq":"5725","v":"256455","bp":"65845.00","cng":"1014.00","bq":"1","ap":"65904.28","nc":"01.56","name":"sf","tbq":"8792"}]
[{"to":"16901548031.68","lo":"64950.00","e":"mcx_fo","sp":"65847.00","c":"64826.00","ltp":"65845.00","ltq":"1","tk":"220582","bs":"1","ltt":"16\/12\/2020 22:48:54","tsq":"5726","v":"256456","bp":"65842.00","cng":"1019.00","bq":"1","ap":"65904.28","nc":"01.57","name":"sf","tbq":"8781"}]

    In the above one can see that in the first tick tvalue was in the beginning and later it was at the end. If one need to constantly parse and make candled with it, it needs to be consistent. Or im making a mistake here.

    I hope my confusion is clear and thanks once again for your time.

    0 R 1 Reply
  • R

    @rajanprabu @taleevaalam1 I ran the API for 15 minutes, but I didn't see any inconsistencies in the response. I always get the tValue and to in the same list or only to. One thing that I notice is, I received tValue at every 3rd response. Below is the response that I got. Let me know if you have any questions for me.

    [{'name': 'tm', 'tvalue': '16/12/2020 21:55:44'}, {'ap': '65904.83', 'bp': '66071.00', 'bq': '13', 'bs': '1', 'c': '64826.00', 'cng': '1253.00', 'e': 'mcx_fo', 'lo': '64950.00', 'ltp': '66079.00', 'ltq': '1', 'ltt': '16/12/2020 21:55:43', 'name': 'sf', 'nc': '01.93', 'sp': '66079.00', 'tbq': '8994', 'tk': '220582', 'to': '16239543255.47', 'tsq': '5336', 'v': '246409'}]

[{'ap': '65904.83', 'bp': '66071.00', 'bq': '13', 'bs': '1', 'c': '64826.00', 'cng': '1253.00', 'e': 'mcx_fo', 'lo': '64950.00', 'ltp': '66079.00', 'ltq': '1', 'ltt': '16/12/2020 21:55:43', 'name': 'sf', 'nc': '01.93', 'sp': '66079.00', 'tbq': '8994', 'tk': '220582', 'to': '16239543255.47', 'tsq': '5351', 'v': '246409'}]

[{'ap': '65904.83', 'bp': '66071.00', 'bq': '13', 'bs': '1', 'c': '64826.00', 'cng': '1253.00', 'e': 'mcx_fo', 'lo': '64950.00', 'ltp': '66079.00', 'ltq': '1', 'ltt': '16/12/2020 21:55:44', 'name': 'sf', 'nc': '01.93', 'sp': '66079.00', 'tbq': '8994', 'tk': '220582', 'to': '16239609160.30', 'tsq': '5370', 'v': '246410'}]

[{'name': 'tm', 'tvalue': '16/12/2020 21:55:45'}, {'ap': '65904.83', 'bp': '66071.00', 'bq': '13', 'bs': '1', 'c': '64826.00', 'cng': '1253.00', 'e': 'mcx_fo', 'lo': '64950.00', 'ltp': '66079.00', 'ltq': '1', 'ltt': '16/12/2020 21:55:44', 'name': 'sf', 'nc': '01.93', 'sp': '66079.00', 'tbq': '8994', 'tk': '220582', 'to': '16239609160.30', 'tsq': '5355', 'v': '246410'}]
    0 R 1 Reply
  • R

    @taleevaalam1 Yes I have the same issue as well. But I wonder how others are using it. @rjbanna any inputs regarding this ?

    0 R 1 Reply