Placing orders in the native python IB API

This the fourth in a series of posts on using the native python API for interactive brokers. You should read the first, second, and third, before this one.

It is an updated version of this older post, which used a third party API (swigibpy) which wraps around the C++ API. I've changed the code, but otherwise the post is pretty similar.

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.

Where do I start?

It's worth reading the relevant part of the documentation.

You need to get the code from the following gist.

If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account (password: demo123), but the results might not be reliable. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.

WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.

Contract details - what is it?

As in the previous post there is the tedious business of creating an object to talk to IB and resolving the contract we want to trade:

app = TestApp("127.0.0.1", 4001, 1)

ibcontract = IBcontract()
ibcontract.secType = "FUT"

ibcontract.lastTradeDateOrContractMonth="201812"

ibcontract.symbol="GE"

ibcontract.exchange="GLOBEX"
## resolve the contract

resolved_ibcontract = app.resolve_ib_contract(ibcontract)

Getting full contract details from the server... 

For the rest of the post anything you see in that nice purple bold font is python output.

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format. 

ANOTHER WARNING; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.

Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. We build the order object and pass it a client function.

order1=Order()
order1.action="BUY"

order1.orderType="MKT"

order1.totalQuantity=10

order1.transmit = True
orderid1 = app.place_new_IB_order(ibcontract, order1, orderid=None)

print("Placed market order, orderid is %d" % orderid1)

Getting orderid from IB

Using order id of 1

Placed market order, orderid is 1

In the client class:

def place_new_IB_order(self, ibcontract, order, orderid=None):

    ## We can eithier supply our own ID or ask IB to give us the next valid one    

    if orderid is None:
        print("Getting orderid from IB")
        orderid = self.get_next_brokerorderid()

        if orderid is TIME_OUT:
            raise Exception("I couldn't get an orderid from IB, and you didn't provide an orderid")

    print("Using order id of %d" % orderid)

    ## Note: It's possible if you have multiple traidng instances for orderids to be submitted out of sequence    

    ##   in which case IB will break
    # Place the order    

    self.placeOrder(
        orderid,  # orderId,

        ibcontract,  # contract, 

        order  # order    )

return orderid   

Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.

Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties:

• A fill has arrived when a trade actually executes, in which case the reqId will be -1 (stored as FILL_CODE here)
• We can ask for fill information, in which case the reqId will accompany the information. More on this shortly

def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    

    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    

   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)

In this case as we haven't yet asked for execution data reqId will be -1, and we'll put the data into my_executions_stream. This is a 'permanent' queue that hosts execution data: I describe it as permanent because we didn't need to run an init_ ... method to set it up in place_new_IB_order like I've shown you in previous posts; the queue is created when the instance of TestWrapper is created.

(Also this would be the function that would update your order status database; as a pro at this stuff, naturally you would have such a thing).

To see the fill data I need to do this:

print("Recent fills")
filldetails = app.recent_fills_and_commissions()
print(filldetails)

There is quite a lot going on under the hood here which is worth understanding so lets examine the client function we're calling.

def recent_fills_and_commissions(self):
    """    Return recent fills, with commissions added in
    :return: dict of execInformation objects, keys are execids    

    """
    recent_fills = self._recent_fills()
    commissions = self._all_commissions() ## we want all commissions
    

    ## glue them together, create a dict, remove duplicates    

    all_data = recent_fills.blended_dict(commissions)

    return all_data

The first thing to note is that this returns information coming from two sources: execDetails (labelled recent_fills here) and commissions. Frustratingly this data is provided by two separate parts of the wrapper function. We've already seen the executions, here it is for commissions:

def commissionReport(self, commreport):

    commdata = execInformation(commreport.execId, Commission=commreport.commission,
                    commission_currency = commreport.currency,
                    realisedpnl = commreport.realizedPNL)


    ## there are some other things in commreport you could add    

    ## make sure you add them to the .attributes() field of the execInformation class
    ## These always go into the 'stream' as could be from a request, 

    ## or a fill thats just happened    

    self._my_commission_stream.put(commdata)

As with execution details this can get called eithier when we've requested execution details (of which more later), or whenever we get an actual fill (as we're doing here). Unlike the execDetails function however we never get a reqId. So all the commissions data is dumped into a 'permanent' _stream queue, rather than one we've initialised already.

Once the executions and commission data has arrived in the 'permanent' _stream queues, we can access it via these functions:

def _recent_fills(self):
    """    Returns any fills since we last called recent_fills
    :return: list of executions as execInformation objects    

    """
    ## we don't set up a queue but access the permanent one    

    fill_queue = self.access_executions_stream()

    list_of_fills=list_of_execInformation()

    while not fill_queue.empty():
        MAX_WAIT_SECONDS = 5        

        try:
            next_fill = fill_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_fills.append(next_fill)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            

            pass
    ## note this could include duplicates and is a list    

    return list_of_fills

Notice that we clear the queue once we've accessed the data. And _recent_commissions(self) is in a similar vein.

def _recent_commissions(self):

    ## we don't set up a queue, as there is a permanent one    

    comm_queue = self.access_commission_stream()

    list_of_comm=list_of_execInformation()

    while not comm_queue.empty():
        MAX_WAIT_SECONDS = 5        

        try:
            next_comm = comm_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_comm.append(next_comm)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            pass
    ## note this could include duplicates and is a list    

return list_of_comm

However because all commissions live in the commission_stream queue I actually use another function to get commission data, which returns both the last chunk of commissions, plus any other commission data I've collected:

def _all_commissions(self):

    ## self._commissions is created when the client instance is __init__    

    original_commissions = self._commissions

    latest_commissions = self._recent_commissions()

    ## these are just simple lists so we can glue them together
    all_commissions = list_of_execInformation(original_commissions + latest_commissions)

    self._commissions = all_commissions

    # note this could include duplicates and is a list    

    return all_commissions

Finally we have to glue these together. The final line in the function before the return populates the fill data with the relevant commission levels.

def recent_fills_and_commissions(self):    

    recent_fills = self._recent_fills()

    commissions = self._all_commissions() ## we want all commissions
    

    ## glue them together, create a dict, remove duplicates    

    all_data = recent_fills.blended_dict(commissions)

    return all_data

Recent fills

{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}

Notice this returns a dict; the keyword is the execId. I use some simple objects to gather up and merge together these two streams of data but you can do it differently of course.

Just to reiterate the _recent functions clear the relevant queues as you can see when I try and get the fill data again (although the commission data is saved so self._all_commissions() will still work).

## when I call again should be empty as we've cleared the memory of recent fills
print("Recent fills (should be blank)")
morefilldetails = app.recent_fills_and_commissions()
print(morefilldetails)

Recent fills (should be blank)

{}

IMPORTANT DETAIL: It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

• 3 lots @ 100.0
• 6 lots @ 100.2
• 1 lot @ 100.5

 Then the fills that come in will look like this:

• qty: 3 lots, price=100.0
• qty: 9 lots, price=100.13333333
• qty: 10 lots, price=100.17

So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.

Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.



This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.



Here is how we do it.


print("Executions today")
execdetails = app.get_executions_and_commissions()
print(execdetails)
def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    ## store somewhere    
    execution_queue = finishableQueue(self.init_requested_execution_data(reqId))

    ## We can change ExecutionFilter to subset different orders    
    ## note this will also pull in commissions 
    self.reqExecutions(reqId, execution_filter)

    ## Run until we get a terimination or get bored waiting    
    MAX_WAIT_SECONDS = 10    
    exec_list = list_of_execInformation(execution_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if execution_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting exec / commissions")
 
<snip>

(Note: We can change ExecutionFilter to subset different orders)     
This is a more familiar pattern: we create a queue for the wrapper to put stuff which will be terminated by a finish event, and then request the IB API to start sending data, some method in the wrapper populates the queue, and we then pull the contents of the queue in. The relevant wrapper function should look familiar:
 

def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    
 
   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)

Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to the requested queue for a given reqId.

The only wrinkle is that the commission data needs to be brought in and merged, as we did before with fills:


def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    <snip>
    ## Commissions will arrive seperately. We get all of them, but will only use those relevant for us    
    commissions = self._all_commissions()

    ## glue them together, create a dict, remove duplicates    
    all_data = exec_list.blended_dict(commissions)

    return all_data



   
execdetails = app.get_executions_and_commissions()
print(execdetails)

Executions today

{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}


As before the executions are listed in a dict with the execId as the key.

Placing limit orders

No self respecting trader will use market orders (see my post), so how to define limit orders?


order2=Order()
order2.action="SELL"
order2.orderType="LMT"
order2.totalQuantity=12
order2.lmtPrice = 100.0
order2.tif = 'DAY'
order2.transmit = True

orderid2 = app.place_new_IB_order(ibcontract, order2, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

Getting orderid from IB
Using order id of 2
Placed limit order, orderid is 2

This is just a tiny selection of the available orders, see the docs here.

 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return unfilled limit orders like the one we've just submitted.

open_orders = app.get_open_orders()
print(open_orders)

Client function:

def get_open_orders(self):

    ## store the orders somewhere    

    open_orders_queue = finishableQueue(self.init_open_orders())

    ## You may prefer to use reqOpenOrders() which only retrieves orders for this client    self.reqAllOpenOrders()

    ## Run until we get a terimination or get bored waiting    

    MAX_WAIT_SECONDS = 5    

    open_orders_list = list_of_orderInformation(open_orders_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if open_orders_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting orders")

    ## open orders queue will be a jumble of order details, turn into a tidy dict with no duplicates    open_orders_dict = open_orders_list.merged_dict()

    return open_orders_dict

Wrapper functions:

def init_open_orders(self):
    open_orders_queue = self._my_open_orders = queue.Queue()

    return open_orders_queue


def orderStatus(self, orderId, status, filled, remaining, avgFillPrice, permid,
                parentId, lastFillPrice, clientId, whyHeld):

    order_details = orderInformation(orderId, status=status, filled=filled,
             avgFillPrice=avgFillPrice, permid=permid,
             parentId=parentId, lastFillPrice=lastFillPrice, clientId=clientId,
                                     whyHeld=whyHeld)

    self._my_open_orders.put(order_details)


def openOrder(self, orderId, contract, order, orderstate):

    order_details = orderInformation(orderId, contract=contract, order=order, 

                                     orderstate = orderstate)
    self._my_open_orders.put(order_details)


def openOrderEnd(self):

    self._my_open_orders.put(FINISHED)

{2: Order - contract: 56825063,GE,FUT,20181217,0.0,?,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: order: 2,1,1017786930: LMT SELL 12@100.000000 DAY orderstate: <ibapi.order_state.OrderState object at 0x7f5e7bcfe860> status: Submitted filled: 0.0 avgFillPrice: 0.0 permid: 1017786930 parentId: 0 lastFillPrice: 0.0 clientId: 1 whyHeld: }

This should be a familiar story now. The only interesting thing is that we get order details from two wrapper functions, orderStatus and openOrder. I don't bother segregating them, instead I add them to the same queue, and then merge them with open_orders_dict = open_orders_list.merged_dict()

In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway).

As with fills the output is returned as a dict, and the keys are orderids. We just have the single active limit order since the market order (orderid 1) has long since been filled.

Order modification

To modify an existing order we submit a new order, but with an existing orderid.

order3=Order()
order3.action="BUY"

order3.orderType="LMT"

order3.totalQuantity=5

order3.lmtPrice = 10.0

order3.tif = 'DAY'

order3.transmit = True

orderid3 = app.place_new_IB_order(ibcontract, order3, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

print("Open orders (should be two)")
open_orders = app.get_open_orders()
print(open_orders.keys())

Getting orderid from IB

Using order id of 3

Placed limit order, orderid is 2

Open orders (should be two)

dict_keys([2, 3])

print("Modifying order %d" % orderid3)

order3.lmtPrice = 15.0
print("Limit price was %f will become %f" % (open_orders[orderid3].order.lmtPrice, order3.lmtPrice ))

app.place_new_IB_order(ibcontract, order3, orderid=orderid3)
time.sleep(5)
open_orders = app.get_open_orders()
print("New limit price %f " % open_orders[orderid3].order.lmtPrice)

Modifying order 3

Limit price was 10.000000 will become 15.000000

New limit price 15.000000 

Its advised that you only change the quantity and limit price (where relevant) of an order; not the order type.

Order cancelling - what if I don't want it any more?

Cancelling an order is pretty simple:

print("Cancel order %d " % orderid2)
app.cancel_order(orderid2)
open_orders = app.get_open_orders()

Cancel order 2 

IB error id 2 errorcode 202 string Order Canceled - reason:

My code checks that the order has been cancelled:

def cancel_order(self, orderid):
    self.cancelOrder(orderid)

    ## Wait until order is cancelled    start_time=datetime.datetime.now()
    MAX_WAIT_TIME_SECONDS = 10
    finished = False
    while not finished:
        if orderid not in self.get_open_orders():
            ## finally cancelled            
            finished = True
        if (datetime.datetime.now() - start_time).seconds > MAX_WAIT_TIME_SECONDS:
            print("Wrapper didn't come back with confirmation that order was cancelled!")
            finished = True
    ## return nothing
... but you can also check yourself:

print("Open orders (should just be %d)" % orderid3)
print(open_orders.keys())

Open orders (should just be 3)

dict_keys([3])

Note this will only cancel orders where the clientid (the third number in this call app = TestApp("127.0.0.1", 4001, 1)) is the same as the current client. To cancel all orders, regardless of the client:

print("Cancelling all orders")
app.cancel_all_orders()

print("Any open orders? - should be False")
print(app.any_open_orders())

Cancelling all orders

IB error id 3 errorcode 202 string Order Canceled - reason:

Open orders? - should be False

False

Are we finished?

 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html

http://qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html

The next, and final, post is:
http://qoppac.blogspot.co.uk/2017/03/getting-position-and-accounting-data.html