Send of0 packets from workqueue, to avoid recursive locking of ofN device.

Send of0 packets from workqueue, to avoid recursive locking of ofN device.

lockdep reported the following warning, showing that dev_queue_xmit()
for of0 and for another ethernet device (eth0?) were taking locks of
the same class.  It was not a problem in this particular case, because
of0 != eth0, but if a flow were to send a packet back to of0 then we
would have a deadlock.

Solution: use a workqueue to send packets to avoid recursive locking.
We will still waste a lot of CPU time if we get a packet that loops back
to of0.  Solution for that still needed.

=============================================
[ INFO: possible recursive locking detected ]
2.6.26.5 #1
---------------------------------------------
memcheck/1258 is trying to acquire lock:
 (_xmit_ETHER){-+..}, at: [<c025b126>] __qdisc_run+0xa0/0x18a

but task is already holding lock:
 (_xmit_ETHER){-+..}, at: [<c024f3cd>] dev_queue_xmit+0x23c/0x334

other info that might help us debug this:
9 locks held by memcheck/1258:
 #0:  (genl_mutex){--..}, at: [<c0262037>] genl_rcv+0x12/0x2b
 #1:  (dp_mutex){--..}, at: [<c88e9a13>] dp_genl_openflow+0x58/0x91 [openflow_mo
d]
 #2:  (rcu_read_lock){..--}, at: [<c024e749>] netif_receive_skb+0x9e/0x328
 #3:  (rcu_read_lock){..--}, at: [<c0267f77>] ip_local_deliver_finish+0x2a/0x1d7
 #4:  (slock-AF_INET/1){-+..}, at: [<c027f1b8>] tcp_v4_rcv+0x29f/0x5ef
 #5:  (rcu_read_lock){..--}, at: [<c024f308>] dev_queue_xmit+0x177/0x334
 #6:  (_xmit_ETHER){-+..}, at: [<c024f3cd>] dev_queue_xmit+0x23c/0x334
 #7:  (rcu_read_lock){..--}, at: [<c88ea214>] dp_dev_xmit+0x0/0x6c [openflow_mod
]
 #8:  (rcu_read_lock){..--}, at: [<c024f308>] dev_queue_xmit+0x177/0x334

stack backtrace:
Pid: 1258, comm: memcheck Not tainted 2.6.26.5 #1
 [<c02c11cc>] ? printk+0xf/0x13
 [<c01368c5>] __lock_acquire+0x8c2/0xbe4
 [<c0134d35>] ? add_lock_to_list+0x64/0x8a
 [<c0107a78>] ? native_sched_clock+0x82/0x94
 [<c0136c3e>] lock_acquire+0x57/0x73
 [<c025b126>] ? __qdisc_run+0xa0/0x18a
 [<c02c3919>] _spin_lock+0x1c/0x49
 [<c025b126>] ? __qdisc_run+0xa0/0x18a
 [<c025b126>] __qdisc_run+0xa0/0x18a
 [<c024f382>] dev_queue_xmit+0x1f1/0x334
 [<c88e9cde>] xmit_skb+0x5b/0x65 [openflow_mod]
 [<c88e9e60>] dp_output_port+0x178/0x1ae [openflow_mod]
 [<c88eaf91>] do_output+0x2a/0x4c [openflow_mod]
 [<c88eb11f>] execute_actions+0x16c/0x198 [openflow_mod]
 [<c88eb622>] run_flow_through_tables+0xe9/0xf6 [openflow_mod]
 [<c88eb63e>] fwd_port_input+0xf/0x3d [openflow_mod]
 [<c88ea260>] dp_dev_xmit+0x4c/0x6c [openflow_mod]
 [<c88ea214>] ? dp_dev_xmit+0x0/0x6c [openflow_mod]
 [<c024f042>] dev_hard_start_xmit+0x20f/0x276
 [<c024f3e2>] dev_queue_xmit+0x251/0x334
 [<c026bfcb>] ip_finish_output+0x1ea/0x222
 [<c026c081>] ip_output+0x7e/0x83
 [<c026b37a>] ip_local_out+0x18/0x1b
 [<c026baa2>] ip_queue_xmit+0x288/0x2c9
 [<c0136a0c>] ? __lock_acquire+0xa09/0xbe4
 [<c0111264>] ? kernel_map_pages+0xfc/0x113
 [<c027ddad>] ? tcp_v4_send_check+0x80/0xba
 [<c027a281>] tcp_transmit_skb+0x695/0x6cd
 [<c016060a>] ? __kmalloc_track_caller+0xee/0x12a
 [<c024aacc>] ? __alloc_skb+0x51/0xff
 [<c027a43c>] tcp_send_ack+0xdf/0xe7
 [<c02781e4>] tcp_rcv_state_process+0x389/0xc33
 [<c027eebd>] tcp_v4_do_rcv+0x3bd/0x419
 [<c027f2c7>] tcp_v4_rcv+0x3ae/0x5ef
 [<c026805f>] ip_local_deliver_finish+0x112/0x1d7
 [<c0268185>] ip_local_deliver+0x61/0x6a
 [<c0267d14>] ip_rcv_finish+0x2a4/0x2c3
 [<c0267f23>] ip_rcv+0x1f0/0x21a
 [<c024e990>] netif_receive_skb+0x2e5/0x328
 [<c024ea52>] process_backlog+0x7f/0xca
 [<c024d867>] net_rx_action+0x72/0x127
 [<c0120ea1>] __do_softirq+0x7b/0xf2
 [<c0105a5d>] do_softirq+0x66/0xb3
 [<c024ec69>] netif_rx_ni+0x29/0x2e
 [<c88ea2cd>] dp_dev_recv+0x4d/0x6c [openflow_mod]
 [<c88e9e06>] dp_output_port+0x11e/0x1ae [openflow_mod]
 [<c88eaf91>] do_output+0x2a/0x4c [openflow_mod]
 [<c88eb11f>] execute_actions+0x16c/0x198 [openflow_mod]
 [<c88eb312>] recv_flow+0x1c7/0x2a5 [openflow_mod]
 [<c88eb14b>] ? recv_flow+0x0/0x2a5 [openflow_mod]
 [<c88eab37>] fwd_control_input+0x53/0x60 [openflow_mod]
 [<c88e9a27>] dp_genl_openflow+0x6c/0x91 [openflow_mod]
 [<c02621ce>] genl_rcv_msg+0x17e/0x198
 [<c0262050>] ? genl_rcv_msg+0x0/0x198
 [<c02614c6>] netlink_rcv_skb+0x30/0x76
 [<c0262043>] genl_rcv+0x1e/0x2b
 [<c0261053>] netlink_unicast+0x1a9/0x20f
 [<c02612dc>] netlink_sendmsg+0x223/0x230
 [<c0245384>] sock_sendmsg+0xca/0xe1
 [<c012c501>] ? autoremove_wake_function+0x0/0x33
 [<c015f6ce>] ? cache_free_debugcheck+0x2a3/0x2be
 [<c0107a78>] ? native_sched_clock+0x82/0x94
 [<c0134382>] ? lock_release_holdtime+0x1a/0x115
 [<c01f52b7>] ? copy_from_user+0x34/0x11b
 [<c024b6a5>] ? verify_iovec+0x40/0x6f
 [<c02454da>] sys_sendmsg+0x13f/0x192
 [<c0168d51>] ? pipe_write+0x434/0x43f
 [<c0107a78>] ? native_sched_clock+0x82/0x94
 [<c0107a78>] ? native_sched_clock+0x82/0x94
 [<c0134382>] ? lock_release_holdtime+0x1a/0x115
 [<c0107a78>] ? native_sched_clock+0x82/0x94
 [<c0134382>] ? lock_release_holdtime+0x1a/0x115
 [<c0246189>] sys_socketcall+0x14e/0x169
 [<c0102d43>] ? restore_nocheck+0x12/0x15
 [<c0102ce2>] syscall_call+0x7/0xb
 =======================