One of the benefits of connection you Exalytics appliance to a ExaData appliance is that you can connect them via infiniband. Both appliances are equipped with infiniband connectors. InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices.
By default a Oracle ExaData (full rack) setup is equipped with 3 infiniband switches, one spine switch and 2 leave switches which are connected a shown below which hardens the setup against the failure of one of the switches. This makes the appliance setup more resilient and enables you to update firmware of your switches without the need for downtime.
When connecting a single Exalytics to your exadata appliance it is common practice to connect to both leave switches. You will have 2 infiniband ports on the back of your Exalytics appliance, those will be used to connect to the spine switches. This helps you to make the connection resilient against the failure of one of the switches.
More information can be found on the slidedeck about this subject shown below.