How to Manage and Monitor the Oracle ZFS Storage Appliance

Oracle storage solutions in the form of ZFS appliances are widely used in oracle focused and none-oracle focused IT footprints. This is for good reason as Oracle provides a good storage solution with the ZFS storage appliance range. I already posted a number of blogposts on ZFS, including on how to analyse the ZFS storage appliance and on how to use the ZFS storage appliance simulator.

The Oracle ZFS storage appliance simulator is a Oracle VirtualBox appliance which can be used to use for testing and learning purposes.

To add to the already mentioned posts is now this post, providing you the below video on how to manage and monitor the Oracle ZFS storage appliance which might provide you additional information when exploring the options of ZFS appliances.

Oracle ZFS storage appliance configuration

Oracle is incorporating its ZFS storage appliances in more and more engineered systems. Even if you are not a pure storage administrator or consultant and more into Oracle software and engineered systems it is good to have some basic understanding of how a ZFS storage appliance is working and what you can potentially do with it to enhance your solution and provide a better performing and maintainable solution.

The issue with hardware based solutions commonly is that you cannot just play with it without ordering the device. This is holding a lot of people back from gaining experience before they are getting involved in a project where this specific hardware solution is used. The Oracle ZFS storage appliance is bit different, reason for this is that Oracle has decided to create a virtual appliance you can use to play with the solution. The virtual appliance provides you all the options to test and work with the storage appliance in a Oracle VirtualBox image in the same manner as you would do when you would have purchased the real physical hardware.

The virtual Oracle ZFS storage appliance can be downloaded from the Oracle site. After unpacking and importing it into Oracle Virtualbox you will be up and running in  matter of minutes. One thing is good to keep in mind, this is a system to play around with, it is not intended to be used in any serious solution except playing and testing. When the initial boot has been completed you will notice that the welcome screen of the host informs you where you can point your browser to.

A minimal setup is done during the initial boot process, the full configuration and setup will be done via the browser. This is exactly the same manner as you would do when you use the real physical ZFS appliance in your datacenter.The primary things you need to completed are during the inital setup are:

• Host Name
• DNS Domain
• Default Router
• DNS server
• Password

After completing those steps you will be pointed to a https://{ip}:215 address which will be the main URL for maintaining the ZFS storage appliance, or rather the ZFS storage appliance simulator in this case.

Oracle ZFS configuration STEP 1:
Before we can configure the machine you will have to login, for this you can use the root account in combination with the password you entered during the initial CLI configuration.

After login you will be shown the shown the welcome screen which again tells you that this is only to be used for demonstration purposes. You can use this also for some extreme small testing however, remember this system is not a solution for a storage need and just to play with.

Oracle ZFS configuration STEP 2:
The next step is to ensure you have all the correct networking in place to be able to use our ZFS appliance in the right manner within your corporate infrastructure.

As you can see from the above screenshot there is a datalink and an interface already configured however still stating "untitled" which is giving a hint that you need to do some configuration to it before it will become usable. By clicking the pencil icon you can edit the details of both the datalinks and the interfaces as shown below.

After configuring a the ZFS storage appliance interfaces and datalinks you will be asked to configure the routing tables, DNS and NTP.

By having done this the pure network configuration steps are done. Optional you can now select a manner on how you will embed the new storage into your corporate authentication and authorization solution. You can solutions like; NIS, LDAP or an active directory solution you might already have in place within your corporate IT infrastructure.

More information on how to connect a new ZFS appliance to an already existing Microsoft Active Directory can be found in the Oracle documentation.

Oracle ZFS configuration STEP 3:
In step 3 the actual storage configuration will be done. Here you will have to select how you will use the disks and what type of data profile you will be using. All previous steps are more concerning how you will fit the appliance in your existing IT infrastructure. Those steps are concerning how you will actually configure and use the appliance on a storage level. It is advisable to have given this some thorough thoughts before you do the actual implementation of the appliance.

The first decision you will have to make is to decide how many storage pools your device will have (initially).

During this implementation we will only be using a single storage pool. The next important decision that needs to be made is what kind of storage profile you will be using within your pool or pools. You can have different storage  profiles per pool. The following strorage profiles are available:

Double parity
RAID in which each stripe contains two parity disks. This yields high capacity and high availability, as data remains available even with the failure of any two disks. The capacity and availability come at some cost to performance: parity needs to be calculated on writes (costing both CPU and I/O bandwidth) and many concurrent I/Os need to be performed to access a single block (reducing available I/O operations). The performance effects on read operations are often greatly diminished when cache is available.

Mirrored
Data is mirrored, reducing capacity by half, but yielding a highly reliable and high-performing system. Recommended when space is considered ample, but performance is at a premium (for example, database storage).

Single Parity, Narrow stripes
RAID in which each stripe is kept to three data disks and a single parity disk. At normal stripe widths, single parity RAID offers few advantages over double parity RAID -- and has the major disadvantage of only being able to survive a single disk failure. However, at narrow stripe widths, this single parity RAID configuration can fill a gap between mirroring and double parity RAID: its narrow width offers better random read performance than the wider stripe double parity configuration, but it does not have quite the capacity cost of a mirrored configuration. While this configuration may be an appropriate compromise in some situations, it is generally not recommended unless capacity and random read performance must be carefully balanced: those who need more capacity are encouraged to opt for a wider, double-parity configuration; those for whom random read performance is of paramount importance are encouraged to consider either a mirrored configuration or (if the workload is amenable to it) a double parity RAID configuration with sufficient memory and dedicated cache devices to service the workload without requiring disk-based I/O.

Striped
Data is striped across disks, with no redundancy whatsoever. While this maximizes both performance and capacity, it comes at great cost: a single disk failure will result in data loss. This configuration is not recommended, and should only be used when data loss is considered to be an acceptable trade off for marginal gains in capacity and performance.

Triple mirrored
Data is triply mirrored, reducing capacity by one third, but yielding a very highly reliable and high-performing system. This configuration is intended for situations in which maximum performance, and availability are required while capacity is much less important (for example, database storage). Compared with a two-way mirror, a three-way mirror adds additional protection against disk failures and latent disk failures in particular during reconstruction for a previous failure.

Triple parity, wide stripes
RAID in which each stripe has three disks for parity, and for which wide stripes are configured to maximize for capacity. Wide stripes can exacerbate the performance effects of double parity RAID: while bandwidth will be acceptable, the number of I/O operations that the entire system can perform will be greatly diminished. Resilvering data after one or more drive failures can take significantly longer due to the wide stripes and low random I/O performance. As with other RAID configurations, the presence of cache can mitigate the effects on read performance.

The decision which profile to apply is depending on a number of variables like what the type of performance you need will be and for example how "secure" your data should be in relation to data loss and hardware failure. The decision you make has a direct impact on performance as well as usable storage on your appliance. It is of the highest importance that, before you do the installation, have discussed the options with the consumers of your storage. This can be for example database and application administrators or even the business.

After having completed this section of the setup you should have similar situation as shown below.

This is completing the primary initial setup and you will be able to start distributing the storage to servers and users who will make use of the new ZFS appliance within your corporate IT infrastructure.

Oracle ZFS analytics capabilities

When Oracle integrated Sun Microsystems a couple of years ago part of the acquired technologies was the ZFS filesystem. ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. The ZFS filesystem is (theoretically) capable of holding a maximum of 16 Exbibytes fo data.

As we can see in the Oracle strategy for storage is that they are building and shipping at this moment a number of storage and backup appliances based on the ZFS technology. At this moment they do ship the ZFS storage appliances ZS3-2, ZS3-4, 7120, 7320 and the 7420 and also Oracle is shipping the Sun ZFS backup Appliance.

A lot of exciting technologies are included in the ZFS storage appliances both on hardware level and software level which can help you get performance gains especially when used in combination with Oracle databases. However, often forgotten is that there is a management suite to manage and monitor you storage appliances.

The strategic product roadmap of Oracle, which is not officially communicated, shows that all management and monitoring solutions for both hardware and software should be integrated within the Oracle Enterprise Manager solution or at least (for now) interact with it. For the ZFS storage appliances a plugin is available to include functionality for managing and monitoring ZFS appliances from Oracle Enterprise Manager. You can download the plugin from the Oracle website.

However, next to the integrated way you have a standalone solution for managing and monitoring your ZFS appliances. This solution is holding the ZFS storage appliance analytics which helps tuning your storage to an optimum. The entire analytics solution is based on the dtrace capabilities, this means that a deep core analysis can be done.



In the above video you can see a bot more about the capabilities of the Analytics that you are able to pull out of a ZFS storage appliance and how they can help you in tuning your storage in a more efficient way.

The common analytics that are provided are:
- CPU: Percent utilization
- Cache: ARC accesses
- Cache: L2ARC I/O bytes
- Cache: L2ARC accesses
- Capacity: Capacity bytes used
- Capacity: Capacity percent used
- Capacity: System pool bytes used
- Capacity: System pool percent used
- Data Movement: Shadow migration bytes
- Data Movement: Shadow migration ops
- Data Movement: Shadow migration requests
- Data Movement: NDMP bytes statistics
- Data Movement: NDMP operations statistics
- Data Movement: Replication bytes
- Data Movement: Replication operations
- Disk: Disks
- Disk: I/O bytes
- Disk: I/O operations
- Network: Device bytes
- Network: Interface bytes
- Protocol: SMB operations
- Protocol: Fibre Channel bytes
- Protocol: Fibre Channel operations
- Protocol: FTP bytes
- Protocol: HTTP/WebDAV requests
- Protocol: iSCSI bytes
- Protocol: iSCSI operations
- Protocol: NFSv bytes
- Protocol: NFSv operations
- Protocol: SFTP bytes
- Protocol: SRP bytes
- Protocol: SRP operations

Next to the common analytics there are also a number of things where you can get more detailed and more advanced analytics on;
- CPU: CPUs
- CPU: Kernel spins
- Cache: ARC adaptive parameter
- Cache: ARC evicted bytes
- Cache: ARC size
- Cache: ARC target size
- Cache: DNLC accesses
- Cache: DNLC entries
- Cache: L2ARC errors
- Cache: L2ARC size
- Data Movement: NDMP bytes transferred to/from disk
- Data Movement: NDMP bytes transferred to/from tape
- Data Movement: NDMP file system operations
- Data Movement: NDMP jobs
- Data Movement: Replication latencies
- Disk: Percent utilization
- Disk: ZFS DMU operations
- Disk: ZFS logical I/O bytes
- Disk: ZFS logical I/O operations
- Memory: Dynamic memory usage
- Memory: Kernel memory
- Memory: Kernel memory in use
- Memory: Kernel memory lost to fragmentation
- Network: IP bytes
- Network: IP packets
- Network: TCP bytes
- Network: TCP packets
- System: NSCD backend requests
- System: NSCD operations

Getting all those analytics can be done via the GUI that is provided by Oracle. The mentioned analytics can help you tune your appliance and the way applications are interacting with it. One thing however is of vital importance, that you have a deep understanding of what the figures mean. A good starting guide is analytics guide from Oracle. However, this alone will not be sufficient. When running a mission critical system which is based upon a ZFS storage appliance and you have to deliver the most optimum performance a deep knowledge of ZFS and storage solutions will be needed.

Linux mount USB

USB disks are used more and more and almost nobody thinks about how a USB drive is attached to the computer. You plugin your disk and it works. When you are done you get your drive and move on. When using windows or most of the Linux GUI driven systems there is nothing to think about. However, if you run a couple of commandline only Linux systems you have to do more than only plugin your USB cable. However, it is not very hard to do this from a Linux shell in a manual way. Basicly it is the same as mounting a new drive or network drive on your system.

The trick is to find out where the drive is mounted under /dev and then mount it at a location on your system by using the standard commands you are used to.

When you attache your USb drive to the system it can be good to have a tail -f running on your /var/log/messsages. You will find a message somthing like the one below when you attache your drive.

Sep 16 17:26:09 homebox00 kernel: [2063013.339435] usb 4-1: new high speed USB device using ehci_hcd and address 4
Sep 16 17:26:10 homebox00 kernel: [2063013.472602] usb 4-1: configuration #1 chosen from 1 choice
Sep 16 17:26:10 homebox00 kernel: [2063013.476190] scsi4 : SCSI emulation for USB Mass Storage devices
Sep 16 17:26:10 homebox00 kernel: [2063013.477177] usb 4-1: New USB device found, idVendor=1058, idProduct=1001
Sep 16 17:26:10 homebox00 kernel: [2063013.477849] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Sep 16 17:26:10 homebox00 kernel: [2063013.478958] usb 4-1: Product: External HDD    
Sep 16 17:26:10 homebox00 kernel: [2063013.479588] usb 4-1: Manufacturer: Western Digital 
Sep 16 17:26:10 homebox00 kernel: [2063013.480205] usb 4-1: SerialNumber: 574341553433383836343732
Sep 16 17:26:15 homebox00 kernel: [2063018.486535] scsi 4:0:0:0: Direct-Access     WD       10EAVS External  1.05 PQ: 0 ANSI: 4
Sep 16 17:26:15 homebox00 kernel: [2063018.504738] sd 4:0:0:0: [sda] 1953525168 512-byte hardware sectors (1000205 MB)
Sep 16 17:26:15 homebox00 kernel: [2063018.506813] sd 4:0:0:0: [sda] Write Protect is off
Sep 16 17:26:15 homebox00 kernel: [2063018.509230] sd 4:0:0:0: [sda] 1953525168 512-byte hardware sectors (1000205 MB)
Sep 16 17:26:15 homebox00 kernel: [2063018.511297] sd 4:0:0:0: [sda] Write Protect is off
Sep 16 17:26:15 homebox00 kernel: [2063018.512656]  sda: unknown partition table
Sep 16 17:26:15 homebox00 kernel: [2063018.518501] sd 4:0:0:0: [sda] Attached SCSI disk

From this you can see that the new drive is at /dev/sda . At the system I am attaching this drive I have a directory at /storage/disk0 and I will use this location to mount the USB drive.

You can now simply use the below command:
mount /dev/sda /storage/disk0

If you do a df -h you will see the new drive attached to your /storage/disk0 location. This can be any other location you want.

Linux and EXT4

It was already announched and we could already find it in the latest Friday build of Ubuntu, EXT4 support under Ubuntu. EXT4 is a Journaling file system, a journaling file system is a file system that logs changes to a journal (usually a circular log in a dedicated area) before committing them to the main file system. Such file systems are less likely to become corrupted in the event of power failure or system crash.

And an other great thing is that it can be really fast, you can read some benchmarking at phoronix.com and you can read some nice information about testing softpedia.com. Ubuntu booting in 21.4 seconds,..... that is very very fast in my opinion. How much this is due to the EXT4 filesystem, I think there are also a lot of other things that can help in Ubuntu 9.04.

However the support of EXT4 under Linux is a good step, one of the great values of supporting EXT4 is that you now can work with very large volumes. EXT4 is supporting currently volumes up until 1 exbibyte and files up until 16 TiB. A other thing is the way EXT4 can 'reserve' space on the disk, this can come in handy when creating large files for for example virtual disks when running virtual machines also streaming and databases come to mind.

"The ext4 filesystem allows for pre-allocation of on-disk space for a file. The current methodology for this on most file systems is to write the file full of 0's to reserve the space when the file is created (although XFS has an ioctl to allow for true pre-allocation as well). This method would no longer be required for ext4; instead, a new preallocate() system call was added to the linux kernel for use by filesystems, including ext4 and XFS, that have this capability. The space allocated for files such as these would be guaranteed and would likely be contiguous. This has applications for media streaming and databases."

Besides this EXT4 is also trying to prevent in all kinds of ways to prevent fragmentation, EXT4 has an online defragmenter. Even with the various techniques used to avoid it, a long lived file system does tend to become fragmented over time. ext4 will have a tool which can defragment individual files or entire file systems.

NDMP backup stalls

We are currently using a netvault backup system in combination with a Netapp filer. To be able for netvault to communicate with the tape library and the tape drives to make the backup ndmpd is used as a deamon. However in some cases those sessions will be hanging. Only in extreme cases. This is what I found in a help document, it can come in handy if you have a problem:

NDMP backup says "Writing to Media" even though it is not writing to media.
Affected NV Version: 7.4.x
OS Version: All
Plugin version: 6.3.x
Application version: N/A

Description:
Several jobs continually say "Writing to Media" even though they are not writing to media. If you look at the logs, it shows a "Channel Error" near the bottom of the log. In "Device Manager" the drive used for these backups says: DRIVE 1 (Locked by Session(Hard).

Solution/Workaround/Procedure:

Most likely the ndmp sessions on the filer have entered a hung state. Issue the following commands at the console prompt of the filer;

ndmpd killall
ndmpd off
ndmpd on
ndmpd status (shows if all strays processes have been eliminated)

This should resolve issue.

Fastest Database machine

The Oracle Exadata server is claimed to be the fastest hardware capable of running Oracle databases. The machine is developerd by HP and Oracle and is specially tuned to run Oracle databases. Oracle states the following: "Exadata delivers outstanding I/O and SQL processing performance for data warehousing applications by leveraging a massively parallel architecture to enable a dynamic storage grid for Oracle Database 11g deployments."

According to Oracle the main problem with current architectures is that they are unaware of the fact that they are housing a database. When having, for example a hundred terabyte plus, database this will reside on a large amount of disks which are coupled to your server(s) via Host Bus Adapters to your storage area network. The way traditional HBA's are handling the IO load is not optimal for a very large datasets. The Exadata server makes use of a massively parallel architecture to enable more pipes, also the pipes are made wider to transport more data. Also the storage part is SQL aware and will only 'ship' the data needed by the requesting SQL statement. Those improvements will provide a architecture in which you pipes are not filled with data you do not need and the system van make use of more and wider pipes to the server so you will lose the bottlenecks of I/O in your architecture.

The Exadata server makes use of InfiniBand interconnect to connect the storage to the server. You can check the InfinBand speeds on this wiki page. InfiniBand is a switched fabric communications link primarily used in high-performance computing. Its features include quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. It is a superset of the Virtual Interface Architecture.

Oracle and HP have chosen to make the ExaData solution robust by making use of disk mirroring provided via Automatic Storage Management (ASM), and hot swappable Exadata disks, ensure the database can tolerate the failure of individual disk drives. Data is mirrored across cells to ensure that the failure of a cell will not cause loss of data, or inhibit data accessibility. This massively parallel architecture delivers unbounded scalability and high availability.

The SQL processing optimization is done by function shipping of SQL statements. A statement is offloaded from the database server tier and shipped to the storage tier. By doing this the CPU time is not used on the database server, the statement is executed on the storage tier and only the result is shipped back to the database server. Meaning a where statement is executed on the storage tier and only the filtered results are send back. In a traditional way all the data would be send to the database server and the filtering would be done on the database server. By working like this you will have less CPU time on the database server which enables the database to handle more calls and there will be less raw data-traffic.

By doing this online transaction processing and real-time query operations on life data sets will have a huge benefit. Because you will have more than one 'cell'/ 'storage bay' with its own CPU and all the disks in a cell will work in parallel you will be able to run large numbers of very complex queries without having to compromise on the performance of your system. To make optimal use of the ExaData architecture a Oracle 11g database should be used.

Oracle database 11G is equipped with iDB in the kernel. iDB stands for the Intelligent Database protocol and is build on the industry standard Reliable Datagram Sockets protocol RDSv3. The protocol runs over InfiniBand. ZDP (Zero-loss Zero-copy Datagram Protocol), a zero-copy implementation of RDS is used to eliminate unnecessary copying of blocks. Multiple network interfaces can be used on the database servers and Exadata cells. This is an extremely fast low-latency protocol that minimizes the number of data copies required to service I/O operations. iDB is used to ship SQL operations down to the Exadata cells for execution and to return query result sets to the database kernel. Instead of returning database blocks.

Reliable Datagram Sockets is a open standard under GPL license and is maintained by The OpenFabrics Alliance. Oracle is running a couple of projects on RDS, for example you can check this oss.oracle.com site on the performance and testing results of RDS.

Oracle Exadata Storage Server

Oracle is getting into hardware. Larry today announched that they have created in combination with HP the world’s fastest hardware for running databases and especially data warehouses: the Exadata Storage Server.

With databases and data warehouses getting bigger and bigger and well above the 1 Tb limits Oracle found that they also have to optimize the hardware to provide customers with a good and well preforming application. Because of this they have developed with HP the Exadata Storage Server. First see the movie below and I will post more details on this weblog in the next post.

Oracle VM and storage solutions

The past couple of days it is raining information about all the hardware vendors who are launching products that are certified and/or developed to use with Oracle Enterprise Linux and Oracle VM. Qlogic announced the launch of certified FC equipment, Compellent launched some information about how to use there storage solution in combiantion with Oracle VM, NetApp has launched a whitepaper on how to use NetApp in combiantion with oracle VM.

And now Pillar Data Systems released a pressrelease named: "Pillar Data Systems Announces New Application-Aware Storage Profiles for Oracle Database Environments". You can read the complete press-relase on businesswire.com

Compellent storage and Oracle VM

"Compellent Supports Oracle(R) VM to Deliver Advanced, Virtual Storage and Server Infrastructure". Compellent today announced that they will expand their products and services in such a way that it will optimally work with Oracle VM. This, in combiantion with the announchement today from Qlogic that they now will ship Oracle VM certified hardware, will make the adoption of Oracle VM and Oracle Enterprise Linux in 'high storage demanding' enviroments even more logical. Besides this it is also giving a good alternative for those customers who like to have an other storage solution than using EMC or NetApp.

"Compellent and Oracle are aligned in our approach to helping organizations realize the full benefits of a virtual enterprise, and we work together to certify and provide solutions that are seamlessly interoperable," said Bruce Kornfeld, vice president of marketing, Compellent. "Oracle's expertise on the server side, and our storage credentials, are enabling companies to improve performance and availability of business critical applications while reducing costs and simplifying IT management."

Linux disk scheduler fixes

Thanks to Google we will now have better performance on disks when using the latest linux kernels. Google sponsored for six months development and research at Gelato@UNSW who did benchmarking and created fixes for the following schedulers: noop, deadline, anticipatory and for “Complete Fair Queuing (CFQ)”. Also they created two new schedulers

- IOScheduling/VRscheduler: an implementation of the V(R) scheduling algorithm;
- IOScheduling/FIFOscheduler: a pure FCFS scheduler.

A disk scheduler is the part of the Linux kernel that reorders, delays, and merges requests for disk I/O to achieve better throughput and lower latency than would happen if all the requests were sent straight to disk. The scheduler aims to minimize disk head movement as far as possible. You can think of the scheduler as a queue of requests waiting for service at the disk.

If you like to check the latest kernel go to kernel.org

Oracle RAC Cluster on Linux

Oracle released a paper on how to build your own Oracle RAC Cluster on Oracle Enterprise Linux and ISCSI. The guide will help you to build your own RAC cluster for under US$2600. I however states that this is only for a testing environment and not for a production environment. The paper will go over most basic points of your hardware environment and storage solutions.

On the storage part there are some notes that go “against” the use of a standard NAS/NFS and tend to go more for a Direct NFS Client approach, Direct NFS Clients will integrate NFS directly to Oracle and by doing this Oracle will be able to control and fine-tune the I/O path between the Oracle software and the NFS server resulting in significant performance gains. In the example however will be worked with a ISCSI solution.

For the rest it will guide you step by step installing all the nodes and setting up networking and other environment parts. A great guide when you like to start with RAC in a test environment.

Google Gdrive online storage

The Wall Street Journal reported today that Google has come out with plans to provide a online storage solution. Even do we all have seen this coming for a long time it now becomes reality in, most likely, a quite short time.

The article states that you will be able to "buy" more storage on-top the standard amount of storage data you now get across your Google applications as Gmail etc etc. How it will turn out, the near future will most likely tell us. However, if Google launches this new application we can be sure it will become a success as most of the launches they have done (not counting Orkut). Most likely it will enable you to share your files with your desktop computer and most likely it will have a nice integration with Google applications so you can save the files you create here in the same pool as your regular files.

Difference between df and du

Under Linux / UNIX we have the du and the df command. The du command display disk usage statistics and the df command is to display free disk space.

In some cases it is possible that there is a difference between the outcome of df and du. In 99.9% of the cases this is caused by a lock on the file system. Meaning, if an application opens a file or files and an other application deletes these files while they are opened they will still be calculated by df.

The df command calculates this by looking at the inode table and the inode table has not yet released those files, meaning that, according to the inode table, the files are still using space on the disk.

Generally the solution is to stop/start the process that has a lock on those non existing files and the results returned by df and du will be the same again.