Reporting Large Files To The NDR

If you are finding it is taking longer than you expect to load data into the NDR, it might not simply be a case of your bandwidth.

When uploading data to the NDR the system has to check the integrity of the upload to the storage location.
 

Reasons that might be causing slow upload include:

1. Your network speed.
2. Your computer hardware set up - are you using a suitable petro-technical machine?
3. Your CPU usage during the calculation of integrity checks and compressing before uploading the data. In the below example, you can see that it is maxing out one of the computer cores. The little blue line bursts are the actual upload which is not saturated at all.

The CPU activity is intensive and if you have a remote connection via  VPN  or a virtual machine, for example, it may not be clear what CPU power you are actually accessing and it may be insufficient to optimally do the job.

For large datasets, an upload job may run for an extended period of time.  It is recommended that the user refresh their credentials periodically, and we recommend this is done approximately every 12hrs or so.  This requirement is prescribed by Microsoft Azure where certain applications using an authorization code flow have a refresh token.  To do this, open another tab in the chrome window and login there too to refresh the token before it expires. i.e. morning and evening.  This will keep the upload running smoothly.

An example case:

An operator runs 3 parallel upload tabs on their 4 CPU machine (i.e. N-1) and they log in to the NDR on the 4th tab every 12 hours to ensure their credentials are refreshed. 

They are only loading three SEG-D files per tab.

They are otherwise doing no other work on their machine. This enables them to upload ~1.8 TB of SEG-D in @24 hrs.