How can I split a large data set?

Overview

Field ops data can be imported to Granular Insights on the web as a zipped folder (no larger than 10 GB).

In some situations, you may find that your data is too large to load. If your data takes too long to load, or if your zipped folder is larger than 10 GB, the upload may fail, or there may be issues while processing the data. 

In this article, we will cover how to split your large data set into multiple folders, for faster upload. Here are the steps to follow:

1. Create a copy of your raw data
2. Create multiple new folders for your data with the correct folder structure
3. Copy your data into the new folders 

Requirements

• Data from a supported monitor

• Knowledge of the folder structure for monitor data

1. Create a copy of your raw data

We do not recommend making edits to the original data from your monitor. Instead, you should start by creating a copy of your data. 

1. Create a new folder where you will put your copied data.
   • Note: to quickly find your data, we recommend that you create this folder on your computer's desktop or another easily accessible folder.

2. Select the folder on the data card.

3. Right click on the folder that contains the data. Select Copy from the menu.

   

4. Navigate to the destination folder (the new folder you created in step 1).

5. Right click and select Paste.

   

6. Open the new folder you created. You should now see a complete copy of your machine data.

 2. Create multiple new folders for your data with the correct folder structure

When creating the new folders that you will use to split your data, we recommend that you have two File Explorer windows open, side-by-side. Having two windows open will allow you to quickly reference the original file structure while you mirror the structure in your new folder. In the instructions below, we will refer to these windows as the "left window" and the "right window", for clarity.

1. In one File Explorer window (left), navigate to the folder that contains your copied data (see part 1 above for how to copy your data).
2. In the second File Explorer window (right), navigate to the same folder as above. At this point both of your File Explorer windows should be showing the same thing.
3. In this window (right), create a New Folder and name it 'Part 1'.
   
   
4. Double-click the newly created folder ('Part 1') to open it.
5. In this folder, recreate the required folder structure for your data type. You can double-click into the copied data on the left window to open each folder and display the series of nested folders that you will need to recreate. For example:
   • If you have John Deere GS4 data, your folder structure could look like:
     • Folder > JD-Data > log > [Year_Grower_Farm_Field] > .jdl
   • If you have Precision 20/20 data, your folder structure could look like:
     • Folder 1 > Folder 2 > [data files] > .dat
6. Continue to repeat steps 3-5 until you've created enough structured folders to split apart all of your data. For example, if you want to split your copied data into three parts, you will complete these steps three times. The new structured folders can be named 'Part 1', 'Part 2', Part 3'.

3. Copy your data into the new folders

1. In the left window, ensure you are still looking at your copied data. Navigate to the folder that contains the data files (the 'log' folder in the example below).

2. Select some of the folders to copy. We suggest copying 5-10 folders. To select multiple folders at a time, hold down the Shift button and click on multiple folders to select them.
   
   

3. Right click the selected folders and click Copy.

4. In the window on your right screen, open the newly created folder (i.e. 'Part 1'). Open the nested folders until you get to the same folder as is showing on the left window (i.e. 'log' in the example above). Right click and select Paste. The copied folders from the left window should now appear in the right window.

5. Continue to copy folders from the left window into the new folders on the right window ('Part 2', 'Part 3', etc.) until all the data folders from the left window have been copied and allocated to a folder on the right.

Now that your data set has been split between multiple folders, you can zip them and upload them into Insights. Note: you should not zip the folder labelled 'Part 1'. Instead, zip the folder you see immediately after opening 'Part 1'.

 

Related Articles:

Manually Upload Machine Data

Supported File Types and Monitor Brands

Monitor Folder Structure and File Types