Data Import Pipeline

How to turn your raw experimental data file into functioning data sets which you can interact with TeselaGen

Written by Eduardo Abeliuk
Updated over a week ago

Your process to turn your raw data into organized and functionable data starts with data parsing with the Parse Experiment Data tool.

This tool parses your file (*.csv or *.xlsx) with experiment data and stores the content in Data Grid(s). The Data Parsers are customizable and are set-up as Node-RED flows at the Integration Server. Once the data is correctly parsed and successfully stored in Data Grid(s) you can run the import experiment data into Assay to get it into a TEST Assay.

Accessing the tool:

You can access the data parsing tool by first going to the BUILD module of the Teselagen platform and select Tools from main menu. Once you click on it, the Tool Library will open up.

Go to the search bar and type Parse Experiment Data and launch the tool from there to initiate the process.

Uploading your file:

Once you click the Launch Tool button, it will upon you the Upload Data File screen.

You can either click on the Up arrow icon to upload the experimental date file or just drag the file on the uploading area.

After you've upload the file click on the Select a parser drop-down in the Select data parser file section. Once you've imported the spreadsheet that file gets parsed through a node-red flow.

The node-red flow* is a part of Teselagen's integration which allows users to create their custom logic-flows in the node-red interface.

Your file will be transferred to the flow created on node-red and it will parse it and generate data-grids.

(go to the end of this document to see how you can access node-red)

Once the grids are generated, your data file will be created will be created which you can access by clicking on the hyperlinked name as indicated by 1 in the image below.

This will open up the Data section of the BUILD module and you can find your parsed file their indicated by the file's name.

You will now see your data in a tabulated format with the status "The data grid hasn't been imported into an Assay yet." That is the second part of the data-import pipeline.

Importing Data into an Assay

The next part of the data import pipeline is importing parsed data into an assay so that we can interact with it too such as plotting it on a graph or run statistical analysis.

You can initiate the process by clicking on Import into Assay.

The Import Experiment Data into Assay tool takes in a Data Grid, usually created by parsing a Data File with the Parse Experiment Data tool, and imports its content into a new or existent Assay. Here the data columns will need to be mapped to a TEST Data Types (metadata). At completion, the experimental data will be fully imported and ready for visualization in TEST or used by DISCOVER Machine Learning workflows.

Once inside the Import Experiment Data into Assay tool select or create an experiment and an assay, you can also use the toggle switch button in the Selected Data grid section to show or hide the data-grids.

Once you've selected your experiment and assay, click on the Next button to start the subject link process. This links assay subjects to aliquots.

Data Mapping

The column names of your data have to be linked to a column type in the Teselagen platform in order for the platform and the machine to understand the column.

Inside the Data Mapping section, you will map each column with the appropriate TEST metadata class and subclass. You can also choose one of your own saved mappers or use TeselaGen's Smart Mapper suggestions.

You can either use one of your own saved mappers or use the smart-mapper feature to link and map your columns. Make sure to review the columns and their auto-proposed class and sub-classes are in accordance with your data type.

Once the mapping is done, click on validate and then submit to finish the process. If there is something wrong with the mapping, you'll see error messages to help you fix them.

*Accessing Node-Red:

You can access the node-red integration server by clicking on your profile icon at the top-right of your screen and then going into settings

Once you're in the settings, click on integration from the side-menu and choose file to data-grid to view or add parsing logic flow on node-red.

Did this answer your question?