Supported data
Supported data types
Encord enables you to work with several types of data. Whether you're looking to annotate sequences of images, videos, or even DICOM files - we've got you covered!
Comparing file formats
Video Tutorial - Comparing images, image groups, and image sequences
Comparison of image file types
| Single Image | Image Group | Image Sequence | |
|---|---|---|---|
| 'Write' permissions required in cloud storage? | No | No | Yes |
| Multiple images per task? | No | Yes | Yes |
| Data loss during creation? | No | No | Yes* |
| ML features available for labeling? | No | No | Yes |
* Data is lost during the creation of an image sequence when images of varying dimensions are made into a single image sequence. For more information about this, please see the section on image sequences below.
Supported file formats
| Single Image | Image Group | Image Sequence | Video | |
|---|---|---|---|---|
| Supported file formats | .jpeg .png .webp .avif .bmp .tiff* .tif* | .jpeg .png .webp .avif .bmp .tiff* .tif* | .jpeg .png .webp .avif .bmp | .mp4 .mov .webm .mkv |
* TIFF files can only be viewed using the Safari browser.
Single images
Single images are individual images that are uploaded to Encord as separate files. Each single image constitutes its own data package, hence images will be listed individually.
We support the following file types for single images:
.jpeg, .png, .webp, .avif, .bmp, .tiff*, .tif*
* TIFF files can only be viewed using the Safari browser.
Image groups
Image groups are collections of images that are grouped together into a single data unit. They can contain images of varying orientations and no data is lost in the process of creating an image group.
We support the following file types for image sequences:
.jpeg, .png, .webp, .avif, .bmp, .tiff*, .tif*
* TIFF files can only be viewed using the Safari browser.
Image sequences
Image sequences are collections of images that are grouped together as a single data unit, and annotated in the same way as videos. As a result, image sequences render faster in the browser (provided images are sequential), and are able to make use of powerful machine learning features such as 'auto-segmentation' and 'automated labeling'.
We support the following file types for image sequences:
.jpeg, .png, .webp, .avif, .bmp
Creating image sequences using the web-app
If you use the Encord web-app to upload your images and create image sequences, Encord will automatically group images of different heights and widths into separate image sequences. This is done to preserve the quality of each frame within the resulting image sequences.
Creating image sequences from cloud storage using JSON or CSV
If you upload your data from cloud storage using JSON or CSV file formats, you retain full control over which image sequence every image will end up in. However, if images you choose to form into a sequence have varying dimensions they will be homogenized. This means that every image within your defined sequence will be made to have the same dimensions as the first image in that sequence. As a result, data may be lost when forming image sequences.
The same applies when uploading image sequences from the SDK.
Creating image sequences using the SDK
For information on how to create any type of dataset using the SDK, please see our dedicated SDK documentation.
Videos
Video files can be uploaded onto the Encord platform and annotated frame by frame, enabling you to track elements within the video.
File uploads must be less than 2 GB or 200,000 frames (2hrs @ 30FPS).
Even though frames are treated as individual images when being annotated, the video itself constitutes a single data package.
Encord has powerful machine learning features such as 'object detection and segmentation' available that can help automate parts of your video annotation process.
We currently support the following video file types:
.mp4 .mov .webm .mkv
However, for optimized performance we recommend using the following video formats:
.mp4file with h.264..webmwith vp9.
DICOM
Encord provides native support for Digital Imaging and Communications in Medicine (DICOM) browser rendering and data annotations. Our ontologies allow you to create any type of labeling protocol - for example RECIST, which requires measuring the longest diameter of a lesion.
With the DICOM editor, you can:
- Annotate modalities such as CT, X-ray, and MRI.
- Label using any annotation type in 2D (with 3D in the works), and seamlessly toggle between axial, coronal, and sagittal views.
- Render 20,000 pixel intensities and set custom window widths & levels.
- Natively display DICOM metadata.
- Track and interpolate objects between slices.
- Reduce manual annotations with automation features.
Due to DICOM being a large and open standard Encord will accept most data types. Examples include: CT, MR, US, MG, TOMO, but many more will be accepted.
To learn how to upload dicom files from your cloud storage please see:
Multi-frame DICOM Support
We support multi-frame DICOMs, making it even easier and faster to load data onto our platform! Multi-frame DICOM support can save time and streamline workflows due to a reduced amount of header data repetition.
Simply upload your multi-frame DICOM files like you would any other files - either via the user interface, or using our python SDK.
Caching DICOM data
Caching allows frequently accessed DICOM files to be stored securely in your local browser memory, allowing them to be retrieved quickly and efficiently. As a result you can expect faster load times when accessing medical imaging data.
This is particularly beneficial for busy healthcare professionals who require quick access to patient records and diagnostic imaging.
Data Optimization
Whatever data you upload, it will have to fulfill certain criteria in order to be used seamlessly within our platform. This section will run through all topics related optimizing your data to work with Encord's features.
Re-encode your data
If we detect that the video you uploaded can't be added to a dataset, it will have to be re-encoded. This simply means bringing your video in line with our requirements to avoid frame synchronization issues and ensure your labels are as accurate as possible.
Your video needs re-encoding if you see 
after successfully uploading your video. Hover over the icon to read what issues we have found with the video.
To automatically re-encode your video, simply select the file in question and click the Re-encode (auto) button to prevent any frame synchronization issues from arising. A notification telling you your video is re-encoded should appear at the top of your screen.
When in doubt, you can always check the progress of any process by clicking the 
icon. The number displayed in red signifies how many new notifications you have.
The way in which individual frames within different browsers are displayed may vary. To avoid browser-related frame synchronization issues we strongly recommend the Google Chrome browser when using our platform.
To learn more about frame synchronization issues and re-encoding your data please visit the relevant section below.
Re-encode locally
Some cases, such as corrupted metadata, might require you to re-encode your data locally before uploading them to the Encord platform.
To do this, try the following ffmpeg command, replacing "video.mp4" with the name of the file you would like to re-encode, and "re-encoded-video.mp4" with the name you'd like the re-encoded file to have:
ffmpeg -err_detect aggressive -fflags discardcorrupt -i video.mp4 -c:v libx264 -movflags faststart -an -tune zerolatency re-encoded-video.mp4
Fixing cloud storage buffering issues
A number of factors determine how fast remote content is rendered and displayed in a web browser:
- Geographical distance from host bucket.
- Local internet speed.
- Use of a Virtual Private Network (VPN).
- The bitrate of the video file being requested.
While these factors are outside of Encord's influence, we have put together a set of common issues when loading data from private cloud, along with possible solutions to improve your user experience.
You can see the speed at which a video is being downloaded at the bottom of the 'editor pane' in the 'label editor' on Encord's platform. It also displays the download speed required to display a video without buffering.
Geographical distance from host
The physical distance between you and the server hosting your data will affect how quickly content can be accessed. If, for example, an Amazon S3 bucket is hosted in the 'EU (London) eu-west-2' region, then users from India, or Brazil might experience latency issues due to data having to travel a long way.
The solution is to make sure that users always connect to their closest server, reducing the time it takes for data to travel from one place to another - however, setting this up differs between different cloud storage providers.
Amazon S3
AWS allows you to set up Multi-Region Access Points that provide a single global end point to access a replicated dataset spanning multiple S3 buckets. This means that users all around the world are always connected to their closest server when accessing content from an S3 bucket.
To make this work in Encord, please follow the steps below:
Set up Multi-Region Access Point on Amazon S3. Their step-by-step guide can be found here.
Integrate your Multi-Region Access Point with Encord. The process is documented here.
Upload data to a dataset from using the Multi-Region Access Point integration. When specifying which files to upload in your JSON file, make sure to use the ARN of the Multi-Region Access Point followed by the object name, instead of the object URL, as documented here.
GCP
You can check your latency to Google Cloud regions here, and the latency between different regions here to assess the connection speed to your GCP bucket.
We recommend you use the 'Premium' network service tier if you experience buffering issues while using data stored on GCP. This will ensure a high-performing network experience with a global footprint.
Local internet speed
Test your internet connection to make sure it's fast enough to download your video from cloud storage without buffering. Consistently low internet speeds might mean your provider isn't providing you with high enough internet speeds to load large quantities of data from your cloud storage.
You can see the speed at which a video is being downloaded at the bottom of the 'editor pane' in the 'label editor' on Encord's platform. It also displays the download speed required to display a video without buffering.
VPNs
VPNs can slow down your connection by requiring the data travel through the VPN's server before arriving to you.
We recommend you turn off your VPN when using Encord's platform if you are experiencing buffering issues.
High bitrates in videos
Video content itself varies widely in terms of frames rate, compression, definition and size. The most important factor when determining how quickly a remote video will load in the browser is the bitrate. The higher the bitrate, the more data will need to be loaded per second of video.
Lowering a video's bitrate will reduce the time it takes to load from your cloud storage. This can be achieved using tools such as FFmpeg.
Lowering the bitrate involves decreasing the video quality. Therefore it may only be suitable for clients who don't rely on high-quality videos.
Compatability
Data integrity is a top priority at Encord.
Modern browsers give no control over deterministically displaying frames and rendering videos - therefore, we've introduced security mechanisms to ensure that annotations are correctly displayed in the browser. Please follow the guidelines below to ensure those security mechanisms will be working for you.
Supported data formats
We support all the major video and image formats, however not all video formats are supported by all browsers for playback. If you have trouble with playback in the 'Label editor', it may be a browser issue. Refer to your browser documentation for more information if issues persist after implementing the guidelines listed below.
For videos, the Label editor expects integer frame rates from 10-30 fps. Non-integer frame rates will generate a warning in the Label editor, and although labeling can continue with non-integer frame rates, please be aware this can introduce errors into your annotations.
Supported browsers
We strongly encourage the use of Chromium-based browsers, Google Chrome in particular.
We haven't tested frame synchronization and security mechanisms on other browsers, so we strongly encourage to use Google Chrome for annotating data and using our platform in general.
We also recommend turning off hardware acceleration in Chrome. Hardware acceleration adds another layer of indeterminism to video rendering in the browser. To eliminate unexpected behaviour it is best to disable it.
Frame synchronization
Our servers use the FFmpeg project to extract frames from a video one by one. That way every label for a video is mapped to a specific frame index within that video, providing a reliable way to map labels to the specific frames in the video.
This process is not as straight forward in the browser. When a video gets embedded in the browser, we have no control over how exactly the individual frames are displayed when the video is played. Even if you upload a video which has a constant frame rate of 30 fps, this unfortunately does not guarantee that the browser will play exactly 30 frames in a second. In order to seek individual frames we have to seek specific timestamps, and thus we are at the mercy of the browser to decide which frames start and end at which timestamps.
We have worked hard to create the necessary safety mechanisms to detect possible issues and provide a robust solution to ensure labels are saved to the correct frame index.
Frame synchronization issue causes and solutions
We report frame synchronization issues in the dataset view after a video has been uploaded.
Hover over the icon in the video row to see which issues have been identified.
We also report frame synchronization issues when the video is loaded in the label editor with the following warning:
Our recommended solution is to re-encode problematic videos in your dataset. You can use Encord to re-encode your videos using the following steps:
- Navigate to your dataset in the dataset settings.
- Select all the videos that need re-encoding.
- Hit the Re-encode button.
- Wait until you see the re-encoded videos added to the dataset. The re-encoded videos will have the same title as the selected videos, with "_normalized" appended in the title.
- Optionally remove the previously selected videos to avoid having problematic videos being labelled.
When you click the re-encode button, our platform will apply the least invasive transformation to the video to fix the issues. This is so that the re-encoded video and the original video are as similar as possible which will reduce possible frictions in your integration pipeline with the Encord platform. You can read more about what happens during re-encoding in the examples below.
To detect some of these problems you can inspect your video with ffprobe. You will need to download FFmpeg to do so.
Audio frames
A video has "audio frames" whenever the video has any audio with it. We have observed unexpected behaviour where the browser video player would sometimes increase the display length of specific frames when audio frames were present. As we can only seek frames by timestamps in the browser, whenever we detect audio frames we recommend to remove those.
Detecting the problem:
- Check the warnings on the Encord platform
- Listen for sound in the video. Unfortunately this is not a guarantee as sometimes there can be audio frames even when there is no audible sound present
- By running
ffprobe -i $YOUR_FILE -show_streams -select_streams a -loglevel error
If there is no output, there are no audio frames. If you see some output, there are some audio frames.
Fixing the problem: Re-encode the video via the Encord platform. This will copy all the video frames but will drop all the audio frames.
Ghost frames
Different video players will display between 0 and all of the frames with negative timestamps. With no control over the browser's video player, we cannot deterministically tell how many negative keyframes the browser will display.
We want to make sure that we are not labeling any negative timestamp frames. Therefore, we require you to re-encode the video which will remove any negative timestamps.
Detecting the problem:
- Check the warnings on the Encord platform
- By using different media players and observing whether some of them start at different frames in the video.
- By running
ffprobe -show_packets -select_streams v -read_intervals %+3 -of json $YOUR_FILE
This will show the video packets of the first 3 seconds in a JSON format. You will see an output such as
[...]
"packets": [
{
"codec_type": "video",
"stream_index": 0,
"pts": -16384,
"pts_time": "-1.066667",
"dts": -16896,
"dts_time": "-1.100000",
"duration": 512,
"duration_time": "0.033333",
"size": "71396",
"pos": "37476",
"flags": "KD"
},
[...]
If in any of the packets you have a negative value for pts, you know that there might be some ghost frames.
Fixing the problem: Re-encode the video via the Encord platform. These are the transformations we apply when we detect ghost frames:
- The audio frames are dropped
- The video frames are re-encoded (i.e. the images in the videos are decompressed and compressed again)
- All unsupported file formats are converted to
.mp4files - Corrupted frames are dropped
Variable frame rates
Detecting the problem:
- Check the warnings on the Encord platform
- Run the same ffprobe command as for ghost frames (above) and check if the distance between the
ptsis variable
Fixing the problem: Re-encode the video via the Encord platform. These are the transformations we apply when we detect a variable frame rate:
- A constant frame rate is applied to the video. This requires some dropping and duplicating of frames and is handled by the
-vsync cfrFFmpeg flag - All the transformations that are applied to videos with ghost frames
Existing labels before re-encoding
In some cases you might already have some existing labels that you upload on our platform. If this is the case and we suggest you to re-encode the video, then your new labels might be misplaced. This could be the case if
- The labels were created incorrectly before, because of frame synchronization issues on other platforms
- You have labels for a variable frame rate video and now the frame rate has changed
If you upload labels and need to re-encode your video we strongly encourage you to check for the correctness of your pre-existing labels after re-encoding the video.
If you believe that your labels are misaligned, please do reach out to the Encord team so we can help you with the uploading process.